On interfaces and event processing

Interfaces is one of the main next frontiers for event processing systems.  In the picture you can see an illustration of brain-computer interface which is the most advanced interface that is mentioned in a recent human-computer interface survey,  this survey starts from command line, moves through mouse and keyboard and gets to gesture detection, and up to the mostly futuristic brain interface.  My daughter Daphna got for her recent birthday XBOX machine with Kinect, which is capable of voice and gesture recognition and provide advanced game experience.   


Getting to event processing, the human computer interface is crucial in order to raise the abstraction level and enable to extend the type of developers and users, currently the development interfaces are geared mostly towards programmers, and are in essence close in nature to the implementation abstractions, a separation between the implementation abstractions and the development abstractions is one of the major challenges, in general in software, but an important one in event processing, since realizing the event processing potential entails the ability to make application development accessible  to other communities such as the spreadsheet programming community (e.g. business analysts).   I have written in the past about this requirement, and will probably deal with it more in 2011. 


Talking about interfaces, there is another type of interfaces the application programming interfaces (API), this is important since one of the major utilization of event processing is as embedded capabilities inside applications/middleware/solutions.   Thus, APIs that are appropriate for multiple uses are gaining importance for the interoperability and  for making EP  easily embeddable.   API design has become an emerging area, a blog posting by Shanley Kane from Apigee (and API management provider) provides trends and predictions for APIs.  It is interesting to note that the first prediction is entitled "APIs go real-time big-time" (in the source all words are capitalized, but I have adopted Manning's writing style while writing the EPIA book which is against over-capitalizing).   This prediction talks about the popularity for APIs of push oriented pub/sub systems.   This is elementary event processing,  APIs for more sophisticated event processing is the next step.
Standard APIs for EP will enable plugging EP components that are doing aggregation, translation and pattern matching components as part of an "event-driven Web" (AKA event-processing fabric) that was declared as the grand challenge in the EP Dagstuhl seminar.  I'll write more about this grand challenge soon.


In any event, both human computer interfaces and application programming interfaces are major part of the work needed towards the next generation of event processing systems. 

On event processing as a technology and as a business

Philip Howard, an analyst who covers event processing for several years now, has posted a blog entitled: what's happening with event processing, observing that event processing is getting integrated with other areas such as: BPM, data integration and more. This is not a new phenomenon; in the EPIA book, we mention that among the event processing trends is moving from standalone to integrated even embedded, and this trend is evident with the evolution of event processing as a start-up universe, to having bigger software vendors as dominant forces. However - will event processing as technology going to disappear? I don't think so. There is common functionality among event processing utilization in various industries, applications, and hosting technologies, in all of them there are functions of filtering, event transformation, aggregation, pattern detection, and routing. It is not cost-effective to re-invent the wheel for each individual use (although there are variations). This is a similar situation to databases; database can be used for various reasons, and also be embedded with various other technologies and products (e.g. application servers, BPM, system management products, messaging systems - all use databases), while there are also variations, it is not the case that each of these areas develop database technology in an ad-hoc fashion. Thus, I see event processing continuing to evolve as a technology, and having both research and development activities that build generic event processing tools. From the business point of view, there will always be some niche for event processing stand-alone applications, but as Philip writes, most of the market will indeed be in the integrated area, this fact already reflects on the event processing technology in terms of need for standards, interoperability features, and ability to have embeddable collection of building blocks and components. More about this - later.

On Complexities and event processing

For those who read the title and grinned -- not again, discussion about what is the meaning of the term CEP, relax --- I explained in a previous posting entitled: "Is my cat CEP" , why such a discussion is futile, and I am typically consistent. BTW - when I have written that posting I did not have a cat, since than my daughter has adopted one, and he does not seem to me complex.

However, I would like to answer a more interesting question that somebody asked me recently -- what are the sources of complexity in event processing ?

In high school we have learned about "complex numbers", we liked this topic, since it was one of the most simple topics in the matriculation exam in Mathematics... Complex number is just a combination of two numbers, thus the complexity is in the structure. David Luckham also coined the term "complex events", where the complexity is also in the structure. However, there are more levels of complexity that may serve as a motivation to use COTS instead of hand-coding this functionality. What type of complexities can we observe beside the structural
complexity ?

Complexity derived from uncertainty:

• The applications specification is not known a priori and has to be discovered, example: fraud detection. This is related to the "pattern discovery" I have discussed in the previous posting.
• There are no reliable sources to obtain the desired events, or the events achieved can have uncertainty associated with them. This is a distinct complexity, since there may be the case where the application specification is well defined but the events cannot be obtained, and vice versa-- the patterns are unknown, but once discovered, the required events are easily available.

Complexity derived from connectivity:

• Producer related complexities --- semantic differences among various sources, problems of time synchronization among various sources etc..
• Consumer related complexities --- similar to the producer ones, these two are, of course, orthogonal to each other, and to all other complexities.
• Interoperability complexity where various processing elements are involved.

Complexity derived from functionality:

• Complex functions requirements -- e.g. complex patterns that may involve temporal, spatial, statistical operators and combinations of them.
• Complex topology of the event processing graph, with a lot of dependencies among the various agents, which creates a complexity in validation and control.
• Complex subscription / routing decisions.
  

Complexity derived from quantities

• High throughput of input events.
• High throughput of output events.
• High number of producers
• High number of consumers
• High number of event processing agents (imagine 1M agents in a single application)
• Requirement to maintain high amount of space for processing state.
  

Complexity derived from quality of service requirements:

• Hard real-time latency constraints.
• Compliance with QOS measurements such as threshold on average latency, threshold on percentage of events that don't comply with some latency constraint etc...
• High availability requirements.
  

Complexity derived from agility requirements

• Dynamic, frequent changes in the logic of the event processing
• Need for programming by various types of "semi-technical" people among the business users community...

I am sure that this list is not complete, but it provides some indication...

Of course, a single application may be the ultimate complex application of event processing and need ALL of these complexities, finding this application is, for sure, the dream of every researcher --- getting a lifetime of research challenges, but in reality different applications have different combinations of complexities. An application can be simple in all metrics, but have hard real time constraints, it can have very complex functionality, but no quality of service, or quantities issues. Another applications may need pattern discovery, but again the rest is simple, another combination can be relatively simple application, with complexity in quantity of producers and consumers and in semantic integration with all of them, and with the wonder of combinatorics, one can get to many more combinations....

More on complexities - later.

On event Driven BPEL

WS-BPEL has become a leading standard in the area of business process management, and since event processing is typically part of a bigger picture, it often interacts with business process management systems. One of the success factors is to make the environment ready for event processing, i.e. being able to get the right events, and being able to perform the right actions, while the event processing network has detected a situation (in one of the next postings I'll discuss how situation is related to the EPN world).

My student, Alex Kofman, has been doing recently a M.Sc. thesis looking from the BPEL perspective, he has determined what need to be done in order to extend BPEL to be "event driven", and proposed detailed modifications to the standard. This is intended to enable easy interaction between event processing system and BPM systems that plays as producer and consumer from the event processing viewpoint, while from the BPM viewpoint, input events can serve in decisions about orchestration, while output events can provide decisions external to the BPM based on the BPM state.

Here are some of the main points raised in this thesis:

1. Ability of BPM subscribe/unsubscribe to events.
2. Ability of BPM to publish events.
3. Ability to invalidate one or more processes as a result of an event (partially exists).
4. Ability to start a new process as a result of an event (partially exists)
5. Ability to invoke a task upon occurrence of an event
6. Adapt process execution upon occurrence of an event
7. Support the definition of an event processing pattern requested by the BPM engine, as a "callback" to an event processing system.
This in general will raise the level of abstraction and make the integration easier...

More about this project, later -- the thesis is now being written, Alex gave presentation about it last week in the department seminar, so this is a good time to start communicating these ideas.

The Babylon tower and event formats

Still in the USA, after the OMG meeting in the Washington DC area, I got to the Boston area, and now I am in Burlington, MA, visiting the (former) Aptsoft guys. When driving abroad I am renting a car with GPS (see above to determine which one), and it typically gets me to where I want - however, it is still not totally reliable, in the last day it confused me twice, one yesterday night to find the hotel, it told me to turn left, and meant in the next turn, not in the current turn, but it did not say so, thus, I found myself back on the I95, and had to turn around at the next exit, and try again. Today it sent me to some shopping center instead of the Aptsoft site, and after I ignored it and found it - the local people told me that the GPS maps have the numbers of the street in the wrong directions (starting from the other end) - well, sometimes the technology is fine, and the weakest link is the data it uses.

Talking about data, one of the topics that were discussed in the OMG meeting about standards is the topic of -- semantic/structural standards for events. I have used the term "Babylon tower" in one of the earliest postings in this Blog and meant that we have Babylon tower of languages - like the original tower who separated the languages. However, there is another Babylon tower that relates to event format - syntax and semantics, and here the problem is even harder, since there are multiple formats in multiple domains, nobody even made an inventory survey. One of the presenters said (and it is true in some cases) that 80 percent of the cost of building EP application is to set up the events from the sources and transform them to a processable format by composing adapters (hand coded, or by using transformation engines). This is some of the domains that need more investigation, and perhaps we need a meta-data standard here.

event processing as a part of a bigger picture

The picture is the view seen from "Midreshet Sde Boker" , where I have driven yesterday (3.5 hours drive each direction !) to attend the opening ceremony of the activity which my second daughter (out of four), Adi, is going to participate during the coming year. This is an activity in which high-school graduates postpone their army duty for one year, and invest it in volunteering activities, and learn leadership skills, in the Negev dessert, "in the middle of nowhere". From the Greek philosophy we have two competing views of the relations between the individual and the society -- the individual-centered, where society is aimed to serve the individuals, and society-centered, where individuals are part of society. Our universe is very much individual-centered, so it is a refreshing to find a group of young people who are ready to contribute one year from their life (yes, without being paid), to do volunteering work that help the society.

In moving back to "event processing", it is clear that event processing is not a detached thing, as the customer really wants a manufacturing system, or trading system, or billing system, and technology is only a way to achieve it, it is also true that there are typically no pure event-driven applications, but parts of applications that do event processing, which makes the interoperability, the ability to embed event processing within existing paradigms, and the ability of systems and applications to sense and respond, is a critical success factor.

Consequently, we'll see more and more event processing systems embedded in application integration middleware, rather than a stand-alone engines. More on that - later.