More on taxonomy and classification in event processing

Back to reviewing some recent Blogs on event processing, this time I am looking at Hans Gilde's Blog talking about the two types of event processing applications: detection oriented event processing and operations oriented event processing.

Detection Oriented Event Processing applications want to locate interesting information in a flow of data. Operations Oriented Event Processing applications want to take an action (which involves making decisions) based on incoming events.

This classification does not seem to be mutually exclusive, there are actually systems that locate interesting information in a flow of data, and after finding this interesting information are taking actions, which involves making decisions, based on that detection. Hans is right that these are two different things, one deals with the detection, the other deals with the action, and decision around the action.

Getting this classification further, let's look at two different issues: first -- what does the event processing system do (the "detection oriented") , and what are the results of the event processing system are used for (the "operation oriented").

Looking at the first aspect -- an event processing system can just filter and route event, it can also transform events (translate, enrich, aggregate etc...), and create derived events, it can detect predefined patterns, and it can detect patterns on the fly based on some reasoning, it can do part of these things, or all of them together, so an event processing platform is actually a tool box of all the above.

The event is then consumed by some consumer -- person, dashboard, system, and is (hopefully) used for something -- and this is the second aspect -- it may be used just for notification to a person, and what the person does or does not do is beyond the scope of the system, it can observe something and update a dashboard, it can diagnose something, it can do some real-time action based on decision, either in reactive mode or proactive mode (to mitigate some predicted state). This classification is based on a work we have done in IBM a few years ago by looking at applications in many industries, I have described this work in the past.

These two parts are typically decoupled. The event processing system gets events from some event producers, do some processing with it, and produce some events to the event consumers. There are certain set of techniques to do all these types of processing. The second type is done on what is called in the event processing architecture the event consumer. It gets event from an event processing system (or directly from an event producer if no processing is needed) and decides what do with it, the decision itself may apply business rules, and/or some quantitative analytic tools -- optimizers, simulators etc..., the decision can be totally automated ("autonomic"), semi-automated ("decision support system") or completely manual.

In essence, there are more and more systems that have complexity in both parts, thus the tool box to build this entire system is more extended, and while the execution is decoupled, the logic may be viewed in integrative way to build "decision processing network" in which "event processing network" is a subset, but it also includes business rules and quantitative analytics (and maybe some decision visualization tools).

I'll write more in the future abut the notion of "decision agents", introduced by James Taylor. This term reflect the atom of processing in both of the types that Hans is mentioning, however, in each of these type the agents are doing different things. I have mainly concentrated so far on the "event processing agents" type of processing in this Blog, but needs also to pay some attention to the other types of decision agents as well...

More - Later.

Review of Mark Palmer's event processing predictions for 2010

As an skeptic person, I have never really believed in astrology, but reading it from time to time, and somehow there is some correlation between years in which I had major changed in life, to years in which the astrology has predicted it, but this is not a scientific evidence, of course.

Anyway, the current prediction about this year came from Mark Palmer, the dynamic CEO of Streambase, who made nine predictions. Of course, his predictions were made from the viewpoint of his current role, but I thought that there is some value to review it. I'll say a few comments on them, one by one.

Prediction #1: 100+ Event Processing Applications Per Firm Milestone will be Passed.

This is something that I would like to see, and if will happen it will show that event processing is really part of the main stream computing. Personally I don't know any organization that is even close to 100+ applications, but I guess that Mark has somebody in mind. It will be even more interesting, if the 100+ applications will not be just thousand islands, but will also be connected, use heterogeneous event processing platforms...

Prediction #2: The "Deluge of Real-Time Data" Will Not Drive Event Processing Adoption

A refreshing view when it comes from Streambase which in the past over-hyped the mythical "event per second",

I totally agree, the main value of event processing seems to be the agility and reduction of total cost of ownership.

Prediction #3: Cognitive Physics Will Be as Important as Computing Physics

This is a challenging one, I was not familiar with the term cognitive physics, but with the help of all mighty Google, I found something about it on the Web. Anyway, Mark means that the way we communicate with computerized systems should align better with the way we think, getting to higher level abstractions, and consequently again to agility to implement new stuff. I totally agree that higher level languages is one of the most important directions for event processing, as a matter of fact, this is one of my major interest areas.

Prediction #4: Quantitative Thinking will Trump Traditional Thinking

This again has to do with higher level way to express quantitative thinking. Event processing is just one of various techniques to express quantitative thinking, the rest are - business rules and various analytics. Decision platforms will be combined of all. Event processing is certainly a player here, but not the only one.

Prediction #5: Software Stacks will Continue to Miss the Mark

There are systems in which event processing can be stand-alone island, but in many cases the producers and consumers for the events are coming from applications that have already been developed using the software stacks, thus, there will be a benefit to have good integration of event processing with these software stacks. As mark said customers are interested in solution to their business needs and not in technology, in some of the event processing systems I've seen, the main investment was not in the event processing system itself, but in its connectivity with the various producers and consumers. I agree that it is not cost-effective to buy an entire stack only to use event processing, but the same organizations that will have 100+ applications of event processing (prediction #1) are also those who already have BPM/messaging/application servers and other type of middleware and they are looking for good integration story, so this is not a black-or-white situation.

Prediction #6: A New Event-Based Software Stack will Emerge

The main claim here is that alternative stacks will emerge as combination of innovative technologies that comes from small companies. I've heard this idea before, in order to make it happen, more interoperability standards are needed, since the different parts of the stack need to talk to one another. I guess these standards will come, but from my limited experience, standards is a slow world.

Prediction #7: Stream-Based Platforms will Continue to Lead a Siege Against the $15B Data-Base Market

I am not sure what is the meaning of stream-based platforms, since the first predictions were made about event processing, and the body of this prediction also talks about event processing. Anyway, I agree that event processing is positioned as complementary to the business intelligence and general DBMS market. Analysts also claim that BI tools will have event processing capabilities (another software stack!), but still have some way to go until becoming main-stream there.

Prediction #8: Open Source CEP Won't Impact the Market, But Open Source Will

This is probably true for 2010, but might not be true for the longer future. When the event processing area will mature more and have its own standards, the event processing instance of MySQL may emerge.

Prediction #9: Event processing will Yield a Great New Software Powerhouse

Sometimes the emergence of new market can grow a software vendor from start-up to a medium or large software vendor, this has happened before, it might happen again - if indeed event processing will become a multi-billion business, I guess that part of it will be done by the current super powers in the software world, but some can go to growing companies. This will probably not happen in 2010, but I wish Mark success in his aspirations. Thinking big is a good practice, and a necessary but not sufficient condition for success.