More on taxonomy and classification in event processing

Back to reviewing some recent Blogs on event processing, this time I am looking at Hans Gilde's Blog talking about the two types of event processing applications: detection oriented event processing and operations oriented event processing.

Detection Oriented Event Processing applications want to locate interesting information in a flow of data. Operations Oriented Event Processing applications want to take an action (which involves making decisions) based on incoming events.

This classification does not seem to be mutually exclusive, there are actually systems that locate interesting information in a flow of data, and after finding this interesting information are taking actions, which involves making decisions, based on that detection. Hans is right that these are two different things, one deals with the detection, the other deals with the action, and decision around the action.

Getting this classification further, let's look at two different issues: first -- what does the event processing system do (the "detection oriented") , and what are the results of the event processing system are used for (the "operation oriented").

Looking at the first aspect -- an event processing system can just filter and route event, it can also transform events (translate, enrich, aggregate etc...), and create derived events, it can detect predefined patterns, and it can detect patterns on the fly based on some reasoning, it can do part of these things, or all of them together, so an event processing platform is actually a tool box of all the above.

The event is then consumed by some consumer -- person, dashboard, system, and is (hopefully) used for something -- and this is the second aspect -- it may be used just for notification to a person, and what the person does or does not do is beyond the scope of the system, it can observe something and update a dashboard, it can diagnose something, it can do some real-time action based on decision, either in reactive mode or proactive mode (to mitigate some predicted state). This classification is based on a work we have done in IBM a few years ago by looking at applications in many industries, I have described this work in the past.

These two parts are typically decoupled. The event processing system gets events from some event producers, do some processing with it, and produce some events to the event consumers. There are certain set of techniques to do all these types of processing. The second type is done on what is called in the event processing architecture the event consumer. It gets event from an event processing system (or directly from an event producer if no processing is needed) and decides what do with it, the decision itself may apply business rules, and/or some quantitative analytic tools -- optimizers, simulators etc..., the decision can be totally automated ("autonomic"), semi-automated ("decision support system") or completely manual.

In essence, there are more and more systems that have complexity in both parts, thus the tool box to build this entire system is more extended, and while the execution is decoupled, the logic may be viewed in integrative way to build "decision processing network" in which "event processing network" is a subset, but it also includes business rules and quantitative analytics (and maybe some decision visualization tools).

I'll write more in the future abut the notion of "decision agents", introduced by James Taylor. This term reflect the atom of processing in both of the types that Hans is mentioning, however, in each of these type the agents are doing different things. I have mainly concentrated so far on the "event processing agents" type of processing in this Blog, but needs also to pay some attention to the other types of decision agents as well...

More - Later.

Review of Mark Palmer's event processing predictions for 2010

As an skeptic person, I have never really believed in astrology, but reading it from time to time, and somehow there is some correlation between years in which I had major changed in life, to years in which the astrology has predicted it, but this is not a scientific evidence, of course.

Anyway, the current prediction about this year came from Mark Palmer, the dynamic CEO of Streambase, who made nine predictions. Of course, his predictions were made from the viewpoint of his current role, but I thought that there is some value to review it. I'll say a few comments on them, one by one.

Prediction #1: 100+ Event Processing Applications Per Firm Milestone will be Passed.

This is something that I would like to see, and if will happen it will show that event processing is really part of the main stream computing. Personally I don't know any organization that is even close to 100+ applications, but I guess that Mark has somebody in mind. It will be even more interesting, if the 100+ applications will not be just thousand islands, but will also be connected, use heterogeneous event processing platforms...

Prediction #2: The "Deluge of Real-Time Data" Will Not Drive Event Processing Adoption

A refreshing view when it comes from Streambase which in the past over-hyped the mythical "event per second",

I totally agree, the main value of event processing seems to be the agility and reduction of total cost of ownership.

Prediction #3: Cognitive Physics Will Be as Important as Computing Physics

This is a challenging one, I was not familiar with the term cognitive physics, but with the help of all mighty Google, I found something about it on the Web. Anyway, Mark means that the way we communicate with computerized systems should align better with the way we think, getting to higher level abstractions, and consequently again to agility to implement new stuff. I totally agree that higher level languages is one of the most important directions for event processing, as a matter of fact, this is one of my major interest areas.

Prediction #4: Quantitative Thinking will Trump Traditional Thinking

This again has to do with higher level way to express quantitative thinking. Event processing is just one of various techniques to express quantitative thinking, the rest are - business rules and various analytics. Decision platforms will be combined of all. Event processing is certainly a player here, but not the only one.

Prediction #5: Software Stacks will Continue to Miss the Mark

There are systems in which event processing can be stand-alone island, but in many cases the producers and consumers for the events are coming from applications that have already been developed using the software stacks, thus, there will be a benefit to have good integration of event processing with these software stacks. As mark said customers are interested in solution to their business needs and not in technology, in some of the event processing systems I've seen, the main investment was not in the event processing system itself, but in its connectivity with the various producers and consumers. I agree that it is not cost-effective to buy an entire stack only to use event processing, but the same organizations that will have 100+ applications of event processing (prediction #1) are also those who already have BPM/messaging/application servers and other type of middleware and they are looking for good integration story, so this is not a black-or-white situation.

Prediction #6: A New Event-Based Software Stack will Emerge

The main claim here is that alternative stacks will emerge as combination of innovative technologies that comes from small companies. I've heard this idea before, in order to make it happen, more interoperability standards are needed, since the different parts of the stack need to talk to one another. I guess these standards will come, but from my limited experience, standards is a slow world.

Prediction #7: Stream-Based Platforms will Continue to Lead a Siege Against the $15B Data-Base Market

I am not sure what is the meaning of stream-based platforms, since the first predictions were made about event processing, and the body of this prediction also talks about event processing. Anyway, I agree that event processing is positioned as complementary to the business intelligence and general DBMS market. Analysts also claim that BI tools will have event processing capabilities (another software stack!), but still have some way to go until becoming main-stream there.

Prediction #8: Open Source CEP Won't Impact the Market, But Open Source Will

This is probably true for 2010, but might not be true for the longer future. When the event processing area will mature more and have its own standards, the event processing instance of MySQL may emerge.

Prediction #9: Event processing will Yield a Great New Software Powerhouse

Sometimes the emergence of new market can grow a software vendor from start-up to a medium or large software vendor, this has happened before, it might happen again - if indeed event processing will become a multi-billion business, I guess that part of it will be done by the current super powers in the software world, but some can go to growing companies. This will probably not happen in 2010, but I wish Mark success in his aspirations. Thinking big is a good practice, and a necessary but not sufficient condition for success.

On mobile phones and standards

Today I received a new mobile phone, since one of our family's mobile phone was broken, we did some shifting around, and it was my turn to get a new one. Starting to work with it was somewhat frustrating, I found myself in unknown territory, although it was produced by the same vendor of the old phone, they did not succeed to move the ringtones from my previous phone (although I purchased them in their store), since there is no standard for ringtones and each instrument represents it differently so they gave me a credit to get free ringtones from their store. They succeeded to move the list of my contact persons (there is a standard for that), but not the different groups defined for the contact person (no standard), the user interface is totally different, and requires some learning and more.

I have changed laptops several times in the last few years, and the transition was smooth, everything just worked -- no need to study anything new.

In our world of event processing we are not in better shape then in mobile phones, actually we are worse. In the event processing course that I have been teaching this semester, the students had a task to study six languages (each team has its own languages) and on Monday they will present the result of their experiments. I thought about it when struggling with the phone, if one of them was in the team that studied language X in the course, and tomorrow they find themselves working with another language, this might be somewhat confusing for them. In the course I am teaching our building block approach, which is valid for all languages. Working with standard oriented software is easier, we still have a challenge here.

Requiem to a small yellow cat

The small yellow cat adopted by my daughter a year ago passed away today. Eight days ago we realized he is sick and took him to the veterinary clinic, where they diagnosed that he was attacked by a violent virus, which his immune system could not handle. We were told that he'll probably last for 2 months or so, but it seems that this has been over-estimate. He had short life, only a little more than a year, but had a good life relative to his peers that live in the street. I have never been a cat fan, but over the last year he became part of the family, and was very attached to Hadas, my third daughter, he was very sad every time she has been away from home for a few days. For some strange reason the battery in my watch went dead at the same time he died, and my watch stopped. I have already replaced the battery, replacing a living creature will be more difficult, but we might adopt a new kitten after my daughter will get over...

More on intrapreneuring

This picture is taken from a NY Times article entitled "who says innovation belongs to the small" showing some system developed in IBM Research dealing with traffic analysis, this article was quoted in Mark Palmer's Blog saying that "Big companies can innovate if they act small". Mark is right, I have dealt with intrapreneuring for years, and recently even talked briefly about it in an internal talk I have given to an internal audience in IBM Haifa Research Lab about the internal recoginition we received. In my previous posting about intrapreneuring, I have quoted the original ten commandments that Pinchot, the person invented this term, has established; Mark mentions them in his Blog as well, however, recently I got Pinchot's follow-up book, "Intrapreneuring in action", and found out that he has somewhat changed the 10 commandments, some were changed, and some were changed order (assuming that the order says something on importance", so for all fans of intrapreneuring here are the revised 10 commandments:

1. Remember, it is easier to ask for forgiveness than for permission

2. Do any job needed to make your project work, regardless of your job description

3. Come to work each day willing to be fired

4. Recruit a strong team.

5. Ask for advice before resources.

6. Forget pride of authorship, spread credit widely

7. When you bend the rules, keep the best interests of the company and it customers in mind

8. Honor your sponsors

9. Underpromise and overdeliver

10. Be true to your goals, but realistic about ways to achieve them.

Some comments:

• The current first commandment has been promoted from being the 8th in the old list; my experience indeed shows that this is the most useful advice.
• "Recruit a strong team" is new, there was similar one with longer phrasing.
• "Ask for advice before resources" is also a new one
• "Forget pride of authorship" is also a new one, emphasizing the team game.
• #7 is also new one, though I have to say that "best interests" is a very subjective term.
• #9 is also new.
  
  

Some commandments that disappeared. but I thought they are useful:

• Work underground as long as you can – publicity triggers the corporate immune mechanism.
• Never bet on a race unless you are running in it.

And still, the immune mechanism of big corporates is not an easy line to cross, I have also been in a small company, it has other issues, however being entrepreneur is definitely easier than being intrapreneur.

On predictive, anticipative and proactive enterprise

As a skeptic person, I am not a big believer in astrology, but follows it from time to time; twice in my life the astrology predictive for the year for me was that there is going to be career shift in the next year, and twice it really happened, but this is still not a scientific evidence. Anyway, astrology is a kind of prediction source, but there are other sources like business intelligence tools. Richard Veryard in his Blog has recently written that the enterprise need to go beyond predictive, in the sense that the enterprise needs not only to passively predict the future, but also taking action in advance to the future, he used the term: anticipative for this kind of functionality.

First, I agree with the observation. One of the next frontiers is to go beyond predictive from BI point of view, as well as reactive from event processing point of view. I prefer the term proactive rather then anticipative to what Richard has described.

Some comments about the distinctions between predictive and proactive:

1. predictive typically relates to the strategic level --- watch trends and change the inventory policy; proactive may relate to operative --- anticipate that something in the operative level may happen (e.g. massive water leakage from the pipe infrastructure) and do something to mitigate it.
2. predictive typically collects data over long period of time, processes it in batch, and apply it periodically, or on request; proactive works in short-term, sometimes with time constraints.

Proactive may be seen as evolution both of BI and of event processing.

More about proactive -- later.

Are event processing technologies applicable for static data?

An interesting question was asked by Hans Gilde to one of my previous posts, "You focus on EP for moving data, but I wonder what you have found as fas as using the concepts for stored events".

The fact that I am concentrating on event processing is true, I would not equate "event" with "moving data", since event has some semantic meaning of something that happens, and has some time dimensions associated with it, so I would classify is as a kind of moving data, some moving data are not events.

There are indeed some benefits to use event processing concepts over stored data, I'll talk about one aspect which is the notions of pattern and context.

An expression like:

The vehicle moving in consistent direction south within the temporal context that starts with surveillance start and ends with vehicle stops partitioned by vehicle.

This is (more or less -- it should be done with a visual language rather than textual) a combination of pattern and context. This is somewhat easier than writing the same in SQL. There were attempts to mix SQL and patterns, but keeping the SQL semantics makes it somewhat complex. If we can use such a language, programming may be easier, this can be translated to SQL in the background.

There are some other benefits to use EP concepts for databases as well as for messaging systems, BRMS, and other related technologies.

On another matter: I am reading with interest Mark Palmer's nine predictions for the future of event processing, I promise a review after he'll finish to write all nine (now he is in #3).

More - Later.