On Distribution and parallelism in Event Processing

This picture, taken from the site of Nature Reviews as part of an article about "parallel processing in mammalian retina", illustrates that structures like the human body distribute the functions it needs to perform, and performs many of them in parallel and by specialized systems.

Getting to event processing, the producers and consumers of event processing can be distributed, as events can come to many sources, and situations may be consumed by many sinks. The first generation of event processing was mostly centralized in processing, the centralization has been twofold: functional centralization using an monolithic engine that performs all processing fucntions, and location centralization, this engine runs on a single server.

Today I'll concentrate on the second aspect of centralization, there are various reasons to decentralize the processing, one is to do some of the activities closer to the producers or consumers, example: if a producer produces events, where only 1% is relevant to the defined event processing, and it can be done by independent filtering that does not depend on other events, then it will be more efficient that the filtering will take place at or close to the consumer site, and thus eliminate the unnecessary network traffic.

Another reason to distribute the functionality is the scalability aspect, which is really an old idea to "divide and conquer" problems. The challenge is how to do a "good" partition. First there is a need to define what a "good" partition is, i.e. looking at it as an optimization problem, what is the goal function, then solving it is a function of the topology, semantics and behavior of a particular application which can be dynamic.

IBM has recently released the first version of WBEXS (Websphere Business Event Exterme Scale) and made a statement of direction for another product: Infostreme Streams
both are aimed to handle scalability by distribution in different environments. While details about IBM products you can obtain from the appropriate people in IBM, we in the IBM Haifa Research lab are working on related topics, we have exposed initial results in DEBS 2008, in the fast abstract session introducing the statification approach.The project has substantially advanced since that time, and I'll discuss it further in future Blogs (well - I need to go over the Blog and list all the topics I promised to discuss later and have not done so yet...).

More - later

On Event Processing Web Sites

In the last few days there is a debate in the community around the launch of Marco's wiki entitled "event processing wiki" - one opinion said - we already have a portal - David Luckham's site that contains a discussion forum and there will be more visibility to postings on a more known site. The other opinion is "let a thousands flowers bloom" an indication for the vitality of a community is in multiple sites, wikis, blogs etc.

You may ask yourself - what is the Rabbi's picture doing here? do I think that this is a religious question, or have I become religious in my old age. The question is - none of the above - this discussion reminded me an old Jewish story (nothing to do with Rabbi Malkior, an ex-minister and member of the Israeli parlaiment, whose picture you can see) :

An husband and wife came in front of the Rabbi so that he can judge in a dispute among them. The Rabbi listens to the wife and says - you are right, then the husband told the Rabbi his claimes and the Rabbi said - you are right too. After they have gone away - the Rabbi's assistant asks him - how is it possible that they both have been right, they told you contradicting things - and the Rabbi answered patiently - you are also right, of course.

This, more or less, reflects my opinion on the discussion. I totally agree that we, as a community, should not have a central control, and communities are distributed by nature and not centralized. I also agree that posting on a known an popular portal has more visibility then on not-yet-established one, so I'll leave it to the "market forces".

Talking about portals - there is another interesting EP portal now, dedicated mostly to scientific papers in this discipline, and include references to many articles: event based.org

Bottom line: At least cross-reference...