Saturday, January 17, 2009

On Distribution and parallelism in Event Processing

This picture, taken from the site of Nature Reviews as part of an article about "parallel processing in mammalian retina", illustrates that structures like the human body distribute the functions it needs to perform, and performs many of them in parallel and by specialized systems.

Getting to event processing, the producers and consumers of event processing can be distributed, as events can come to many sources, and situations may be consumed by many sinks. The first generation of event processing was mostly centralized in processing, the centralization has been twofold: functional centralization using an monolithic engine that performs all processing fucntions, and location centralization, this engine runs on a single server.

Today I'll concentrate on the second aspect of centralization, there are various reasons to decentralize the processing, one is to do some of the activities closer to the producers or consumers, example: if a producer produces events, where only 1% is relevant to the defined event processing, and it can be done by independent filtering that does not depend on other events, then it will be more efficient that the filtering will take place at or close to the consumer site, and thus eliminate the unnecessary network traffic.

Another reason to distribute the functionality is the scalability aspect, which is really an old idea to "divide and conquer" problems. The challenge is how to do a "good" partition. First there is a need to define what a "good" partition is, i.e. looking at it as an optimization problem, what is the goal function, then solving it is a function of the topology, semantics and behavior of a particular application which can be dynamic.

IBM has recently released the first version of WBEXS (Websphere Business Event Exterme Scale) and made a statement of direction for another product: Infostreme Streams
both are aimed to handle scalability by distribution in different environments. While details about IBM products you can obtain from the appropriate people in IBM, we in the IBM Haifa Research lab are working on related topics, we have exposed initial results in DEBS 2008, in the fast abstract session introducing the statification approach.The project has substantially advanced since that time, and I'll discuss it further in future Blogs (well - I need to go over the Blog and list all the topics I promised to discuss later and have not done so yet...).

More - later

Thursday, January 15, 2009

Welcome the ILOG team to the blue giant

Recently, the acquisition of ILOG by the blue giant (IBM, the company who pays my salary) has been completed.
It was interesting to read the InformationWeek article which had event processing in the title of this article.

ILOG is, of course, well known for its business rules product ( I had some history in this area in my past...).

There are various relationships between business rules and event processing technologies, and there are also various opinions about the various relationships.... I have blogged a year ago about this issue.

While I looked at business rules from EP perspective, Some ILOG guys also investigated the opposite direction, in the last EPTS meeting we had met Pierre-Henry Clouin, who blogged about it , prior to the EPTS meeting. Now we can continue the short discussion we had in Stamford .

Another ILOG guy who had a guest posting about EP issues is Changhai Ke.

Joining a big corporate is certainly a cultural change (shock for some), but also opens some horizons, so I wish Pierre-Henry, Changhai and the rest of the ILOG team a smooth transition and we'll certainly meet in one of the blue corridors.... Welcome on board.

Monday, January 12, 2009

Some footnotes to recent blogs

Today's experience relates to the remote control which opens the doors of my car (well - mine does not have a panic button), for some strange reason I have noticed that it resides on the floor of my office decomposed to basic components, one screw has broken or something... I picked up all parts, held them tightly together and went straight to the contact person of the leasing company in IBM Haifa Labs, he looked at the broken device and said -- these days we are not allowed to replace them if they work, so he took the following working tools (see below) -

and now I have a remote control with cello tape around it, it turns out that the atmosphere of "reducing expenses" in our leasing company gets to the tiny details.... I also heard that some almost forgotten professions like shoemaker who knows how to fix shoes got alive these days...

Anyway today I am in reactive mode -- since several of the recent Blogs deserve some footnotes:

A footnote to the Blog with the provocative title: SOA is dead; long live services
by Anne Thomas Mannes (along with the many other responses)- I once heard a good talk about the many interpretations of SOA, which means different things to different persons, so it may be dead for some, and alive for others... anyway, good ideas live more than marketing TLAs that come and go with fashions. As an analog, The TLA CEP may survive or change with other fashionable term that will have some other blend of technologies, however, the more interesting thing is not the marketing term, but the substance behind it. If it solid and have value to customers, it will survive and prosper, and I believe that event processing (as a discipline - see: my previous posting on EP as a discipline) is one.

A footnote to Mark Palmer's Blog -
Speaking on event processing - it was interesting to read Streambase's report by Mark Palmer stating that Q4 was the best quarter ever for Streambase, which is a pure play event processing vendor. While this may or may not be an indication for a more general phenomenon, as I've started this posting, everybody is trying to reduce costs these days, and one of the ways to do it , automate processes that are event-driven by nature (e.g.automated exception detection and handling), getting alerts on cases where there is a potential expenses leakage (auditing), compliance with regulations that corporates see as "tax" on their operations, and would like to invest as little resources as they can, straight through processing and other activities associated with cost reduction are event-driven in nature, and thus can benefit from use event processing technologies, thus, the positive correlation between troubled times and growth in the use of event processing software may not be surprising, again, this may or may not explain Streambase's report, I am not familiar with the details.

A foontnote to Marc Adler's Blog - Marc cites a study about the influence of Blogs on purchasing. I am amazed every time to see the power of Blogs... from my personal experience, I am getting a lot of private communication based on my Blog, including some surprising offers, will write about it one day; one the RFPs that I got from our sales team to advice on, was traced by them to be copied from one of the area Blogs... customers started to see Blogs as authority, and this can be of course dangerous since not everybody who Blogs about something is really an authority on the area he or she Blogs on and get into the trap --- that's all for now -more alter.