Quantum Leap -- take II

This morning was a sunny Saturday after a few rainy ones, and along with many other people, I went out with my family to the nature... We live in Haifa, which besides its beaches and beautiful view of the bay, has also a close by big nature reserve called "Carmel forrests", not really a Forrest in global terms, but has many nice hiking trails, 15 minutes drive from home. Here are some of the flowers we watched today... good to take a break sometimes..

As a follow up to my previous posting on quantum leap, here are some more insights, we in IBM Haifa Research Lab have signed up to look at the "next generation of event processing", and are working on this topic, I may present a tutorial about our findings in DEBS 2009, if accepted.

Here are some initial insights:

• Like in databases, there need to be a formal model that will have wide acceptance (over time) to enable the quantum leap, since acceptance provides a critical mass of work directed to the same direction. Our belief is that the "event processing network" model is the one, but it still lacks solid formal basis.
• Besides this -- there are four areas that will show in the future significant developments, if they will be done on the basis of the model -- it can provide a coherent play. The pyramid below shows the four :
  

• Platform: While the first generation of event processing is the "engine" land, we are starting to see movement for platforms which will provide shared services (e.g. - global state management, routing, load balancing, security, high availability...) and a possibly heterogeneous collection of event processing agents will run in these platforms. There may be platforms with various orientations -- grid platforms, database oriented platforms, messaging oriented platforms, streaming (data flow) oriented platforms to name a few. The platforms may be an "event processing platforms" or platforms with wider functions (e.g. event processing agents and other decision agents). Some analysts are talking about -- extreme transaction processing (XTP) and context-oriented platforms, maybe the platform will mix some of all of the above. Like the area of application servers in enterprise computing, the platform orientation is one of the facets of the next generations.
• Engineering: The engineering progress is not really considered as revulsion, but they are required to enable the higher layers to work in reality. This is the equivalent in other areas to query optimization, tuning, configuration, scheduling, load balancing, parallel programming assignments and various of other systems related topics. The relational databases became widespread only after the vendors succeeded to get the engineering parts right, so advancement in this area is critical.
• Functional: The functionality that products have today is just the start, more functionality will be supported, maybe even substantially more. Some directions: the "intelligent event processing" direction -- looking at discovery of unknown pattern and prediction of future events, adding more context information - like geo-spatial, getting better temporal handling; probably much more.
• Usability: Here probably will be much of the quantum leap -- getting the abstraction levels higher. Hierarchy of events, and causality, advocated by David Luckham, are really abstractions. However, there are more than just abstractions from the implementations up, there also need to be abstractions from the user thinking down. Instead of trying to visualize and abstract out the implementation model, the opposite direction will be to have the abstractions in the users domain of thinking and translate them (perhaps not 1-1) to implementation.

The quantum leap will occur with a coherent combination of all these aspects. There may be some new vendors which will offer next generations as their first generation, since they are liberated from supporting legacy (and may be acquired by larger vendors) , and there are existing vendors which are going into some of this in an incremental way....

EPTS will attempt to contribute to the thinking about next quantum leap by the work in its working groups; we also saw in the last EPTS event processing symposium that the use cases working group has presented a variety of use cases, which cover broad range of applications types and requirements, this will be one vehicle to determine requirements. Other working groups will contribute in the various areas. In May 2010 we'll do a major summit of industry and academic people (Dagstuhl Seminar), EPTS members will get a more detailed note about it.

More - Later.