This morning was a sunny Saturday after a few rainy ones, and along with many other people, I went out with my family to the nature... We live in Haifa, which besides its beaches and beautiful view of the bay, has also a close by big nature reserve called "Carmel forrests", not really a Forrest in global terms, but has many nice hiking trails, 15 minutes drive from home. Here are some of the flowers we watched today... good to take a break sometimes..
As a follow up to my previous posting on quantum leap, here are some more insights, we in IBM Haifa Research Lab have signed up to look at the "next generation of event processing", and are working on this topic, I may present a tutorial about our findings in DEBS 2009, if accepted.Here are some initial insights:
- Like in databases, there need to be a formal model that will have wide acceptance (over time) to enable the quantum leap, since acceptance provides a critical mass of work directed to the same direction. Our belief is that the "event processing network" model is the one, but it still lacks solid formal basis.
- Besides this -- there are four areas that will show in the future significant developments, if they will be done on the basis of the model -- it can provide a coherent play. The pyramid below shows the four :
- Platform: While the first generation of event processing is the "engine" land, we are starting to see movement for platforms which will provide shared services (e.g. - global state management, routing, load balancing, security, high availability...) and a possibly heterogeneous collection of event processing agents will run in these platforms. There may be platforms with various orientations -- grid platforms, database oriented platforms, messaging oriented platforms, streaming (data flow) oriented platforms to name a few. The platforms may be an "event processing platforms" or platforms with wider functions (e.g. event processing agents and other decision agents). Some analysts are talking about -- extreme transaction processing (XTP) and context-oriented platforms, maybe the platform will mix some of all of the above. Like the area of application servers in enterprise computing, the platform orientation is one of the facets of the next generations.
- Engineering: The engineering progress is not really considered as revulsion, but they are required to enable the higher layers to work in reality. This is the equivalent in other areas to query optimization, tuning, configuration, scheduling, load balancing, parallel programming assignments and various of other systems related topics. The relational databases became widespread only after the vendors succeeded to get the engineering parts right, so advancement in this area is critical.
- Functional: The functionality that products have today is just the start, more functionality will be supported, maybe even substantially more. Some directions: the "intelligent event processing" direction -- looking at discovery of unknown pattern and prediction of future events, adding more context information - like geo-spatial, getting better temporal handling; probably much more.
- Usability: Here probably will be much of the quantum leap -- getting the abstraction levels higher. Hierarchy of events, and causality, advocated by David Luckham, are really abstractions. However, there are more than just abstractions from the implementations up, there also need to be abstractions from the user thinking down. Instead of trying to visualize and abstract out the implementation model, the opposite direction will be to have the abstractions in the users domain of thinking and translate them (perhaps not 1-1) to implementation.
The quantum leap will occur with a coherent combination of all these aspects. There may be some new vendors which will offer next generations as their first generation, since they are liberated from supporting legacy (and may be acquired by larger vendors) , and there are existing vendors which are going into some of this in an incremental way....
EPTS will attempt to contribute to the thinking about next quantum leap by the work in its working groups; we also saw in the last EPTS event processing symposium that the use cases working group has presented a variety of use cases, which cover broad range of applications types and requirements, this will be one vehicle to determine requirements. Other working groups will contribute in the various areas. In May 2010 we'll do a major summit of industry and academic people (Dagstuhl Seminar), EPTS members will get a more detailed note about it.
More - Later.
David Luckham posted on the complexevents site the question: Is there a commercial need for quantum leap in CEP products. David has also a continuation article that discusses event hierarchy abstractions as a quantum leap possibility. Before answering David's question let's me make some observations. Early in my career, 33 years ago, I have been a programmer in the Israeli Air-Force (the Air-Force was in opinion that letting me flying aircraft's will be too dangerous for the public safety...) and we have been early adopters of IMS, which has been relatively new, had severe performance issues, and needed a lot of manual tuning, and did not really work as advertised. IMS was actually a second generation database, it has a predecessor called DL/I and still used the DL/I language. IMS was a huge improvement over file systems that we have used before in level of abstractions, concurrency control, and many other utilities, yet it had many issues that have been resolved over time. The relational databases are actually the third generation of database systems and it also had a rough childhood, until query optimizations have been matured; The relational database has been a disruptive technology, and also had its own childhood problems, until query optimization have been better understood.
Back to event processing --- I assume that event processing products of 2019 will be totally different from those of 2009, the questions are:
- Is there going to be a "disruptive technology" as the relational database has been in the database area, OR "just" gradual evolution will occur.
- What will drive the progress to next generations ?
What will trigger the next generations ?
- Customer requirements that require substantial change
- Competitive pressure
- Disruptive technology
- (more reasons) ?
I'll leave these questions for now as a food for thought and will discuss them in subsequent postings.
Yesterday we kicked-off the "languages analysis working group" of EPTS. The EPTS members have been part of the approval process, and a result of a going on brainstorming in the community that started in the event processing symposium in September 2008, we are kicking off six working-groups: glossary, interoperability analysis, meta-modeling, reference architectures, use cases and languages analysis. These working groups will be the main activity of EPTS in the year 2009, and will hopefully result in better understanding of the area, and better understanding of the potential standard play. In the working group we have around dozen people from the vendor, academic and customer community. I have decided to concentrate my own technical contribution to EPTS (besides the substantial time I invest in facilitating the overall activities) in the language analysis area - since this is the closest to my interest area. My partner in moderating this working group is Dr. Jon Riecke, an experienced programming language researcher, who works for Aleri. It is a challenge to have a diversified team with various opinions, which is a symptom of the challenge to achieve an event processing language standard somewhere in the future.
What is this working group chartered to do ? we have committed on two deliverables, the first
The first one is -- [the exact terminology is still under discussion] --- a [semantic model] that abstracts out the functions (and may be non-functional annotation) of event processing languages, without getting to the question of programming style (SQL or not SQL). We shall look at existing commercial languages as well as languages that have been developed in the research community and try to abstract out, another source will be a feedback from the use cases working group that works in parallel that will attempt to discover more requirements from the use cases analysis.
The second one is discussion and recommendation (possibly with alternatives) about the road- map to standards in the event processing languages area (can we aspire to one standard or multiple standards ? maybe a standard in the semantic "meta" level only, or we may determine to table the issue of standards for a certain period), frankly, I don't have a clue what the conclusion here will be.
We'll submit a tutorial proposal for DEBS 2009, if accepted - we shall present an interim report of the work in July.
I'll report about it as we'll progress.
STAC, as cited in the popular Blog of Tim Bass, has determined that the correct name that should be used for event processing products is EPP (Event Processing Platforms) rather than CEP. I actually like the term EPP, actually I used this term before, so I should have copyrighted this name...
However, EPP should be used in the right meaning.
In event processing software there are two different things:
- Platforms that provide the "programming in the large" -- indeed a container in which different types of functionality can be plugged in.
- EPA implementation Software - that performs the actual event processing work - e.g. pattern matching, enrichment, filtering etc... This is the "programming in the small" (I called it "event processing engines", but not convinced that this is the best name)
In the EPTS glossary terminology, Platforms implement "event processing networks", while the other type implement various types of event processing agents.
These are not the same, there are vendors who provide platforms, but use other software to implement agents For example - BEA provided a platform, and used Esper for various functions, if I am not mistaken this is also true for Event Zero, IBM's Infosphere Streams is also a platform -- all are indeed platforms. Some products provide both the EPN platforms and various EPA implementations, some provide just the EPA implementation and runs on various platforms (or as a centralized stand alone engine).
So, while I agree that the EPN implementations are platforms, I am not sure that the EPA implementations are also platforms, and we probably may need a different name (engines ?, not sure)...
And one sentence about the term CEP. As I have written several times, I am not a big fan of this term at all, I am consistently talking about event processing and not about complex event processing as the name of the discipline that this Blog covers. However, this reminds me that once I have been a member in a Hebrew technical terminology committee, and one of the terms that came for discussion has been "real-time", for some strange reason, in Hebrew it was translated literally as "true time", and when it came to write the official glossary endorsed by the Israeli Academy of Hebrew Language, their representative who knew Hebrew linguistics, but not computer science, insisted the the Hebrew word should be a true translation of "real time", giving a long talk about "real numbers" and other real stuff. I argued that --- from linguistic point of view he is probably right, but, the scientifically wrong name is already well-known in the industry, and decision on another name would not be accepted by the public. After long discussion he agreed to include my wrong version as an alias to his true version. You can guess which of the two is still being used for "real-time" in Hebrew.
The moral of this story is that it may be too late to change names, since the name CEP has been accepted in the industry for any type of event processing system, whether or not it is scientifically accurate, and as somebody said once -- resistance is futile...
I'll continue to use "event processing", will use "event processing platform" for a platform, and still looking for a term for the "EPA implementation" (engines or otherwise). But -- my guess is that the people that use CEP to denote any type of EP will continue doing it, since this name may already penetrated to the ground. More - Later.
