On Maturity

After a week break -- lot of work, some taking care of family health issues - I am back catching up on the community Blogs. One thing that I saw has been a debate whether CEP is not mature enough. I think that both sides have a point, depending on what "maturity" means. I'll make a distinction between: maturity of an application, maturity of a product, and maturity of a technical area.

Maturity of an application is measured in the fact that an application is working with relatively low amount of faults, and satisfy its functional and non-functional goals. Typically there is some time for stabilizing the application followed by a maturity time, and in some point the application becomes an obsolete in something, which requires to initiate the next generation.

Maturity of a software product is materialized in the fact that the product is working, being used in applications, and has relative few bugs, and customers trust it to rely on it.

Maturity of a technical area has to do with the level of understanding of this area by customers, amount of utilization of the area relative to potential, clear concepts and standard support.

In the CEP area we certainly have mature applications, we also have some maturity in the products of the first generation, but we are somewhat far from the maturity of the entire area.

On CEP agent in EPN

This logo shows that we are not alone in the world of TLAs - there are other who use this acronyms, once I have made a list of acronyms of EDA which has shown that this is not a very good choice of acronym, but an acronym these days is context-sensitive. Anyway -- talking about "event processing networks", which we now try to clean its definitions - we have talked about SEP (simple event processing) agents, MEP (mediated event processing), CEP (complex event processing) and IEP (Intelligent event processing). The names can still change, I am not happy about any of them, but it seem to be relatively intuitive, and can be explained to non-technical people quite easily. Anyway, this time I'll concentrate on what is CEP agent. CEP agent have two basic phases:

1. Detect patterns over multiple events (these can be either multiple events of the same type, or multiple events from multiple event types - some are doing distinction here, but I don't see the rationale behind it) - this creates a collection of event-instances that satisfy the patterns.

2. Derive events as a function of the collection of events created in phase 1.

Some comments:

• There can be cases in which one of the two capabilities is used in a degenerated form. The pattern can be trivial (take all events without any pattern), but still a derivation (e.g. count, calculate the average of a certain attribute etc...) is required; on the other hand - the pattern can be complex, however the derivation is trivial (just a concatenation of all participating events).
• The term for the result of phase 1 - can be called "composite event" or "complex events", the two are not exactly synonyms, according to the glossary, as composite event is a syntactic term that is created by operators from primitive events, whereas complex event is a semantic term that includes aggregation of events that may not be obtained from patterns - however, the definition of phase 1 results is consistent with both.
• A CEP agent always lives within a context (temporal, spatial, semantic) -- most common type of context is a time-window, but the concept of context is much wider.
• The semantics of a CEP agent need to be fine-tuned in order to pick the instances of events participating in patterns relative to other possibilities which require to define policies - I'll talk more about semantics fine-tuning is a later postings.
• The result of a CEP agent is one or more derived events (consistent with other types of agents).

On Event Processing Research Challenges

This picture has been taken a year ago on the stairs to the old church in Schloss Dagstuhl, in that meeting we had people both from academia and from industry discussing the state of the art and future of event processing. I have returned to the conclusion of the Dagstuhl Seminar done by my colleague Peter Niblett from IBM recently, after my return to the IBM Research Division from spending several years in the product organization, I was asked about challenges to the research community, as seen by the product development community (or the industry in general), since we had a session about it in Dagstuhl, in which people from the industry expressed their opinions about the same questions, I just had to take it as a basis of a presentation in this area.

In Dagstuhl four major areas has been put as challenge to the Research community (adding my own interpretation and comments)

1. Event Processing Algebra and Meta-Language: Like the database area in the pre-relational era, the first (and probably second) generations of event processing are "engineering based", various vendors are building implementations based on the use cases they see and their innovative ideas, bringing to the table, not only a variety of languages, but also a variety of (typically - implicit) conceptual models behind this implementation. One of the indication for a maturity of an area is the existence of an agreed upon conceptual model. The relational model has done it for databases, the browser concept has done it for the internet, now we need the same for event processing. David Luckham's concept of EPN/EPA is a possible starting point, but more work is needed on this - the challenge is still there; after constructing the "relational model" for event processing we also need the "SQL" (not necessarily SQL extension - but this is a possibility) for it - in term of meta-language that describes the model and can be mapped to various implementation.

2. Software Engineering issues: This is a challenge from a different perspective. Event processing and Event Driven Architectures impose different thinking about computing. We are programmed to think in a certain way in all programming language, and the event based programming is somewhat different paradigm. Even the basic principle of decoupled asynchronous processing is something that is not easy to digest f0r people. We need software engineering tools, methodologies and best practices.

3. Implementation optimization: If we have done the analog of SQL, another analog come to mind - database tuning and query optimization. There are many optimization issues especially when the event processing applications are distributed and may have various QOS requirements - parallel processing, acceleration of hardware, inter-operability, support in illites in general - all are challenges. The goal function in this optimization is not unique, while in some systems it is maximal throughput, in other it may be scalability in number of rules/queries/patterns (there are applications in which the throughput is measured in millions, and others in which the number of rules, the number of producers, or the number of consumers is measured in millions...), the goal function can be a combination of multiple criteria.

4. Variety of "small issues" - examples: uncertain events, out-of-order events, retention and vacuuming of events etc....

It will be interested to look at recent research project to see how much progress has been done on these, and if the list should be modified after a year that has passed. The major research event of the EP community this year is DEBS 2008 , the program has not been published yet, and it will be interesting to compare the accepted papers with this list (I'll do it when surveying the conference itself in July). We also intend to do in the EPTS event processing symposium in September a session with academic people about it - stay tuned for further notice. I am also thinking about a follow-up Dagstuhl Seminar, may be in 2009 or 2010 - to re-discuss event processing in perspective. More - later.

On Complexity and use of COTS

My daughter has returned yesterday from a trip abroad, and brought me a present - since she knows that I have a collection of teas, she brought me another tea, unfortunately she does not really understand much in teas, so what I got as a present is a laxative tea. Well - I thanked her, and wish that I'll not have the need to use it ever.

Catching up in blog-land, I found an interesting posting by Mark Tsimelzon from Coral8,

entitled "complexity scorecard", the direction of defining indications for complexity is useful, however, this can be taken a step further, since the different indications belong to various topics, some of them about the complexity of functionality, and some on different measurements of scale, high availability and interface type, which are orthogonal issues. The assumption in Mark's scorecard is that if one has enough indications that it is cost-effective to use COTS for event processing instead of hand-code it, however, in some cases it is not the aggregation, since some of these indications can justify COTS even if most others don't exist (e.g. complex functionality), and sometimes the boundaries are not that clear, thus, there is a need to further refine the question "when is it cost-effective to use COTS" ? - thus, while the thinking is in the right direction, a scorecard like this looks somewhat simplistic. Do I have a better one ? -- not really, but will join the chase after it... more - later.

On Event Processing Networks - take one

Our BEA colleagues have once published an "Hello World" applications done with event processing network, since "Hello World" is the killer application of every programming paradigm -). Recently, a paper trying to define "Event Processing Networks" by Guy Sharon and myself, has been published in the on-line addition of IBM Systems Journal; anybody interested is welcome to read and comment. This is "take one", and we are still working on refining the terms, as well as looking on implementation and optimization issues in
heterogenous event processing networks - more later.

EPTS - last call for founding members

This is a reminder - we are going to close soon the list of "founding members" for EPTS - the original call has been posted in this Blog six weeks ago as well as in some other places. I am glad to report that by now 50 members have joined - organizations: vendors, analysts and customers; individuals - academic people, analysts, customers and independent consultants. The list is going to be closed, anybody (organization or individual) that want to join the list of "founding members" that will be reported in a press release in early June. If you still want to join - please follow the instructions in the original posting.

Mark your calendar -- the annual F2F meeting of EPTS will take place in Stamford, CT - call for contribution will be published soon.

On the three meanings of CEP

There is an old Jewish story on two people who had some dispute, and decided to go to the Rabbi and ask his opinion. The Rabbi listened to the first person and told him: you are right, then he listened to the second person and also told him: you are right. The Rabbi's assistant who has been present asked him: Rabbi, how can they both be right ? and received the obvious answer: you are also right.

Recently, a dispute between two opinionated persons - Hans Glide and Tim Bass has stormed the network, and somehow got also to my Blog. Somehow both of them understood that what I've written is consistent with their view - which encourages me to change career and become a Rabbi, but I think that there are some skills I lack - so anyway, I'll stay on event processing thinking.

While the dispute started around POSETS, the last posting by Tim Bass in the CEP forum re-focuses the discussion around - what is CEP ? so I'll stay with this topic for a while, since I think that there are (at least) three different interpretations of what CEP is - and this is a source of a lot of confusion. Thus I'll try to explain the three interpretations.

Interpretation One: the glossary interpretation -- complex event is the processing of complex events, where complex event is an abstraction or aggregation of events. According to this interpretation, a software function can be defined as CEP if it involves the processing of multiple events - typically collecting or matching some pattern among multiple events; The test for CEP according to this interpretation is typically support of a state that accumulate the relevant events - since they typically arrive over time. This definition does not say anything about the number of events, number of event types, whether they are totally ordered or not, or whether causality relation is supported or not - these are all attributes of specific implementations.

Interpretation Two: Event Processing = Complex Event Processing. This is a common practice in the commercial work to label all event processing functions as CEP. This spans from support of CEP (according to the interpretation one) and other event processing functions (such as: routing, enrichment, transformation) that don't satisfy the CEP test (since they are stateless and deal with a single event at a time). It can get to an absurd that some product that does not support CEP at all according to interpretation one, calls itself CEP. This is the most confusing interpretation, IMHO.

Interpretation Three: Complex event processing is event processing that has some complexity associated with it. According to Tim Bass - hidden causality and Markov Processes are vital for something to be defined as CEP. This really says that it CEP must involve uncertain events, causality that need to be discovered (by mining and other techniques), and the general usage of CEP is to predict something according to analysis of recent (past) events. According to Interpretation three, indeed the products that current products that call themselves CEP, do not satisfy this criteria, and thus are not CEP.

My opinion: as stated in the past, the term CEP has some inherent ambiguity, therefore I always thought it is confusing term. As far as my own taste in terminology - I prefer Interpretation One, saying that CEP is a subset of EP functions that deal with "complex events", it also seems that this is the closest to glossary definition. Interpretation two is confusing, as it turns CEP from a well-defined term to a marketing buzzword, and thus there is no test for what it is. Interpretation three is interesting, there are certainly applications that require prediction and various usages of stochastic processing and use of AI techniques (machine learning and others) in event processing. Hidden causalities is an important term, and I'll refer to it in another posting, since it has some pragmatic difficulty to obtain. However, I prefer to stick with the concept that CEP is processing of complex events, and not complex processing of events, and for interpretation one, we don't really need (necessarily) to apply AI techniques, this is just one type of application, there are a variety of applications that does not require it, and that detect predefined patterns on events is sufficient.

So the terminology that I personally prefer is:

Interpretation One = Complex Event Processing

Interpretation Two = Event Processing

Interpretation Three = Intelligent Event Processing.