On Complexity and use of COTS

My daughter has returned yesterday from a trip abroad, and brought me a present - since she knows that I have a collection of teas, she brought me another tea, unfortunately she does not really understand much in teas, so what I got as a present is a laxative tea. Well - I thanked her, and wish that I'll not have the need to use it ever.

Catching up in blog-land, I found an interesting posting by Mark Tsimelzon from Coral8,

entitled "complexity scorecard", the direction of defining indications for complexity is useful, however, this can be taken a step further, since the different indications belong to various topics, some of them about the complexity of functionality, and some on different measurements of scale, high availability and interface type, which are orthogonal issues. The assumption in Mark's scorecard is that if one has enough indications that it is cost-effective to use COTS for event processing instead of hand-code it, however, in some cases it is not the aggregation, since some of these indications can justify COTS even if most others don't exist (e.g. complex functionality), and sometimes the boundaries are not that clear, thus, there is a need to further refine the question "when is it cost-effective to use COTS" ? - thus, while the thinking is in the right direction, a scorecard like this looks somewhat simplistic. Do I have a better one ? -- not really, but will join the chase after it... more - later.

On Event Processing Networks - take one

Our BEA colleagues have once published an "Hello World" applications done with event processing network, since "Hello World" is the killer application of every programming paradigm -). Recently, a paper trying to define "Event Processing Networks" by Guy Sharon and myself, has been published in the on-line addition of IBM Systems Journal; anybody interested is welcome to read and comment. This is "take one", and we are still working on refining the terms, as well as looking on implementation and optimization issues in
heterogenous event processing networks - more later.

EPTS - last call for founding members

This is a reminder - we are going to close soon the list of "founding members" for EPTS - the original call has been posted in this Blog six weeks ago as well as in some other places. I am glad to report that by now 50 members have joined - organizations: vendors, analysts and customers; individuals - academic people, analysts, customers and independent consultants. The list is going to be closed, anybody (organization or individual) that want to join the list of "founding members" that will be reported in a press release in early June. If you still want to join - please follow the instructions in the original posting.

Mark your calendar -- the annual F2F meeting of EPTS will take place in Stamford, CT - call for contribution will be published soon.

On the three meanings of CEP

There is an old Jewish story on two people who had some dispute, and decided to go to the Rabbi and ask his opinion. The Rabbi listened to the first person and told him: you are right, then he listened to the second person and also told him: you are right. The Rabbi's assistant who has been present asked him: Rabbi, how can they both be right ? and received the obvious answer: you are also right.

Recently, a dispute between two opinionated persons - Hans Glide and Tim Bass has stormed the network, and somehow got also to my Blog. Somehow both of them understood that what I've written is consistent with their view - which encourages me to change career and become a Rabbi, but I think that there are some skills I lack - so anyway, I'll stay on event processing thinking.

While the dispute started around POSETS, the last posting by Tim Bass in the CEP forum re-focuses the discussion around - what is CEP ? so I'll stay with this topic for a while, since I think that there are (at least) three different interpretations of what CEP is - and this is a source of a lot of confusion. Thus I'll try to explain the three interpretations.

Interpretation One: the glossary interpretation -- complex event is the processing of complex events, where complex event is an abstraction or aggregation of events. According to this interpretation, a software function can be defined as CEP if it involves the processing of multiple events - typically collecting or matching some pattern among multiple events; The test for CEP according to this interpretation is typically support of a state that accumulate the relevant events - since they typically arrive over time. This definition does not say anything about the number of events, number of event types, whether they are totally ordered or not, or whether causality relation is supported or not - these are all attributes of specific implementations.

Interpretation Two: Event Processing = Complex Event Processing. This is a common practice in the commercial work to label all event processing functions as CEP. This spans from support of CEP (according to the interpretation one) and other event processing functions (such as: routing, enrichment, transformation) that don't satisfy the CEP test (since they are stateless and deal with a single event at a time). It can get to an absurd that some product that does not support CEP at all according to interpretation one, calls itself CEP. This is the most confusing interpretation, IMHO.

Interpretation Three: Complex event processing is event processing that has some complexity associated with it. According to Tim Bass - hidden causality and Markov Processes are vital for something to be defined as CEP. This really says that it CEP must involve uncertain events, causality that need to be discovered (by mining and other techniques), and the general usage of CEP is to predict something according to analysis of recent (past) events. According to Interpretation three, indeed the products that current products that call themselves CEP, do not satisfy this criteria, and thus are not CEP.

My opinion: as stated in the past, the term CEP has some inherent ambiguity, therefore I always thought it is confusing term. As far as my own taste in terminology - I prefer Interpretation One, saying that CEP is a subset of EP functions that deal with "complex events", it also seems that this is the closest to glossary definition. Interpretation two is confusing, as it turns CEP from a well-defined term to a marketing buzzword, and thus there is no test for what it is. Interpretation three is interesting, there are certainly applications that require prediction and various usages of stochastic processing and use of AI techniques (machine learning and others) in event processing. Hidden causalities is an important term, and I'll refer to it in another posting, since it has some pragmatic difficulty to obtain. However, I prefer to stick with the concept that CEP is processing of complex events, and not complex processing of events, and for interpretation one, we don't really need (necessarily) to apply AI techniques, this is just one type of application, there are a variety of applications that does not require it, and that detect predefined patterns on events is sufficient.

So the terminology that I personally prefer is:

Interpretation One = Complex Event Processing

Interpretation Two = Event Processing

Interpretation Three = Intelligent Event Processing.

More about event processing application variety

Back in Blog-land after a few days break due to dental surgery - not a very pleasant experience, but what somewhat smoothed by the service given by the clinic - to provide a reflexology during the surgery - here you can see Oded the reflexologist in action (there is some benefit to go to the most expensive clinic in the city...).

Anyway, I am hopefully recovering - and back online. I've read today in the EP Blogland, the posting of Chris Martins from Apama that described some deals done by Apama outside its "early adopter" type of application - algorithmic trading. This is consistent with previous postings in various Blogs that have talked about the variety of applications that can be assisted by event processing technologies. We in IBM have been facing engagements in variety of industries for variety of cases in this area, with variety of reasons and even different business justifications, we are still studying the variance to try to better classify functional and non-functional requirements to classes of applications. There is one benefit of "early adopters" that it serves as a proof of feasibility to all others, and the capital market applications certainly provided this function for event processing. There is one shortcoming in "early adopters" that it may bias the market in a way that is not really consistent with most applications, since people have tendendcy to employ inductive thinking (sometimes induction with N = 1). In our case the bias has been towards the "ultra low latency" - which is not typical to most applications in other areas. Software platforms that wish to be versatile can expect different requirements (or hacking their original software, and test the customer's patient). More on that - later.

On Posets and Red Herrings

Hans Glide, one of the knowledgeable persons in the EP area, has made in his Blog couple of postings about Posets, one of them claimed that the term Poset in EP is a red herring and of interest to mathematicians only, the other one says that causality graph and causality inference are not the same.

Let's make some assertions in this area, in order to clarify (or, god forbid, confuse...):

1. The EP applications space is not monolithic, different applications may have different requirements.
2. The issue of total order vs. partial order has been blown out of proportion, since it is not a major differentiator among products. Products who lack some functionality (see below) can somehow hack it in practice.
3. The main importance of causality relation in general is in the ability to trace all causal descendents of a certain event, or conversely to trace reasons for taking some action, this is a key requirement in anything that requires auditing. This does not cover the entire space of applications, and if the application does not require tracing and auditing then the notion of causality has little value to the application.
4. On the other hand, there are applications that manage scenarios, like software verification, simulation, games etc - in which causality is an important abstraction, as indeed reasoning is done on partially ordered set (directed acyclic graph) of events.
5. There are several types of orders and several types of causality relations.
6. Order may be according to the time that the arrives "happens in reality" (as reported by the source), or according to the time it is discovered by the event processing system (by getting to a system's API). In some cases the order does not play any role, in other cases, such as time series that are far enough, the sequence in the system is assumed to be good enough, and in other cases, the "reality" order has to be preserved -- how orders are preserved, if it is at all posssible, is a topic for another discussion.
7. Causality can be explicit or implicit. Explicit causality is meta-data in the event processing system, it exists because somebody put it there (maybe using mining techniques) saying that if Event E1 happens, than event E2 also happens, and we can assume that E2 happened even if there is no explicit indication that E2 happened. Implicit (inferred) causality is there since E2 is an output to some computation that E1 is an input to. In the class level it can be modelled, in the instance level, it is dynamic and created at run-time.

Closing statement: there is no - "one size fits all" in event processing. For each type of applications there are types of functionality that are more or less important to be supported. Better understanding of the existing types and mapping functionality to these types, is a work we are trying to achieve, by analyzing a significant number of use cases, as one of the EPTS tasks, by a large team of volunteers from all sectors (vendors, customers, academic people). We'll have more news on this work later this year.

On Streams and Events

The picture above is taken from a UCLA project that deals with multimedia stream systems. While the term "data streams" and later "event streams" that deals with continuous queries over structured data, have been introduced in the last decade in the database research community (with spin-off to products), the term "streams" has more general and more traditional meaning - referring to multimedia streams - video, voice, news etc... - which by nature belong to the family of unstructured data. In previous posting I discussed some of problems around "event stream processing", and around classification of event processing technologies. However, in this posting, I would like to point out that "stream processing" in its more traditional meaning is an important complementary technology to event processing.

First - the result of stream processing is in detection that an event has happened. Examples are: detection of vehicle's registration plate in automatic toll roads (we have in Israel one of these roads, there are other roads like this in Canada and Sweden - and maybe in more places), where the event "vehicle with registration plate X entered the highway in entrance Y in time T". This can be further processed (after correlating to the exit event) for billing purposes, but can also serve for security and other applications. In this case, from the event processing architectural view, the "stream processing" is done in a producer application, which generates events that are processed in the event processing system.

Second - the result of an event processing system can be an input to a stream, example: a game is being presented to the players as a video stream. Decisions made by the electornic player (or by the human player) can be assisted by an event processing system. The result of the decision can be movement of a player to a certain direction, and this is fed back to the video stream. In that case, the video stream is being processed in a consumer application, which gets event as an input.

Of course, a producer can also be a consumer, especially in games which are of iterative nature, thus an application is communicating with an event processing system in both side.

Since much of the events that happen in the universe is sensed thtough various unstructured media, the area of creating events out of multimedia streams, and embedding events to control the behavior of multimedia streams, will be one of the future major directions for the future, we can see some of this already hapening.