Wednesday, August 27, 2008

On event processing as a discipline and some subsets

Every parent knows that discipline is very important from very early age (e.g. make the child realize that "sleeping time" is not negotiable), the term "academic discipline" also derived from a body of knowledge related to a certain issue that a student should learn, and indeed one of the interpretation of this word in the Webster dictionary is a field of study. In the DEBS 2008 conference, a disscussion has started whether "event processing" is a discipline by its own right, or subset of another discipline. My answer is that "event processing" is beginning its steps as a discipline, but has to go some way in order to become a full-fledged discipline. One of the interesting questions is -- since event processing applications in one or other form have existed in the past, why it did not develop until now to become a discipline, what has changed recently? my own answer is that earlier work have done ad-hoc event processing for a special purpose, while recently there is an attempt to handle "event processing" as an area systematically. To give a couple of examples:

(1). Processing of time series (see example) have existed long time ago, and also used as inspiration to those in the database community who looked at data stream management. Time series processing assumes that events arrive in fixed intervals, and typically the processing is statistical operations - like aggregation, exponential smoothing, regression, trend analysis and other stuff. The people who has dealt with this area were not interested in the more general picture of event processing (e.g. event processing when event arrive in a sporadic way and not in fixed intervals, processing event at a time, and not set etc..).

(2). Event correlation in network and system management - this has been around in the last 15-20 years (see the Computerworld article) - here again, there is a very specific sub-case, aimed to cope with "event storm" - a network or system administrator is facing a lot of events which are symptoms for problems (e.g. time-out of device can be a symptom for the fact that a router is offline) and "correlate" the symptoms to their "root cause", the problem. There is a notion of looking at patterns - but typically very limited patterns (e.g. conjunction of events over a time interval), while this has been core to system and network management, people who dealt in this area have never investigated event processing in the larger sense (e.g. looking at additional patterns), and this area has also not spawned the event processing discipline. In a previous posting about the term event correlation I have discussed the fact that while in the network and system management this term is well-defined, people sometimes use it in different contexts in an ambiguous way, and thus this is one of the confusing terms (a common misconception is that event correlation is alias to CEP) . I prefer to leave it in its network and system management meaning, and use more precise term for other interpretations.

There are more subsets of event processing, these have been just examples; the difference in the recent years is that the fact that several works both in academia and industry started to look at the bigger picture of "event processing" - what is it, what are all possible utilizations, what are the required functions, what are the non-functional requirements, what techniques have been used in other disciplines (AI, databases, control theory, distributed computing, simulation...) that can help solving this issue. This has not been done before; as said, there is a long way to go on all these topics, but this has been true for each discipline in its beginning, and I see positive momentum.

Last but not least -- I was recently asked if the discipline is called "event processing" or "complex event processing" are they aliases ? -- my own preference is to use "event processing", discipline names typically consist of two words (information retrieval, data management, artificial intelligence, image processing, computer vision...). As noted before, I accept the glossary definition of "Complex Event Processing" according to which denotes a subset of the larger event processing picture. More later -- this issue will be discussed in the EPTS 4th event processing symposium, and it will be interesting to hear the various opinions on that topic.


Hans said...

I wonder how all the new commercial EP products have affected the development of EP as a discipline.

On one hand, many people would be researching in this area without an immediate commercial application. But for firms like IBM, I can't help but think that the demonstration of a market has influenced the direction of research.

Opher Etzion said...

Hello Hans.

short answer: there are cross-influences.

I'll write a specific posting to answer your question soon.