On terminology again

While the EPTS glossary team is about to produce additional version of the glossary,  I received by Email a question about terminology from a student investigating this area, whether "event driven architecture" and "event processing"  (both titles of books) are indeed the same thing  


The answer is -- not really, but the difference is simple:   


Event Driven Architecture is an architecture in which there are event producers, event processors, and event consumers, with the following principles:  1). All players are decoupled;  2). All the communication between players is that they send events to one another:  3). All communication between players is asynchronous. 


Event Processing is a type of processing that takes as input one or more events and produce as output one or more event applying functions such as: filtering, aggregation, transformation and pattern matching.   


The concepts are related in the sense that it is common to implement event processing on top of event driven architecture, but it is not tightly coupled:
Event Driven Architecture can implement  pub/sub systems that are doing routing only and do not do any processing of events;
Event Processing can be be implemented in synchronous mode and not on top of  EDA.

Where does the edBPM term come from?

Writing earlier today about a German friend,  another German friend, Rainer von Ammon has written on the complexevents forum about the source of the term edBPM,  Rainer is also the one who has drawn the nice illustration above showing the reference model, Rainer is the person promoting this term, and organized several workshops around the concept of pairing event processing and BPM technologies.     Rainer wrote that the term came from Gartner and quoted me.   This is almost true,  the original term that Gartner used is "event-based BPM"  and I have slightly modified it to "event-driven BPM" when I asked Rainer to write a value about it in the Database encyclopedia (I have been the editor of the event processing related terms).  
Here is the Gartner's original slide from 2005.  

This is the original slide,  source:  "Event-driven applications make event-driven businesses work better", a presentation by Roy Schulte from Gartner in 2005. 


As you see Gartner classified the event processing functions into: simple event processing, mediated event processing, event-based EPM, and complex event processing.    Roy Schulte has later realized that "event-based BPM" is orthogonal dimension to the three others, and changed the positioning.     So this is the source for all history lovers. 

On event processing as a service

While working on the Website of the EPIA book, we asked the language owners to provide downloadable version of the product implementing their language. I was asked by some of the language owners if instead they can provide instead a possibility to provide their software as a service and let the readers run it on their servers. My answer was positive, and we'll see couple of such examples (one already there, one is coming up).

The book's website is just a resource for readers who wish to study languages, but this brought me to a thought about event processing as a service in general.

Some of the reasons for doing it is to gain the benefits of cloud computing in terms of scalability,

I've recently came across some material about activeinsights which seems to be a new Israeli company developing open source "event stream processing" in the cloud (well - I have some terminology comments to them, but this is not the main point) that advocate the use of event processing in the cloud to cope with scalability issues. Using SAAS model for event processing can give rise to some interesting cost models, that are either related to the input (amount of input event processed) or the output (amount of situations detected by the event processing service, with some cost per situation, or amount of aggregated/transformed derived events) which ties the cost directly to the benefit. One of the barriers of using event processing as a service is lack of standards especially for interoperability, which does not enable just to connect and run, but requires substantial investment in writing adapters in a proprietary way. I assume that we'll see more of that, when the cost/benefit model will be clarified.

There are more interactions between cloud computing and event processing, such as the use of event processing as part of the cloud infrastructure, but this deserves a separate discussion.

Event Processing Platform (EPP) --- yes, but...

STAC, as cited in the popular Blog of Tim Bass, has determined that the correct name that should be used for event processing products is EPP (Event Processing Platforms) rather than CEP. I actually like the term EPP, actually I used this term before, so I should have copyrighted this name...

However, EPP should be used in the right meaning.

In event processing software there are two different things:

• Platforms that provide the "programming in the large" -- indeed a container in which different types of functionality can be plugged in.
• EPA implementation Software - that performs the actual event processing work - e.g. pattern matching, enrichment, filtering etc... This is the "programming in the small" (I called it "event processing engines", but not convinced that this is the best name)
  

In the EPTS glossary terminology, Platforms implement "event processing networks", while the other type implement various types of event processing agents.

These are not the same, there are vendors who provide platforms, but use other software to implement agents For example - BEA provided a platform, and used Esper for various functions, if I am not mistaken this is also true for Event Zero, IBM's Infosphere Streams is also a platform -- all are indeed platforms. Some products provide both the EPN platforms and various EPA implementations, some provide just the EPA implementation and runs on various platforms (or as a centralized stand alone engine).

So, while I agree that the EPN implementations are platforms, I am not sure that the EPA implementations are also platforms, and we probably may need a different name (engines ?, not sure)...

And one sentence about the term CEP. As I have written several times, I am not a big fan of this term at all, I am consistently talking about event processing and not about complex event processing as the name of the discipline that this Blog covers. However, this reminds me that once I have been a member in a Hebrew technical terminology committee, and one of the terms that came for discussion has been "real-time", for some strange reason, in Hebrew it was translated literally as "true time", and when it came to write the official glossary endorsed by the Israeli Academy of Hebrew Language, their representative who knew Hebrew linguistics, but not computer science, insisted the the Hebrew word should be a true translation of "real time", giving a long talk about "real numbers" and other real stuff. I argued that --- from linguistic point of view he is probably right, but, the scientifically wrong name is already well-known in the industry, and decision on another name would not be accepted by the public. After long discussion he agreed to include my wrong version as an alias to his true version. You can guess which of the two is still being used for "real-time" in Hebrew.
The moral of this story is that it may be too late to change names, since the name CEP has been accepted in the industry for any type of event processing system, whether or not it is scientifically accurate, and as somebody said once -- resistance is futile...

I'll continue to use "event processing", will use "event processing platform" for a platform, and still looking for a term for the "EPA implementation" (engines or otherwise). But -- my guess is that the people that use CEP to denote any type of EP will continue doing it, since this name may already penetrated to the ground. More - Later.

On Event Pattern Detection vs. Event Pattern Discovery

This drawing, in various forms, has been used by us for many years to illustrate the notion of pattern, actually in the PowerPoint version it is animated, and the geometric shapes are keep moving. The term pattern is a bit overloaded in event processing, as noted my DEBS 2008 tutorial on this topic, but this illustration refers to the pattern which shows some combination of events, to be more accurate it is a predicate on the event history that if evaluated to the value of
"true" something should happen. This illustration was created by Tali Yazkar-Haham from IBM Haifa Research Lab as an exercise in a presentation course, and was used in dozens of presentations ever since (including presentations of some people outside IBM who typically forgot to give credit to the source).

Paul Vincent, in a continuous debate with Tim Bass, on the complex events forum, has written about "detection of new instance" and "detection of new type". While these terms make sense, I prefer not to overload the term detection and use the terms event pattern detection and event pattern discovery.

Pattern detection deals with detection that a predefined pattern has happened. This is what illustrated in the picture above. Some example can be: a patient is hooked up to a heartbeat monitor, and the physician is pre-setting a pattern "the heartbeat is monotonically increasing within 10 minutes, and the amount of increase is more than 30 during that period". This is actually a predicate over a part of the event history of a single source and type (other examples can involve multiple sources and types, but the principle is the same).



So event pattern detection is defined as detection that a predefined patterns has occurred.
This is equivalent to what Paul called: Pattern instance Detection.
In contrast, when we talk about event pattern discovery we mean that the pattern is not known in advance, and the pattern discovery function determines what is the pattern. The legend says that Archimedes discovered his famous laws about floating bodies when sitting in the bathtub and shouted: Eureka (this illustration was taken from the homepage of a company who has the word Eureka in its name, again animated in the source).
A pattern can be discovered by machine learning techniques using decision trees, statistical modeling, Bayesian Networks and numerous other methods. At the end when a pattern is discovered then it also need to be detected in reality; there are also cases in which there is a continuous detection since the patterns are changing after a short time.

Getting back to the previous example about the heartbeat, it may be the case that this pattern has bot been set by a physician, instead it was detected by some method that has looked at past events and found out that this pattern has some significance.

Most people thinking about "complex event processing" are actually talking about pattern detection, regardless of whether the patterns were composed by a human or discovered by machine learning. The illustration at the top of this page illustrates what people typically mean. As stated in the past, I don't want to get into the meaning of TLAs, and leave it to my colleagues who are doing marketing. Thus the term that I used "event pattern detection" is the more accurate one.

Another observation about the difference is that event pattern detection can be applied as COTS -- a user can use such a product, compose some patterns, hook it up to event sources, and get the pattern to be detected.

On the other hand --- while there are many tools that can help in the event pattern discovery, we cannot hook it up to event source and tell it: discover all. There is a need to do some formal modeling of the system, kind of patterns that are sought etc... In other words, this is not something that a typical developer or business analyst can do, since it requires some expertise,

It is getting late - so I'll finish at this point and return to this topic at some later point.

Is my cat CEP or not?

One of my favorite cinema artists is Mel Brooks, and one of the great films of Mel Brooks is known as "silent movie" (see flyer above). To those who don't know or don't remember, silent movie is a film about doing a silent movie, and most of the plot involves around Mel Brooks and his gang going to famous actors asking them to participate in a silent movie, they agree and this sums up their participation, since their voice is not heard; there is one exception - when asking the world famous mime Marcel Marceau to be in the film he replies, "Non", the only spoken word in the entire film, since he talked, he did not participate in the silent movie.

So here is my dilemma -- recently the blog-land is full of postings on the topic "is X CEP?" - personally I don't find that this debate leads anywhere, so my inclination is not to participate in this discussion, so I'll contribute my part by explaining why I don't find this discussion interesting.

There are different reasons why I don't find it interesting to spend time on these discussions are:

1. I don't see that the results of this debate are important to the reality.


2. One can debate terms forever - but this is not useful, there is a better way.


3. This debate will not converge anyway, the same arguments are being repeated and it does not enrich the reader's intellect to read them again.

Let's start with the reality -- what the current vendors attempt to do is to build generic products that will enable to develop various sorts of applications in the area of event processing. This is different from the past, where event processing has been done in hard-wired, hand-coded, ad-hoc fashion to build a single application. While any single application may be challenging, the challenges in building generic products that will be a cost-effective alternative to hand-coding, are somewhat different from building ad-hoc applications. The content of these products is determined by several factors - not by the boundaries between "simple" and "complex" - there are multiple dimensions of complexity, and different applications may exhibit different type of complexity (functionality, topology, scalability, event types)... No matter where we'll put the border between the "simple" and "complex", a generic product will have to support both sides of the border. Bottom line -- the importance is ROI for the customers and not whether something is called "complex" or not.

Debate over terms: when we have done the first event processing symposium, in March 2006, the conference started by nine presentations about the "state of the art", and a very quick observation has been that each of us has invented his own set of terms. Consequently we have decided that the first community effort will be to produce a glossary edited by our non-vendor colleagues, to reduce bias. They have collected input from many sources with different opinions, sometimes strong opinions (it is amazing that people are so caring about terms). The glossary has been published recently and its announcement was endorsed by many people in the community. I am sure that nobody is happy about all terms, but as a compromise among all opinions, the editors have done a pretty good job.

Personally I don't have strong feeling towards terminology, so the glossary definition is a possible one, there are others as well, there is no absolute truth here, but - I think that is good to have a precise definition of terms that are accepted by large community vs. continue debating on terminology forever, even if I personally would have defined it otherwise. Among the possible interpretations, the editors chose to define CEP: Computing that performs operations on complex events, where complex event is defined as:An event that is an abstraction of other events called its members. As far as I am concerned, this is what CEP is.

Last but not least -- If somebody decides to insist on his own (different) definition of a term T, and judge everything according to this own definition, then obviously, if a term T have two different defintions, then X may be T according to one definition, and not T according to the other definition, when the two sides stick to their definitions, then no one will convince the other anyway, and arguments will still repeat themselves. Actually, I have not learned any new insight from this discussion. Have I mentioned that the arguments repeat themselves? in case that I have not done it , I'll mention that the arguments repeat themselves.

Bottom line -- is my cat CEP ? well - I don't have a cat(I wonder if Roger King who has written the famous article "my cat is object oriented" had a cat).

There are plenty of more useful topics to Blog about - so no more on that issue.

On relationships among: derived event, composite event, complex event and situation

This is the water park "Luna Gal" where I am planned to spend the late afternoon with my children (it is my wife's turn to have a business trip this week) as part of a fun day for employees, meanwhile I am again at my (air-conditioned) office with the morning coffee and the Blog. One of my somewhat neglected obligations that I am trying to catch up these days are writing some terms to the encyclopedia of database systems, since we are in the glossary era, I would like to answer a FAQ question about the relationship between four terms: derived event, composite event, complex event and situation. The first three are glossary terms, the last is not, but is being used a lot in the community jargon. Tim Bass has recently posted in his Blog a simple solution model which is originated in the experimental psychology research.


The following illustration clarifies the relationships:

Derived event is an event that is created as a result of a computational derivation process as a function of one or more events.

Complex event is an event that is an abstraction of other events.

Composite event consists of collection of events that satisfy some pattern.

These are not the glossary definitions but they are explanations of the definitions that will enable to determine the relationships. As we can see not every derived event is a complex event - a derived event can be just an enrichment of a single event, and as such not a complex event, on the other hand, complex event can be created by explicit grouping of events and not by computational derivation, thus a complex event may not be a derived event. The relationships between derived events and complex event are - partial overlapping.

On the other hand, composite event is a subset of both complex event (since it is a collection of event and as such an abstraction) and derived (since it is derived by a computational process). Interestingly composite event is not the intersection between derived event and complex event, since the intersection also include scalar derived events that aggregate the raw events but don't keep the collection of events. Thus composite event is a subset of the intersection.

How does situation fit in ? - well, situation is a different beast, it is not some syntactic operation on event, but it is a semantic abstraction that denotes -- something that requires reaction. An event of any type may designate or approximate a situation, for example: fraud suspicion is the situation and a certain pattern is an indication (either exact or approximate) to the occurence of this situation, when there is a determination that a situation has occurred - a derived event may be issued to designate this situation, and the pattern is the condition (again - exact or approximate) for the situation. Note that an event pattern may have some certainty measure (e.g. probability) for the fact whether it designates a situation , not all cases are deterministic.

Does every derive event represent a situation? --- not really. A derived event can be used as an interim event or part of dissemination system and not have semantic role of situation. The same is true for all other event types.

Last but not least --- Paul Vincent in his Blog reported on a tutorial he has given about the ETPS glossary (useful presentation, I'll ask permission to borrow it). He asks a question - whether event inheritence is derived event ? well - let's check the definition of derived -- creating an event as a result of computational process as a function of one or more events. Does it describe the inheritance process ? -- IMHO it does, thus I think that inherited event is a type of derived event, though the derivation process is somewhat different from the derivation process in other cases. more - later.

On DEBS 2008

Still in Rome, still in the hotel's bar since my room does not get enough wireless signal. The DEBS conference site is just next door to the church in which the famous Moses status is located.

I have also finished tonight my participation in DEBS (it goes one more day, but I will be heading for home tomorrow according to the plan). The DEBS conference (disregarding some organizational challenges) have gone well, while it has been a scientific conference, there has been a relatively big participation from industry - I have seen there people from vednors as well as customers, DEBS has been originally the home of the pub/sub and messaging community, however they have successfully extended it to include various other event processing "sub-communities" like the CEP guys, the streaming guys, people from modeling area etc..

I have contributed my part by providing a tutorial about EP patterns - which is a work in progress for building a meta-language which abstracts the semantics behind event processing functionality.

Since it is late tonight I'll defer discussions that I had in the conference to later posts, will just make a note that I owe everybody discussion on three points:

• Boundaries and territories among disciplines -- in the wake of discussion with Dieter Gawlick from Oracle.
• The term "event processing agents" - some discussions, and Blog comments from Paul Vincent
• The notion of pattern in event processing.

Overall - very good conference, an opportunity to meet a lot of friends, and (I am ashamed to admit) my first visit in Rome ever.

On the three meanings of CEP

There is an old Jewish story on two people who had some dispute, and decided to go to the Rabbi and ask his opinion. The Rabbi listened to the first person and told him: you are right, then he listened to the second person and also told him: you are right. The Rabbi's assistant who has been present asked him: Rabbi, how can they both be right ? and received the obvious answer: you are also right.

Recently, a dispute between two opinionated persons - Hans Glide and Tim Bass has stormed the network, and somehow got also to my Blog. Somehow both of them understood that what I've written is consistent with their view - which encourages me to change career and become a Rabbi, but I think that there are some skills I lack - so anyway, I'll stay on event processing thinking.

While the dispute started around POSETS, the last posting by Tim Bass in the CEP forum re-focuses the discussion around - what is CEP ? so I'll stay with this topic for a while, since I think that there are (at least) three different interpretations of what CEP is - and this is a source of a lot of confusion. Thus I'll try to explain the three interpretations.

Interpretation One: the glossary interpretation -- complex event is the processing of complex events, where complex event is an abstraction or aggregation of events. According to this interpretation, a software function can be defined as CEP if it involves the processing of multiple events - typically collecting or matching some pattern among multiple events; The test for CEP according to this interpretation is typically support of a state that accumulate the relevant events - since they typically arrive over time. This definition does not say anything about the number of events, number of event types, whether they are totally ordered or not, or whether causality relation is supported or not - these are all attributes of specific implementations.

Interpretation Two: Event Processing = Complex Event Processing. This is a common practice in the commercial work to label all event processing functions as CEP. This spans from support of CEP (according to the interpretation one) and other event processing functions (such as: routing, enrichment, transformation) that don't satisfy the CEP test (since they are stateless and deal with a single event at a time). It can get to an absurd that some product that does not support CEP at all according to interpretation one, calls itself CEP. This is the most confusing interpretation, IMHO.

Interpretation Three: Complex event processing is event processing that has some complexity associated with it. According to Tim Bass - hidden causality and Markov Processes are vital for something to be defined as CEP. This really says that it CEP must involve uncertain events, causality that need to be discovered (by mining and other techniques), and the general usage of CEP is to predict something according to analysis of recent (past) events. According to Interpretation three, indeed the products that current products that call themselves CEP, do not satisfy this criteria, and thus are not CEP.

My opinion: as stated in the past, the term CEP has some inherent ambiguity, therefore I always thought it is confusing term. As far as my own taste in terminology - I prefer Interpretation One, saying that CEP is a subset of EP functions that deal with "complex events", it also seems that this is the closest to glossary definition. Interpretation two is confusing, as it turns CEP from a well-defined term to a marketing buzzword, and thus there is no test for what it is. Interpretation three is interesting, there are certainly applications that require prediction and various usages of stochastic processing and use of AI techniques (machine learning and others) in event processing. Hidden causalities is an important term, and I'll refer to it in another posting, since it has some pragmatic difficulty to obtain. However, I prefer to stick with the concept that CEP is processing of complex events, and not complex processing of events, and for interpretation one, we don't really need (necessarily) to apply AI techniques, this is just one type of application, there are a variety of applications that does not require it, and that detect predefined patterns on events is sufficient.

So the terminology that I personally prefer is:

Interpretation One = Complex Event Processing

Interpretation Two = Event Processing

Interpretation Three = Intelligent Event Processing.

Why I prefer to use "event processing" with prefix, infix or suffix - a subjective tour of acronyms

Recently there has been more discussions about terms and acronyms, I am not sure that this is so important issue to spend much time on, but before moving to a more interesting points, I would like to provide some personal thoughts about acronyms in this area.


First, as you can see from the Blog's name, I prefer the term "event processing" with any prefix, infix, or suffix. The reason is that I view it as a name of a discipline and not as trend. Disciplines typically consist of two words: signal processing, information retrieval, machine learning, software engineering etc.. although there are exceptions. Three letter acronyms AKA TLA, are typically not names of core disciplines but of other things - protocols, architectures, trends etc..


Historically, when the first "event processing symposium" (which created EPTS) has been established we needed a name - the original founders were - David Luckham, Roy Schulte, Mark Palmer (from Progress Software) and myself. David, of course, thought that CEP is an appropriate name for the discipline, while Mark proposed ESP - "Event Stream Processing" since he did not like the word "complex" (read further about it). Roy and mysrelf proposed to take the part that both agree "event processing". Both David and Mark were not completely happy, but agreed, thus we advanced with the name "event processing symposium" and used "event processing" ever since.



Getting back to history - I have prefered to use the name "active technologies" being a veteran of the active database community, and although the autonomic computing community adopted the "active" term and had conferences named "active middleware services", this name actually did not get into the main stream, David Luckham used the term "complex event processing" in his famous book that used the term. The term "complex event processing" has ambigious meaning - one interpretation is that this is processing of complex events, where complex event is an event that consists of more than one event (analog to complex object), the other interpretation is that this is complex processing of events. I have started to use the term CEP in 2004 to differentiate such functionaity from "event correlation" in system management since there has been some confusion in IBM around this terms. I also made a modest contribution to get the name CEP known by giving a tutorial in ICWS in July 2004, attended by many people, whose common denominator has been tht they have not heard this term before. Anyhow - there are two school of thoughts around CEP


Interpretation one ("the monolithic approach") : CEP = EP, everything is a subset of CEP.
Interpreation two ("the layered approach") : EP is a collection of technologies, whereas CEP is one of them (a link in the chain). Some people takes the first interpretation, saying that "simple" event processing (whether it is simple event or simple processing) is a subset of complex event processing, the rational behind it that if an engine is capable of doing complex things it is surely capable of doing simple things. Interpreation two comes from Roy Schulte (Gartner) who introduced in December 2005 the following slide:

In this slide Roy Schulte talks about four types of processing (later he realized that the BPM one is of another category) - simple event processing (filter and route), mediate event processing (transform and enrich) and complex event processing (statefull pattern detector). This is consistent with a market view since there are products that do only simple event processing (messaging), other products who do mediated event processing (ESB) and CEP as the next layer as a stateful engine. I think that this approach is liked by those who are putting CEP on top of existing middleware, while the first ("monolithic") approach is liked by those who have stand-alone CEP engine. Anyway - the existence of this two approaches, and the fact that people may not understand that the other person is taking the second interpretation is causing a confusion.

Next acronym has been "event stream processing", the term "data stream manager" has been coined in Stanford in a similar meaning, but with SQL API, and continuing with other academic projects, and some descendent products (Coral8 is a descendent of the Stanford project). When Progress Software acquired Apama, Mark Palmer looked for an alternative word for CEP, since he was in the opinion that customers don't like anything labelled "complex", thus, he borowed the term "stream" although Apama's API is not SQL, and has not much to do with the academic stream projects and introduced the ESP term "Event Stream Processing" (which was dropped later). In response, David Luckham published an article to defend the "complex" word, starting with the words: "some people, I'm told, get scared when they hear the word complex, as in complex event processing.... start with the basic question, is life simple ? most people when asked about it will truthfully answer no...." and the rest you can read yourself. It seems that David has won this battle -- all vendors (including the SQL oriented ones) at some point or another have positioned themselves as CEP vendors, which also created some objections - by people who thought that it is important to diffrentiate between ESP and CEP, some saying that ESP is a subset of CEP, and some that these are completely different focus areas - as I have written before, there are many ways to define subsets of EP functionality, and I did not find any evidence that the one defined by this distinction (totally ordered events vs. partially ordered events) is the important one (in many applications we need both types for different purposes).

What other acronyms have flown around ? - well, Forrester at some point made a distinction between CEP and BEM (Business Event Management) that has been defined as - "a process of capturing real-time business events from multiple source and assigning them to the appropriate decision-maker for resolution based on the business context of the events". I have struggled to understand the distinction - maybe the fact that it deals with simple events, however, when they mention context - determining the context may by itself require CEP.

We, in IBM are using the term IEP (Intelligent Event Processing) to denote stochastic and intelligent reasoning beyond the deterministic pattern detection to CEP; this is consistent with the layer approach, the monolithic approach fans, view IEP as part of CEP.

The new term we heard this week from IBM is BEP (Business Event Processing) and this is intended to define event processing applications in which the business user can control the behaior (i.e. define and modify patterns without the help of a programmer), a topic I also discussed in the past.

Last but not least, some people in the academic community don't like the term "processing" which they think is too elementary and talk about "event-based computing" as the name of the discipline.
After this unusally long postings, my bottom lines are :

(1). The upcoming glossary should provide a consistent taxonomy of terms here - there is still much confusion about the names, and the glossary can be a good reference point,

(2). Personally, I still prefer to talk about types of functions and not about boundaries of names, however, I understand the importance of branding.

(3). I still prefer the name "event processing" without prefix, infix or suffix - and thus continue to use this name.

(4). Hopefully, this is the last posting I am writing on the *-E-P; E-*-P; E-P-* topic - I have more interesting topics to deal with.... more - later.

On the classification of events

I am returning to the area of "classification of events", although it has been discussed in this blog several times regarding certain aspects, the reason is - some discussion in which I realized that the confusion is still alive and kicking.

Events can be classified according to their structure, semantics, or pragmatics. This classification is orthogonal.

Based on Structure:

• Simple or atomic event - an event that reports on a single occurrence.

The event structure may be flat (like relational database), or semi-structured (XML) - which also can express non-flat structure (e.g. hierarchical). In some cases events are represented as objects. Typically events contain "header" - information about this event (event class, time stamp, source...), and "payload" - the content of the event.

• Complex event - an event that is composed of multiple events

There is some difference between this and the term "composite event" from active databases which mixes structure and semantic aspect, but I don't see it useful to use this term any more.

Based on source

• Raw event: Event that has been reported to the event processing system by an external event producer.

Note that a raw event can be simple or complex. The term "raw" is relative, since it can be result of some computation process in the event producer that is transparent to the event processing system.

• Derived event: Event that is created by the event processing system as a result of some processing.

Again, the derived event can be simple/atomic - where the attributes are computed as functions of raw and derived events, or complex - in which the derived event is a concatenation of events.

Based on ontology:

• Real event: An event that designates something that occurs in reality
• Virtual event: Hypothetical, simulated or uncertain event.

Again, this is a separate dimension - can be in any structure and by any source.

Based on pragmatics

• Reactionable event (AKA situation): An event that is consumed by an external consumer at the end of the event processing network (either notify or orchestrate).

We used the term "situation" for a phenomenon in the consumer's domain of discourse that the consumer wishes to react to. This can be raw event (and in this case the event processing system is reduced to "simple event processing" - i.e. filtering and routing) or derived event, it can also be real or virtual, and in any structure.

BTW - ECA - "Event-Condition-Action" rules are rules in which the events are situations, and they are executed within the consumer domain, but I'll continue to discuss the conceptual model in the next posting.

CEP and the story of the captured traveller

Reading the recent posting of my friend Tim Bass entitled "CEP and the story of the Fish" I decided to answer with another story (from the other side of Asia) :

A traveller went in the jungle somewhere on the globe and unfortunately was captured by a tribe that is still using ancient weapons. He is brought to the chief, and the chief says - " You have trespassed into the tribe's territory, which is punishable by death, however, I am a very curious person, if you'll show me something I haven't seen before I'll let you go"; our unlucky traveller started to look in his pockets and the only meaningful thing he found was a lighter, so he took his chance, showing it to the chief saying: "this thing makes fire", however, since he was in under a big pressure, he pressed once - no fire, pressed twice - no fire, in the third time the lighter indeed has produced the promised fire, the chief did not hesitate and said "let him go", so our relieved traveller muttered to himself - "I knew that they have not seen a lighter", but surprisingly to him the chief said - "oh, I have seen many lighter, but a Zippo lighter that does not light in the first time I have never seen".

When someone disagrees with somebody else, it is very easy to assume that my point of view is right since I am smarter / knows more / more qualified / older / more experienced / generally always right etc... My preference is not to doubt the wisdom, experience or qualification of anybody that I am arguing / discussing / debating with, but make the arguments on the issue and not on the person who makes the arguments....

Enough introduction -- now for the main message of this posting, the term CEP (Complex Event Processing) has more or less agreed now in the industry to denote "computing that performs operations on complex events", where complex event is an "abstraction or aggregation of events". The term complex does not say that the processing is complex, but that it deals with complex events, as defined. Complex event processing is typically detecting predefined patterns that can be expressed by queries/rules/patterns/scripts and are deterministic in nature. Regardless if I think that this is the best term, I think that it is important to have common agreed terminology, otherwise we are confusing the industry, the customers (and sometimes ourselves). Now, Tim Bass claims that since event processing with stochastic/probabilistic/uncertain nature is more complex than what we call "complex event processing", we have to call this one - "complex event processing", and rename what we call "complex event processing" to be "simple event processing". Unfortunately, it is too late for that - and also not justified, again, since the "complex" in the "complex event processing" does not say that this is "complex processing of events" but that this is "processing of complex events" (very common misconception !). Bottom line: yes - there is another class of event processing capabilities that requires techniques from AI, machine learning, OR etc.. and that is not deterministic in nature; no - I don't think we should call it "complex event processing", we have suggested the term "intelligent event processing" which I have already referred to in previous posting , there are a variety of other postings that I have dedicated to terminology.

More - later

On Event Representation

Back to micro-oriented issue, and today I'll start discussion about -what's behind the definition of the event processing glossary and get to the issue of event representation. As the glossary says - event is something that happens in reality. We also tend to call "event" to the representation of this reality for the purpose of processing by a computer. This notion in event has in the glossary several aliases: event object, event message and event tuple. The various aliases are indications that the space of event representation is not uniform, some think about event as a message that moves around, some thinks of it as a tuple, which is part of a stream, and really the twin brother of a tuple in relational database, some think of it as an object with arbitrary structure (which may also be hidden). Obviously, there is no "universal event", and unfortunately, since in many cases, events are already given from the sources with their given formats, and the event processing designer has little to say about it, then a generic event processing system has to support multiple type of events, or have adapters that translate all types of event to some cannonic type of event (and typically both -- supporting some cannonic type of events and having adapters translating other types of events to the cannonoic type). Event can be structured, semi-structured (XML), and unstructured (the area of unstructured events processing deserves more focused attention). One of the questions is - whether there are common attributes that each event should have to enable event processing. In the data world - the answer is no - there is not a single attribute that must exist in all relations (besides the fact that each tuple should be a member of some relation - no floating tuples). For event processing -- there are some attributes that have been proposed as common attributes:

• Event-type
• Source
• Time-Stamp (or Time-Interval)

Let's look about the question - are they mandatory or not:

• The first question is whether each event is an instance of an event-type (or event-class). The glossary says - yes ! "all events must be instances of event-type". This seems reasonable, however, we may think of some exceptions - such as rare events that have not been classified.. I need to drill down on rare events in some other post.
• The second question is whether the source should be mandatory - again, this is desirable if we want to have lineage or tracing back actions/decisions, but there may be cases in which the source is indefinite, or we wish to hide the source (e.g. leaking of information).
• The third question is whether each event must have a time-stamp (or time-interval in case it happens over an interval - another area that needs more discussion) - the answer is that many event processing patterns are time related, and if we want to know which event occurs first, or if two events occurred within 5 minutes of each other, we need to know WHEN this event occurred in reality. However - in some cases it is not known, in other cases it is not really needed.

It seems that all common attributes are useful, but may be optional in some cases.

There are attributes that are common for types - such as probability for uncertain events, spatial coordinates for spatial events etc -- this is before relating to the content.

The content is determined according to domain related ontologies - and there is a lot of work today in different application domain or industry to define such ontologies. XML is the ontology language, and it has its own benefits, it also carries overhead relative to "flat" events in which the attributes are positional oriented and not keyword oriented.

Events also carry semantic information - such as: reference to entities in certain roles. In fact, event can be thought of a transition between one state to another and the information included in the event refers to a change in the universe such as:

what was changed ? what entities are affected? when it was change ? where did the change take place ? what other information is important about the change ?

This short discussion raised already several open issues that deserve further discussion - so I'll put these topics on the queue for further postings.... more - later.

Revisiting the "classification of event processing applications - part I"

The picture about is an attempt to classify styles of beards, my first beard has been "goatee and mustache", but after a short period I moved to full beard and persists for more than 30 years, unlike people who grow and remove beards, for me it is an integral part of my personality. Anyway, talking about pictures - for some strange reason the almighty Google has chosen the dog picture I've posted in one of the previous blog as the first answer to the query about "dog" in Google image points at this blog... the wonders of search technology :-)

Now - back to the event processing business, following a comment by PatternStorm (my dear colleague Claudi), I have checked the current definition "complex event processing" in the glossary is somewhat wider than I have remembered, thus, according to it - derived events are subset of complex events, where I thought that some derived events (like enrichment of a single event) are not complex event. Anyway, I am revisiting the classification done in the previous blog, and avoiding the terms "simple event processing" and "complex event processing" - since indeed it has nothing to do with the intuitive meaning of the terms : simple and complex. However, the three classes in terms of type of processing still exists - so I'll repeat them and try different names (but names are always a problem - so any help will be appreciated !).

• raw event processing: operations on events that do not change the events - just filter and route them.
• transformed event processing: aggregation and enrichment - this is a type of processing that transform the event. Examples: enrichment it is transformed by adding attributes taken from an external store (e.g. database), translation - create an event with different format / contain translated attributes...
• Pattern-based event processing -- detecting pattern - and derive events based on the events that participate in the detected pattern.

As said, this is just one of the classifications. More discussions about other classifications - later.