On common misconceptions about event processing - the complexity misconception

David Luckham has coined the term "complex event processing", this term has caught as the marketing term behind many of the vendors that provide event processing platforms (comment: IBM, and recently Progress/Apama moved to use the term "business event processing"). While this term succeeded to get traction, it also is a source of on of the common misconceptions, Luckham talked about complex events, and their processing, some people understand it as the complex processing of events, and some just view it as the intersection between event processing and "complex systems". Complex event is defined as "abstraction of one or more other events", which also leads to some interpretations about the nature of abstractions, so this interpretation is easier to understand. However, the misconception is that it is more intuitive to think about "complex event processing" in the second interpretation as "complex" processing of events, and this brings us to the question -- what is complexity? there can be different dimensions of complexity.

1. The complexity may stem from the fact that we don't know what exactly we are looking for, and generally looking for anomalies in the system (e.g. trying to find security violations), thus some AI techniques have to be applied here.
2. Another case is that we know what are the patterns or aggregations we are using, but they require complex computing, in term of functional capabilities.
3. Another case is that the complexity is in some non-functional requirement: such as scalability in several direction (scale-up or scale-down), strict real-time performance constraints, highly distributed system etc..
4. Another case of complexity is in interoperability, the need to obtain events from many producers, and use events in many consumers, which requires instrumentation/modification of a lot of legacy systems.
5. Yet another case of complexity may be unreliable event sources, handling false positives and false negatives.

There are probably more complexity cases, however the interesting question is whether the main goal of an event processing system is to solve a "complex" problem.

Om my scientist hat, it is definitely more exciting to solve complex problems, even better, problems of the type that have never seen before. However, from pragmatic point of view, event processing applications are measured on their business value, and there might be a lot of business value of using event processing to systems that have none of these complexity measures, from complexity point of view they can be quite simple, moreover, there may not be an exciting aspect about the implementation as it is similar to other implementations already done, but on the measurement of "business value" it brings a lot of value, thus the value metric is orthogonal to any complexity metric, and indeed many of the applications in which event processing technology is very useful to is quite simple (according to one of the analysts report the "simple" applications are 80-90% of the potential market for the event processing technology). While there is certainly a segment of application for each type of complexity, and more work is required in these direction, the "simple" application will be the bread and butter.

More misconceptions - later.

On Event Derivation

Today I have participated in the local IBM championship in Backgammon (in Hebrew we use the Turkish name "shesh-besh"), however, my participation did not last very long, I lost the first game, and did not move to the second round, thus, returned to my office... At least the one who won the game over me is the technician in charge of the video conference equipment, so if one wants to conduct video conferences, make sure he likes you, since he is the only one in the lab who knows how to operate this simple equipment.

Today I would like to write a little bit about "event derivation" - in one of the past postings I've used this illustration in order to explain relationships between terms. Why do we need all these terms? -- well - we don't, but they are "legacy" terms, thus it is a good to show the relations among them.


When saying "derivation" -- let's compare it to two other types of derivations:

• Inference -- where the input are: facts and inference rules, for example: fact 1 -- Jack is the father of Mike; fact 2 -- Mike is the fater of Alex; inference rule: X is grandfather of Y if X is father of Z and Z is father of Y. From this system we can infer that Jack is the grandfgather of Alex. This is a deductive inference, there are some other types of inferences, and can be obtained by logic programming, or inference rule engines.
• Data derivation --- although the term is broader --- if we'll take the relational database variation --- it takes a collection of relations as an input, apply query, and gets a relation as an output -- here there is no logical inference, but computation over data. Other variation of data derivation is a spreadsheet formula, typically performing some aggregation operator (sum, count, average...) on neigboring data in the table.
  

Event derivation is neither of the above, but has similar nature to both. The input and output are events, if we present it as functions then:

Inference: Facts X rules ----> Fact
Data derivation: Relations X Query ---> Relation
Event Derivation: Events X Event Derivations ---> Events

Now looking closely at event derivation. First -- why does event derivation occur ?

Event derivation may occur from various reasons -- while in mind it is closely related with pattern detection, it may also be associated with other event processing functions such as: enrichment and transformation.

Some examples:

Example 1:

Input event: Order that consists of the attributes (not including the header) -- customer, item, quantity.
Functionality type: enrichment -- add attributes as: customer-type, customer-reliability-rank to the order.
Output event: the enriched event. In this case the derived event is identical to the event that has been created by the main function (enrichment) and the derivation is just copying.
Output event occurs immediately when function is done

Example 2:
Input event: Order (same as first example)
Functionality: Aggregation -- quantity by item.
Output event: Item-Demand: Item, Quantity. Multiple events - one for each item.
Output event occurs at the end of the context ("deferred").
comment: here a composite event is being accumulated, and at the end of the context is decomposed to individual events.

Example 3:
Input events: Order (same as first), Item-Demand (Output of example 2).
Functionality: Detect Pattern, the pattern is: Quantity of a single order > 2/3 from quantity of the same Item.
Output event: Dominant-Order: Order-Id (from header), Item, Dominance Ratio = (Order.Quantity / Item. Quantity)
Output event occurs as soon as detected.

Example 4:
Input events: Temperature-Reading (every 1 minute).
Functionality: Aggregator/collector -- collects all reading of a certain hour and creates one event.
Output event: Every hour -- Composite event that consists of 60 raw events.

These are some examples. More formal definition of event derivation - later.

On relationships among: derived event, composite event, complex event and situation

This is the water park "Luna Gal" where I am planned to spend the late afternoon with my children (it is my wife's turn to have a business trip this week) as part of a fun day for employees, meanwhile I am again at my (air-conditioned) office with the morning coffee and the Blog. One of my somewhat neglected obligations that I am trying to catch up these days are writing some terms to the encyclopedia of database systems, since we are in the glossary era, I would like to answer a FAQ question about the relationship between four terms: derived event, composite event, complex event and situation. The first three are glossary terms, the last is not, but is being used a lot in the community jargon. Tim Bass has recently posted in his Blog a simple solution model which is originated in the experimental psychology research.


The following illustration clarifies the relationships:

Derived event is an event that is created as a result of a computational derivation process as a function of one or more events.

Complex event is an event that is an abstraction of other events.

Composite event consists of collection of events that satisfy some pattern.

These are not the glossary definitions but they are explanations of the definitions that will enable to determine the relationships. As we can see not every derived event is a complex event - a derived event can be just an enrichment of a single event, and as such not a complex event, on the other hand, complex event can be created by explicit grouping of events and not by computational derivation, thus a complex event may not be a derived event. The relationships between derived events and complex event are - partial overlapping.

On the other hand, composite event is a subset of both complex event (since it is a collection of event and as such an abstraction) and derived (since it is derived by a computational process). Interestingly composite event is not the intersection between derived event and complex event, since the intersection also include scalar derived events that aggregate the raw events but don't keep the collection of events. Thus composite event is a subset of the intersection.

How does situation fit in ? - well, situation is a different beast, it is not some syntactic operation on event, but it is a semantic abstraction that denotes -- something that requires reaction. An event of any type may designate or approximate a situation, for example: fraud suspicion is the situation and a certain pattern is an indication (either exact or approximate) to the occurence of this situation, when there is a determination that a situation has occurred - a derived event may be issued to designate this situation, and the pattern is the condition (again - exact or approximate) for the situation. Note that an event pattern may have some certainty measure (e.g. probability) for the fact whether it designates a situation , not all cases are deterministic.

Does every derive event represent a situation? --- not really. A derived event can be used as an interim event or part of dissemination system and not have semantic role of situation. The same is true for all other event types.

Last but not least --- Paul Vincent in his Blog reported on a tutorial he has given about the ETPS glossary (useful presentation, I'll ask permission to borrow it). He asks a question - whether event inheritence is derived event ? well - let's check the definition of derived -- creating an event as a result of computational process as a function of one or more events. Does it describe the inheritance process ? -- IMHO it does, thus I think that inherited event is a type of derived event, though the derivation process is somewhat different from the derivation process in other cases. more - later.