This is a blog describing some thoughts about issues related to event processing and thoughts related to my current role. It is written by Opher Etzion and reflects the author's own opinions
Saturday, October 6, 2012
More on the semantic overloading of derived events
Saturday, September 22, 2012
The semantic overload of derived events
The term "derived events" is frequently used in the event processing terminology. Here is an example taken from the Gigaspaces Blog of using the term "derived event". Recent discussion with somebody who is just learning the event processing area, made me realized that we overload the term "derived event". On one hand we define event as "something that happens", and say that a derived event is an event, thus one may assume that derived events also happen. However, the way this term is typically used has some semantic overloading. There are actually different types of derived events.
Friday, December 10, 2010
On the 4Ds -- past version and the proactive version
The climate in Israel this year is quite strange, it is December, and today I still saw people going in the street with short dress, the summer just did not go away. However, the forecast for the next three days, starting tonight is of a major winter storm (which here means a lot of rain, not snow) and much colder weather, so getting the winter clothes ready.
Today I've read a blog posting by Jeff Adkins, one of the people with most practical knowledge about event processing, who relatively recently joined IBM GBS (Global Business Services). Jeff blogged about the 4Ds-- detect, derive, decide, do. These four are part of smart systems that sense and respond, what is known as reactive system. I knew that this looked familiar, so went back to my archive and found that seven years ago we were engaged with a project called "active integration" (the actual application was in the insurance area, but we have generalized the concept), roughly what was known by Gartner as "real-time enterprise". Here is the original flow from that project:
I don't think it has been original invention, it was a variation of concepts from control theory, but it looks very similar to the 4Ds, with a more detailed granularity.
The first D: Detect spread into two phases on our model: sense and detect, where the sense dealt with instrumentation and sensing of raw events, and detect with a detection of the meaningful situations by pattern detection.
The second D: Derive is the same: created a derived event (and sometimes also derived data) as a result of this detection
The third D: Decide is partitioned to three phases: Analyze - determine the possible alternatives and recommend, Collaborate - in case of "human in the loop" within the decision, and Decide -- apply some decision procedure to select among the alternatives (simulation, analytic methods, predetermined rules).
The fourth D: Do we called "effect", since it had to effect a running system.
The loop indicates that the after effecting the system, it feedbacks through its instrumentation mechanism and the sense phase is looking again for things that needs reaction.
I agree that the 4D is much more catchy then our more detailed drawing.
And one comment: when we did this work, we worked on REACTIVE system - something has occurred, we detect it, and then do something to repair. This drawing is also a good description of our current project that deal with PROACTIVE systems, but the semantics is somewhat different: the detection is of predicted undesired states instead of situations that require reaction since something already happened.
Tuesday, September 8, 2009
On situations and events

Almost midnight, after a long day that started in 8AM and did not end until now.... some days are like that, tomorrow might be a slightly shorter work day. Before retiring to sleep, I'll write something on this Blog as a kind of doing something else...
I have written before about situations, however, I am not sure that this term is still well understood (the same applies for context, but I'll leave context for another posting). In the EPIA book, when revising the book, we decided that the term situation is a very basic one in event processing, and put it in the introduction chapter. Situation can be thought of as event that requires reaction, but this is somewhat simplistic definition, since indeed situation is an event, but it is something that is considered as event in the user's domain, and not necessarily an event in the physical domain. Sometimes we wish to react to a raw event, sometimes we can say that a derived event represent the circumstances which we want to react, and sometimes a derived event is just approximation for the situation.
Let's look at a couple of example.
Example 1: The situation we would like to react is that a driver crosses toll both without paying -- In Israel, as high-tech country, in the only toll highway, we have camera that takes pictures of the license plate and send the bill to the driver (unless one is subscriber and then it has some identification device in the car), thus we don't really have toll booths, but in the USA it is quite pervasive. Anyway -- this situation can occur if a driver moved through "EZPASS" lane without having a device, or somehow sneaks without paying. Assuming that there are cameras that capture it, then the derived event, which is a disjunction pattern of these two events, is a complete match for detecting the situation.

However, life is not that simple, and we move to example 2. This example, that I used before in different opportunities, is that in a call center we are looking for frustrated customers in order to send a friendly customer relations officer to call them and ease their frustration.

Thursday, June 4, 2009
On temporal semantics of events - or when has the shimpent not arrived ?

In the early 199o-ies, my home away from home, has been Berkeley, where I stayed for a joint work with Arie Segev on temporal databases. In one of the weekends I have strolled along the famous SF Fisherman's wharf, and there was some store for left handed people, since I am part of the deprived minority of left handed people, I was curious and entered the store, among the different items there (mostly not very practical), I saw this clock, if you notice, it is a backwards clock, which goes anti-clockwise. I am sure that the owner of the store was right handed -). Anyway, I recalled this clock, when working on a final version of a paper entitled "Temporal perspectives in event processing" that has been accepted recently for publication, and re-read the paper (as any paper, it is written, submitted, and then after a few months a review arrives and the author has to be reminded what it was, revise according to the comments, send back, and so on, until it is either accepted or rejected), and thought that temporal semantics of events can be a good topic to write about here. The temporal semantics of a backwards clock is, of course, different than that of the regular clock, and this brings me to the temporal semantics of derived events. Some background: event may have two time-stamps (or intervals) associated with it: occurrence time and detection time. Occurrence time is the time that the event happened in reality, detection time is the time in which the event processing system detected the event message sent to it. It is easier to make the processing of the events (when did they happen ? in what order ?) according to the detection time, however, for some applications, this may yield incorrect results. There are several issues around obtaining the correct occurrence time, but let's assume that we know how to do it. While the occurrence time of a raw event (events that has arrived from an external producer that assigns the value) is explicitly provided, the question is what is the occurrence time of derived events. Let's take a simple example: In May 2nd, 10:30 the customer John Galt has issued an order for books, with a guaranteed delivery of 48 hours (see my story with Amazon in its early days as a footnote to this postings). In May4th, 10:30 Mr. Galt looked at his (forward going) watch and said: "the shipment has not arrived by its deadline". The fact that he has not reported on arrival by the deadline caused the event processing system to derive the event "shipment did not arrive", which is a time-out event (or non-event event as some vendors call it). Now the question is WHEN did this event happen ? the detection time is easy, when some computational process derived the event and emitted it to the event processing system then the detection time is set. Let's say that this happened in May 4th in 10:32. The occurrence time is more tricky. Actually I can think of three different interpretations:
1. The occurrence time of the "shipment not arrived" is the same as its detection time, which means May 4th, 10:32.
2. The occurrence time of the "shipment not arrived" is the deadline of when it should have arrived, in this case, May 4th, 10:30
3. The occurrence time of the "shipment not arrived" is the entire interval of the 48 hours, since the shipment did not arrive during this interval [May 2nd 10:30, May 4th 10:30].
What is the right answer for semantics ? there is no right answer, as some more cases in event processing, the system designer should chose among these (and may be other) alternatives.
More about temporal semantics -- later.
Footnote: A story from the early days of Amazon.

Thursday, December 18, 2008
"complex event" and "derived event" - are they synonyms ?

Israel is a small country, and its commercial TV stations just recently discovered the "reality" programs, this week was the final episode of a reality program called "big brother" in which a bunch of people are closed in a house (the one seen in the picture) for 3.5 months (those who survive until the end) doing nothing, without any connection to the outside world, and with cameras everywhere, there was a dedicated TV channel watching them 24 hours, and twice a week - TV show in prime time. This reality program drove the entire country crazy -- got unprecedented rating, and became the main discussion issue among people, today I went to the coffee room to take some coffee and have seen around 10 people there spending their time in a heated discussion around this TV program. Interestingly, in the night of the final, a group of people from the culture and artist community made a big demonstration against the TV channels that spend their production money on realities, and drop drama series -- I personally agree with them.
BTW -- event processing can be used to serve as a "big brother" and trace people's activities, but I'll blog about it another time.
I would like to answer the question of Hans Glide about my previous posting -- the question has been -- in the illustration (below) it is seen that "complex events" is not a subset of "derived events" meaning that there are complex events that are not "derived events" - is it true?

The first case is easy: enriched event is a derived event but it is not a complex event.
What about the other direction ? - well, getting back to what "derived event" is -- this is an event that is created by some "event processing agent" as a result of some event processing function. If an event is a raw event it is not derived event. However, there are in the universe
"raw complex events", not all complex events are derived by software artifacts. For example: Since David Luckham is the copywriter of the term "complex event", I'll use two of his favorite examples:
Tsunami

Economic Crisis

(David referred to the one started in 1929, but our generation also won one of these)
The "economic crisis" is a complex event -- it is certainly an event, and it is aggregation of other events, but this aggregation is not created by software, the raw event is already complex; likewise the "tsunami".
More - later.
Thursday, July 17, 2008
On relationships among: derived event, composite event, complex event and situation
This is the water park "Luna Gal" where I am planned to spend the late afternoon with my children (it is my wife's turn to have a business trip this week) as part of a fun day for employees, meanwhile I am again at my (air-conditioned) office with the morning coffee and the Blog. One of my somewhat neglected obligations that I am trying to catch up these days are writing some terms to the encyclopedia of database systems, since we are in the glossary era, I would like to answer a FAQ question about the relationship between four terms: derived event, composite event, complex event and situation. The first three are glossary terms, the last is not, but is being used a lot in the community jargon. Tim Bass has recently posted in his Blog a simple solution model which is originated in the experimental psychology research.
The following illustration clarifies the relationships:

Derived event is an event that is created as a result of a computational derivation process as a function of one or more events.
Saturday, March 8, 2008
On Forward and Backward CEP
Paul Vincent from TIBCO has recently posted some thoughts on "goal oriented event processing" . As usual, Paul sees event processing during the " through the rule spectacles, so I'll try to predent it in a wider perspective. It is agreed that the role of CEP (which by itself is part of larger EP) is to detect patterns over multiple events. What is forward and backwards in this context ?
- Forward: For each event -- check if this event completes some pattern
- Backward: Given a pattern -- check if this pattern have been satisified (in some time context) in the past.
Both are useful for different cases, and sometimes we need a mix; looking deeper at the differences:
- Forward is always event driven (it is done as a reaction to event)
- Backward is request-response (it is done as part of explicit request), note that the request may by itself be trigerred by an event that makes it hybrid model.
- Forward is used when patterns need to be detected immediately, it is most cost-effective to optimize for all patterns that events may complete then to look for all patterns and see if they are completed. Example: detect arbitrage opporunity between two exchanges.
- Backwards is used when patterns are done periodically, or as ad-hoc queries. Example: At the end of each month, Find all stocks that during the last month have satisfied the following conditions: The stock closing values at the end of the day were strictly increasing over a period of five consecutive working days, anywhere during this month; The stock value in the beginning at the end of the five days value was at least 30 percent more than its value at the beginning of the five days period.
- Mixed is used when in some cases -- a pattern may need "reinforcement" in past patterns -- Example: A person that has deposited (in aggregate) more than $20,000 within a single working day is a SUSPECT in money laundering. To reinforce the suspicion the following retrospective patterns are sought:
There has been a period of week within the last year in which the same person has deposited (in aggregate) $50,000 or more and has withdrawn (in aggregate) at least $50,000 within the same week.
The same person has already been a "suspect" according to this definition within the last 30 business days.
If any of these patterns are satisfied – the event "confirmed suspect" is derived.
Forward and backward pattern detection can, of course, be implemented in various ways - business rules, SQL (stream and regular queries as backward), Scripts and specialized patter-oriented implementation.
Tuesday, January 15, 2008
On situations
Hello from Zurich, got here in a train from Munich - and this is the famous train station of Zurich (in light), I have arrived in the dark, but has been in Zurich quite a lot of times before, tomorrow I am visiting the IBM Zurich Lab (which is actually located in Rüschlikon (a suburb of Zurich) home to four Noble prizes in physics. The hotel is 5 minutes walk from the train, but I forgot to print a map, and had to search a little. In the first half of the day I have participated in the continuation of the CITT conference, and earned my bread by giving a talk. After the talk I had some discussion with people about the term situation that I mentioned in the talk. I have borrowed this word from "situation calculus" in AI, and it fits well with terms in related domain such as: situation awareness. The discussion has been has been around one of the common misconceptions that situation is an alias to derived event. The answer is - no, these terms are not aliases ! Situation is some phenomenon that requires reaction (or notification). It is in the conceptual ontology of the consumer. Now - let's check the relationships with event:
(1). A raw event that is reported by a producer is a situation in the consumer's terms - in this case, only routing is required ("simple event processing").
(2). A raw event is reported by a producer has 1-1 correspodence to a situation in the consumer's terms, but more information or translation is required - in this case, the situation is created by mediated event processing.
(3). The situation occurs in a deterministic way once an event pattern is identified, and it is a functions of the events participating in the pattern - in this case the situation is created by complex event processing.
Note that the situation is not the derived event - but the derived event in this case is an indication that the situation is detected, note also that not all derived events are situations, derived events that are consumed by other agents, are not situations, only those that are consumed by event consumers.
However, situation detection may not be purely deterministic - and here we are getting to "intelligent event processing". Example: the pattern is only an approximation, example: the pattern is - "an event E1 happens at least 4 times within an hour". Now, if the event E1 happens 3 times and the pattern is an approximation, there is some probability that the situation did occur. This is even before looking at probabilistic events.
More about uncertainty in event processing - later.
Thursday, January 10, 2008
On the classification of events
- Simple or atomic event - an event that reports on a single occurrence.
The event structure may be flat (like relational database), or semi-structured (XML) - which also can express non-flat structure (e.g. hierarchical). In some cases events are represented as objects. Typically events contain "header" - information about this event (event class, time stamp, source...), and "payload" - the content of the event.
- Complex event - an event that is composed of multiple events
There is some difference between this and the term "composite event" from active databases which mixes structure and semantic aspect, but I don't see it useful to use this term any more.
Based on source
- Raw event: Event that has been reported to the event processing system by an external event producer.
Note that a raw event can be simple or complex. The term "raw" is relative, since it can be result of some computation process in the event producer that is transparent to the event processing system.
- Derived event: Event that is created by the event processing system as a result of some processing.
Again, the derived event can be simple/atomic - where the attributes are computed as functions of raw and derived events, or complex - in which the derived event is a concatenation of events.
Based on ontology:
- Real event: An event that designates something that occurs in reality
- Virtual event: Hypothetical, simulated or uncertain event.
Again, this is a separate dimension - can be in any structure and by any source.
Based on pragmatics
- Reactionable event (AKA situation): An event that is consumed by an external consumer at the end of the event processing network (either notify or orchestrate).
We used the term "situation" for a phenomenon in the consumer's domain of discourse that the consumer wishes to react to. This can be raw event (and in this case the event processing system is reduced to "simple event processing" - i.e. filtering and routing) or derived event, it can also be real or virtual, and in any structure.
BTW - ECA - "Event-Condition-Action" rules are rules in which the events are situations, and they are executed within the consumer domain, but I'll continue to discuss the conceptual model in the next posting.
Tuesday, January 8, 2008
On the Event Processing Conceptual Model

- Event producers - emit events in push or pull (periodic or on-demand).
- Event Processing Networks - process the events, create derived events where applicable, route the event to event consumers at the edge of the network
- Event consumers - consume the processed events and orchestrate/notify.
As you can also see the event processing network has two bridges - one to the producers and another one to the consumers, in which several of the functions of the EPN may reside.
We believe that this is a general enough conceptual model that can contain various variations.
Note that the "event processing network" requires an "event bus", this is a platform-independent term - in SOA environment - as noted in a previous posting the "event bus" converges with the "enterprise service bus". More deep description of the different ingredients - in later posts
Sunday, December 30, 2007
Event Processing for Business Intelligence
From the various descriptions of Business Intelligence found on the Web, I have chosen the one by Hakki Erbug as a starting point.
As noted in several previous postings, IMHO, event processing is a set of technologies that have multiple usages, and are not really strongly associated with a single type of application. Business Intelligence is similar in the fact that it combines various technologies, but different in the fact that it is focused around area of using data for decision making in various ways. In the event processing case, decision making is one of the possible usages, but not the only. This posting will briefly survey how event processing can enrich business intelligence, and in fact, in the Gartner EPS summit I have seen several BI vendors that are looking at EP as a natural capability to complement their products.
Going on Erburg's illustration clockwise:
1. "Active Data Warehouse" - While traditional warehouses are being updated in batch, the notion of active data warehouse makes the warehouse update itself an event-driven action. The rationale is: when a certain event (raw or derived) occurs, a decision has to be taken, however, the decision relies on a data warehouse, thus, an update of the data warehouse should occur before making this decision. The update can be of the same event that happens or of some collection of data that has still not been updated in the data warehouse and is needed for the decision making. There can be some time constraints associated with the decision (and in turn with the warehouse updates). The time constraints are not necessarily micro-seconds, the constraints can be minutes or hours, but they are typically well-defined.
2. ETL and mediated event processing: ETL has some functional similarity with mediated event processing, it is also about transformation. We see mediated event processing and ETL getting closer to one another, where difference may be in quality of service. In the future there may be a case that ETL will become a specific case of mediated event processing (of course ETL folks may say the same from the opposite direction).
3. Real-time analytics: While analytics (simulation, optimization, mining...) has been used for a while in the decision making part of BI, in the event-driven world, the reaction to an event in some cases is temporally-bound, which means that there is a real-time constraint, or upper limit on the requested reaction time. This provides new way of thinking about analytics - while without time constraints an optimization should strive to get the "best result" (or if heuristics satisfy some approximation condition), in real-time analytics the optimization strives to get "the best result that can be obtained in T time-units as specified (e.g. 18 seconds)". How does event processing play in real-time analytics? it may play in a simulation mode - scenarios are created and simulated events are emitted - they in turn may create simulated derived events which determine the situations of this simulation. This is, of course, in addition to the fact that in an event-driven universe, the entire BI cycle is event-driven and relates to the event and its context.
To conclude -- event processing is a natural step in the BI capabilities, and thus I expect BI suites to support the event-driven flavor... This - again, does not say that BI is the ONLY use of event processing. I still need also to refer to the issue - "has BAM failed because it was not based on BI techniques" as claimed in the article that triggered my discussion on the BI topic - stay tuned.
Saturday, December 15, 2007
On simple events and simple event processing
This is a picture I have borrowed from Siemens, however, I'll use it talking about simple events and simple event processing. There is a constant confusion around the terms "simple" (and "complex") here due to the ambiguity of the phrase: simple (complex) event processing - does it mean: processing of simple (or complex) events? or does it mean simple (or complex) processing in events. In this posting, I am drilling down to the simplicity notion.
Let's start with simple event - I prefer to contrast simple event with composite event where the contrast is in the structure - composite event is a collection of simple events
Now - is there different processing for simple events and composite events ? - in principle no - there are some functions on collections that are not applicable for atomic events, but if we take a collection of simple events that has not been concatenated we can apply the same processing for them.
Thus - my preference is to attach the simple to the processing and not to the event type, and define simple event processing as simple type of processing, no matter what the event structure is. What are the characteristics of simple event processing ?
- processing is done on a single event - not looking on other events, but events are processed "one at a time".
- only types of processing possible are: filtering and routing
- filtering decides whether this event should be passed
- routing decides to whom this event should be passed
Basic pub/sub with filtering is simple event processing.
ECA (Event-Condition-Action) rule is also simple event processing -- this does not say that the event cannot be derived/complex/composite - but regardless of the history of the event, its structure, the reason why it is created, and its source the ECA rule still performs a simple event processing, and processes one event at a time, where the condition provides filtering, and the action is indeed routing to somewhere to do this action.
Furthermore, in many cases, the "simple event processing" is a preamble to the event processing network done by the event producer, or post-processing done by the consumer, however, it can still be part of the Event Processing Network.
More related concepts - in the next postings.