David Luckham, who took upon himself to survey the history of "complex event processing" has written a series of articles about the roots and developments the CEP area, while this is relatively young as discipline, it has roots in other disciplines, and people who worked separately from different perspectives, origin disciplines, and types of applications.
I recommend reading it. I'll make just some small comments for the recent third article :
(1). One of assertions in the article states:There were research groups within these companies engaged on similar projects to those in universities. Some of the groups ended up producing
prototype CEP products. In these cases, the decisions as to whether to let these
prototypes go forward to productization were haphazard, probably because they
were made by a management that didn’t understand the technology or its
potential.
Well - one must realize that corporate executives have infinite amount of wisdom, otherwise they have not been corporate executives, and if a decision looks haphazard, it is only due to the mental limitations of us, simple mortals.
(2). Another assertion is: But the largest number came out of an active database background. This is the reason why many of the commercial languages for event processing are
extensions of SQL
More accurately -- some of the current products we see are indeed descendants of "active databases" which was based on ECA (event condition action) rules. Among the products which apply this approach we can find RuleCore and Aptsoft (now Websphere Business Events); the products which extended SQL came from different paradigm in the database community of "data streams management" according to which, the queries are constant and data flows, instead of databases in which the data stands and queries flow. This has been later development which started with Jennifer Widom's Stream project; the two paradigms, though both came from the database community should not be mixed (although there are certain individuals who were involved in both).
Israel is a small country, and its commercial TV stations just recently discovered the "reality" programs, this week was the final episode of a reality program called "big brother" in which a bunch of people are closed in a house (the one seen in the picture) for 3.5 months (those who survive until the end) doing nothing, without any connection to the outside world, and with cameras everywhere, there was a dedicated TV channel watching them 24 hours, and twice a week - TV show in prime time. This reality program drove the entire country crazy -- got unprecedented rating, and became the main discussion issue among people, today I went to the coffee room to take some coffee and have seen around 10 people there spending their time in a heated discussion around this TV program. Interestingly, in the night of the final, a group of people from the culture and artist community made a big demonstration against the TV channels that spend their production money on realities, and drop drama series -- I personally agree with them.
BTW -- event processing can be used to serve as a "big brother" and trace people's activities, but I'll blog about it another time.
I would like to answer the question of Hans Glide about my previous posting -- the question has been -- in the illustration (below) it is seen that "complex events" is not a subset of "derived events" meaning that there are complex events that are not "derived events" - is it true?
The answer is: indeed - "complex events" intersects "derived event" but there are derived events which are not complex events and complex events which are not derived events.
The first case is easy: enriched event is a derived event but it is not a complex event.
What about the other direction ? - well, getting back to what "derived event" is -- this is an event that is created by some "event processing agent" as a result of some event processing function. If an event is a raw event it is not derived event. However, there are in the universe
"raw complex events", not all complex events are derived by software artifacts. For example: Since David Luckham is the copywriter of the term "complex event", I'll use two of his favorite examples:
Tsunami
Economic Crisis
(David referred to the one started in 1929, but our generation also won one of these)
The "economic crisis" is a complex event -- it is certainly an event, and it is aggregation of other events, but this aggregation is not created by software, the raw event is already complex; likewise the "tsunami".
More - later.
Today I have participated in the local IBM championship in Backgammon (in Hebrew we use the Turkish name "shesh-besh"), however, my participation did not last very long, I lost the first game, and did not move to the second round, thus, returned to my office... At least the one who won the game over me is the technician in charge of the video conference equipment, so if one wants to conduct video conferences, make sure he likes you, since he is the only one in the lab who knows how to operate this simple equipment.
Today I would like to write a little bit about "event derivation" - in one of the past postings I've used this illustration in order to explain relationships between terms. Why do we need all these terms? -- well - we don't, but they are "legacy" terms, thus it is a good to show the relations among them.
When saying "derivation" -- let's compare it to two other types of derivations:
- Inference -- where the input are: facts and inference rules, for example: fact 1 -- Jack is the father of Mike; fact 2 -- Mike is the fater of Alex; inference rule: X is grandfather of Y if X is father of Z and Z is father of Y. From this system we can infer that Jack is the grandfgather of Alex. This is a deductive inference, there are some other types of inferences, and can be obtained by logic programming, or inference rule engines.
- Data derivation --- although the term is broader --- if we'll take the relational database variation --- it takes a collection of relations as an input, apply query, and gets a relation as an output -- here there is no logical inference, but computation over data. Other variation of data derivation is a spreadsheet formula, typically performing some aggregation operator (sum, count, average...) on neigboring data in the table.
Event derivation is neither of the above, but has similar nature to both. The input and output are events, if we present it as functions then:
Inference: Facts X rules ----> Fact
Data derivation: Relations X Query ---> Relation
Event Derivation: Events X Event Derivations ---> Events
Now looking closely at event derivation. First -- why does event derivation occur ?
Event derivation may occur from various reasons -- while in mind it is closely related with pattern detection, it may also be associated with other event processing functions such as: enrichment and transformation.
Some examples:
Example 1:
Input event: Order that consists of the attributes (not including the header) -- customer, item, quantity.
Functionality type: enrichment -- add attributes as: customer-type, customer-reliability-rank to the order.
Output event: the enriched event. In this case the derived event is identical to the event that has been created by the main function (enrichment) and the derivation is just copying.
Output event occurs immediately when function is done
Example 2:
Input event: Order (same as first example)
Functionality: Aggregation -- quantity by item.
Output event: Item-Demand: Item, Quantity. Multiple events - one for each item.
Output event occurs at the end of the context ("deferred").
comment: here a composite event is being accumulated, and at the end of the context is decomposed to individual events.
Example 3:
Input events: Order (same as first), Item-Demand (Output of example 2).
Functionality: Detect Pattern, the pattern is: Quantity of a single order > 2/3 from quantity of the same Item.
Output event: Dominant-Order: Order-Id (from header), Item, Dominance Ratio = (Order.Quantity / Item. Quantity)
Output event occurs as soon as detected.
Example 4:
Input events: Temperature-Reading (every 1 minute).
Functionality: Aggregator/collector -- collects all reading of a certain hour and creates one event.
Output event: Every hour -- Composite event that consists of 60 raw events.
These are some examples. More formal definition of event derivation - later.
Yesterday, I drove my 11 year old (youngest) duaghter, Daphna, with four of her friends (all boys), 65 kilometers to the nearest branch of Max Brenner (see pictures above) a chocolate store which also has a restaurant, most of the menu is the chocolate oriented menu, but for parents there is also some real food. They still did not get to Haifa, but I think that they have brnaches in New York. Anyway, we found the nearest branch and got there...
I have posted an explanation about the approach to the "on off window" that Pedro posted; somebody asked me beyond the explanation -- how this scenario can be expressed in the EPDL meta-language that we are working on. Actually this is a very simple scenario, it is expressed like this:
Let E be the event type; with two attributes param1 and param2 (as given); we also need to route the event e to the agent a, and the output event to a producer - lets define two output pipes, p1 and p2, both of them carry events of type e.
We define:
- Context, name = c, type = temporal, initiated-by = event e with param1 = 2, terminated-by = event e with param2 = 0.
- Agent, name = a, type = filter, within-context = c, filter-condition = all, input = p1, output = p2
Explanation: Context determines which subset of the "event cloud" are applicable, there may be multiple agents within a context, but each agent has a single context. In this case - we can use the simplest type of agent - filter - with a trivial filter-condition - "all". Agent a will not be active before cotnext c will be active. With the event that satisfies the condition, the context is opened, thus agent a receives input, does nothing with it, and post it as output.
This is of course very simple case -- we can also concatenate all selected events to one composite event; we can also use more sophisticated agent e.g. pattern detection.
More educational material about languages -- later.
