on event processing language and modeling

Hello From the IBM Hursley Lab in UK, in the picture you can see the historical building known as "Hursley House" where customers' meetings are being held, where I spent most day today...

In previous posts I have argued for the need for a specialized event processing language since the functions of event processing are specialized, and on the other hand in another posts I have argued for event processing being part of a bigger whole - which argues for more general programming language that can express the "bigger whole". This is clearly a trade-off here.

What can be done to "have one's cake and eat it too", have the specialized language but also remain within the more general framework -

How this can be achieved - some ideas:

• Have a unified meta-model that can generate multiple lower language languages
• Have an embedded EP language inside regular programming language the way that are embedded SQL work
• More ideas ?

This is not just an event processing discussion but leads to larger discussion about specialized vs. generic tools, and the trade-off involved.

on the various temporal dimensions of an event

This is the Winchester cathedral in England, and I am finding myself in additional business trip, but this time to the country-side of England, one hour drive from London, and seems like several centuries distance in time. Talking about time - I'll dedicate this blog to temporal issues in event processing. Well, this is a big topic, so I'll start with the topic of the temporal dimensions of an event. The first issue is - does event happen over a time-point or over a time-interval. A couple of years ago I had a discussion about it with David Luckham

who claims that "nothing happens over a time point", which means - everything happens over an interval. This may be true, but for practical reasons, there are cases in which we want to look at an interval as a time-point, since we are not really interested to decompose the interval further, the abstraction of chronon as atom of time is very useful - saying that we can look at the "smallest significant unit of time" for a certain application, examples: minute for delivery of packages, 1/100 second for athletics, day for personnel applications, month for salaries etc...

Using this we can say that an event occurs over a time-interval, but for some (maybe most) applications it is sufficient to approximate the interval to a time point in the right granularity (the application's chronon). However, this is talking about the time that the event happened in reality - indeed there maybe several temporal dimensions associated with a single events:

• The "event occurrence time" - the time that the event happened in the reality
• The "event detection time" - the time that the event was emitted / signalled / detected by the event processing system (typically a single time-point).
• The "event valid time" - the time-interval in which the event may effect the event processing system - typically starts at the time it starts to effecting the processing, and ends at the time it stops effecting the processing. This is less obvious then the other two, but important for some types of processing.

In some cases the order of events do not matter, however, when it matters - e.g. a pattern is depends on the fact that event e2 occurs after event e1 (such as: the order of the events "printer is offline" and "printer is online" of course matters, since it indicates whether the problem has been resolved or is still there. The order in this case is of course according to the "event occurrence time" (note that if the event occurrence time is an interval, there may not be a total order, but interval relations).

This is only a start of the temporal saga in event processing, there are many more questions that I'll try to include in future blogs:

• What happens if events are getting out of order ? (short and long delays) ?
• What is the "occurrence time" and "detection time" of derived events ?
• What is retrospective event processing, and how it fits in ?
• What is temporal context and how it relates to time window ?

More on all of these - later.

event processing as a part of a bigger picture

The picture is the view seen from "Midreshet Sde Boker" , where I have driven yesterday (3.5 hours drive each direction !) to attend the opening ceremony of the activity which my second daughter (out of four), Adi, is going to participate during the coming year. This is an activity in which high-school graduates postpone their army duty for one year, and invest it in volunteering activities, and learn leadership skills, in the Negev dessert, "in the middle of nowhere". From the Greek philosophy we have two competing views of the relations between the individual and the society -- the individual-centered, where society is aimed to serve the individuals, and society-centered, where individuals are part of society. Our universe is very much individual-centered, so it is a refreshing to find a group of young people who are ready to contribute one year from their life (yes, without being paid), to do volunteering work that help the society.

In moving back to "event processing", it is clear that event processing is not a detached thing, as the customer really wants a manufacturing system, or trading system, or billing system, and technology is only a way to achieve it, it is also true that there are typically no pure event-driven applications, but parts of applications that do event processing, which makes the interoperability, the ability to embed event processing within existing paradigms, and the ability of systems and applications to sense and respond, is a critical success factor.

Consequently, we'll see more and more event processing systems embedded in application integration middleware, rather than a stand-alone engines. More on that - later.

Causality and lineage in event processing

[Today I am drilling into the micro (technical issue), with some macro level lesson at the end]


This is what I've found on the web looking for an image describing causality



David Luckham , in his book has made "causality" as one of the major abstractions in CEP. As you can see in the Wikipedia entry, this term has interpretations in multiple disciplines. Causality simply says that existence of event E2 is caused by event E1. Anybody familiar with discussions around the meaning of "causality" in logic, realizes that there are various approaches and terms in this area. In our context, event processing, we can look at different types of causality.

Type I: predetermined causality - Event E2 always (or conditionally) occurs as a result the occurrence of E1, thus we don't need to have any sensor to get event E2 we may assume it happened if E1 happened (and the condition is satisfied), some time offset or interval may be attached to this causality. Note that in this case E1 and E2 are both raw events.

Type II: The event E1 is an input to an EPA (Event Processing Agent) AG, and the event E2 is an output of AG. In this case E2 is a derived (virtual) event. This can be further refined to other types according to the issues - whether E2 is really a function of E1, this is known since the agent's specification is part of the system.

Type III: The event E1 is an event that is sent from an EPN to a consumer C. C applies (conditionally) some action AC, where the specification of AC is not known to us, but we observe that it emits the event E2. This is another type of causality (the event E2 would not have been emitted, if E2 would not have triggered AC), however, E2 may or may not have functional dependency with respect to E1 (i.e. the value of E2 is somehow function of E1) .

Some questions have been asked on my previous post on ECA rules, if EPA is not just an action, why is this distinction important - and the answer is - in the EPA case we have dependency of type II - which means both causal and functional dependency, while in a general action, it is not known whether there is functional dependency.

The question is -- why is this all causality discussion important ? is there just a theoretical notion, and the answer here is -- lineage tracing.

In some cases, it is important to be able to answer questions like:

• What have been the chain of events and transformations that caused a certain action (decision) to occur ?
• What are all the consequences of a certain events (of a certain type) ?
• What would have happened if a certain event that did not happen would have happened in a certain time-point ? (or the reverse - an event that happened would not have happened).

Applications ? auditing, decision analysis... the last type of question relates to some past work done in the temporal area -- temporal issues are very pervasive in event processing and deserve one (or more) postings at a later point.

More about event processing and business rules

Since my previous post on this topic has triggered some discussion, comments and questions - I am returning to the issue of "event processing and business rules". One observation I got today is that if we'll some of the "event processing typical applications", example: Paul Vincent's post about EDA and CEP in Insurance are considered by the business rule people also as typical applications for business rule engine (and the insurance area has strong presence of business rules). To answer that I'll cite another CEP blogger, whom I don't have the honor to know personally, who used the eternal-wisdom title: When all you have is a hammer everything looks like a nail, while the post talks about SQL, this is true also for business rules. In fact, if some programming tool has enough expressive power, one can, somehow, express everything using this tool, which does not say a lot, since everything can be expressed nicely by a Turing machine (this wikipedia link also points at some freeware implementation of Turing machine, so we have a single programming tool to do everything...). However, there are things that are naturally fit into a certain programming paradigm and things that look rather awkward. Of course, there are some cases in which one can equally implement it within multiple paradigms, this is typically one of two cases - it may be very simple, or there is not (yet) a specific paradigm to express it nicely, so multiple paradigms are awkward in the same way.
It is not surprising that the same phrase about hammer and nails is also going in the opposite directions- I have seen cases where people who had CEP engines, "force" them to do things that are not natural for this paradigm, by creating artificially events and patterns.

And one comment about ECA rules (Event-Condition-Action) --- the contribution of the glorious "active databases" community to the computing industry. Complex Event Processing deals with the E part - taking events, apply some function, and at the end deriving more events. Each derived event that arrive to the "edge of the event processing network", i.e. sent to a consumer, can then conditionally trigger an action. Thus, from architecture point of view, ECA rules are really part of the consumer logic, and not part of the "event processing network". More - later.

Who should define event processing applications ?

This picture is taken from the recent ebizQ's event processing survey and shows an interesting picture, the survey done shows that the customers prefer that rules will be defined in 84% of the cases by non-programmers. This is not the current situation, but this is the way that the customers prefer it. Indeed, it is not possible to have the entire event processing end-to-end story done by non-technical people in this phase (maybe with applying some autonomic computing techniques we can come closer to it, but this is a topic for another Blog), but let's concentrate now in the CEP (Complex Event Processing) part - detection of patterns. Given the right level of abstraction both of the "programming in the small" - the individual pattern, and the "programming in the large" - the event flow, pattern composition may be delegated to business people (well - with some understanding of basic terms like the distinction between "OR" and "AND", but with no need to know how to program in Java, C, SQL and other languages geared for programmers). Bear in mind that the bigger expense in the application's life-cycle is not the cost of development, but the cost of maintenance, and this is especially true when changes are frequent, or there is a need to create new patterns to detect unexpected events within a short time. In fact, the agility, ability to quickly adapt, and ease-of-use for non-technical people is a success factor for the entire concept of event processing COTS, and the ground for competition among vendors. Investments in tools for developers address only 16% of the market, according to this survey. This reminds me that once in my academic past, I attended a teaching workshop given by an external experts, and they started their session saying that - most teachers base their teaching on the students' sense of hearing, while for most people the visual sense is much stronger, so there is a mismatch.... More later.

event processing and business rules

New week is starting after all holidays (in Israel the working week is Sunday - Thursday), so let's start it with writing something in the Blog. I have stated that SQL is not the ideal way to express event processing functionality, so, people ask me what is the alternative - business rules ? the answer is - not really. The term business rules is somewhat overloaded, however, we can group the rule types into two types:

• knowledge-creation rules - rules that create more 'knowledge" - facts, data-elements etc... These rules can use various inference techniques - deduction, induction, abduction...
• behavioral rules - rules that cause something to be done, or prevent something from happen.

The EPL should include constructs with some similarity to these two types, but this does not say that existing rule languages can be taken as a basis, this is because the domain in event processing is events (not facts or data), and that event has a distinct semantics.

• The first type of construct in EPL is the "event derivation rules" which includes transformations (filtering, translation, aggregation, split, and enrichment) - these typically transform an event (aggregation is a special case, we'll dedicate a blog to it), and "pattern based derivation" which derives events, such that the instances of the derivers are selected as part of a detected pattern. We can see that all these types require specialized semantics, thus while there is similarity in ideas - it is not really rule semantics.
• The second is somewhat similar to the "orchestration" constructs of EPL - this is being performed in the leaf of the network - after all derivations occurred, the result event may (conditionally) perform some action related to a consumer (notify, orchestrate etc...), which is also known as ECA rules (event-condition-action). Note, that from architecture point of view, ECA rules are not done in the event processing network (i.e. they are not MEP or CEP), but they occur in the territory of the consumer - at the border of the EPN.

An interesting observation id that Complex Event Processing (CEP) as defined in the glossary, deals with derivation of new events, it is neither an instance nor an extension of ECA rules, since CEP deals with "knowledge creation" and ECA deals with behavior.

Some more discussion on this topics, and relation with "reaction rules" - later.