On the anatomy of event processing agents

Weekends is catch-up times for any workaholic. In my case, I dedicate the spare time in weekend to advance in the EPIA book, which, as expected, is somewhat behind the original schedule. We are now cleaning up the main chapters of the books that deal with : event processing network, contexts and patterns. I'll dedicate some of the coming postings to book related stuff.

In chapter 6 we get a deep dive into the notion of EPN, and its various components: event processing agents, channels and global states. Consumers and producer are discussed earlier in the book. Event Processing Agent (EPA) is a major abstraction we are using. In recent posting I've discussed that the conceptual EPN may be mapped in various ways to implementations, EPA is part of the EPN, as such it is conceptual creature, and can be mapped to implementation in various ways.

The illustration on the top of this page illustrates the (logical) anatomy of an EPA. It illustrates a kind of SCA like component (SCA has now a proposed events extension, I'll discuss it another time), with input terminals and output terminals. In this cases, the EPA gets events from input terminals, each of them has a distinct type of event. Let's take a simple three event conjunction example. A virtual car dealer matches used car sellers and buyers, once there is an agreement on the transaction, it supervises that it is carried out. The used car buying transaction is a conjunction of three events (the order of them may not be important) :

Event1: The seller physical hands over the car to the buyer
Event2: The buyer transfers to the seller the agreed upon amount for the car
Event3: The department of transportation approved and records the ownership transfer -- in Israel it is a service given in post offices, the buyer and seller identify and sign, and the post office clerk has access to the transportation ministry vehicle registration system and can verify who is the registered owner, and if the car has any incumbrance

The three logical steps of EPA are: filtering, matching (if the EPA is looking for a pattern), and deriving. The filtering part may be done outside the EPA, e.g. if the EPA instance deals with a certain transaction, then the channel should route only events related to this transaction to the EPA, I'll discuss more of the notion of channel in the near future. However, there are cases that the events should be filtered by the EPA, e.g. when it involves a condition on more than one event. After the filtering part some subset of the input events survive to be the players in the EPA. If the EPA is doing also pattern matching, it creates matching sets according to the requested pattern. In the case of the used car selling, the pattern is a conjunction of these three events that relate to the same transaction, the order is up to the buyer and seller. When the pattern is matched, it means, in this case, that three events have been detected, which satisfy together the pattern. These three events create a matching set, for each EPA instance there may be zero , one or multiple matching sets during the lifespan of this EPA instance. After the matching set is created, the third phase - the derivation phase is aimed to determine what should be reported out of the results of this EPA. It can just report the composite event that contains the three events in the matching set, and it can contain any other raw or calculated value that is a function of the attributes of these three events. Furthermore, for different purposes, different derived events can be required - consumed by other EPAs or by consumers. This is the way we describe EPA, note that this description is general enough to cover EPAs from different types. The input terminals can take either individual events or sets. I'll discuss more about EPA types, channels, contexts and patterns later

Getting closer to the peak of inflated excpectations

The Gartner hype cycle has a notion called "peak of inflated expectations", which states that different technologies go up in the hype ladder until getting to the peak, where people think that can solve much of the universe's problems, and then, somehow people realize that this is not the case and go through frustration and disillusionment, until realizing the true value (if any !), and getting back on track, now with the right set of expectations. Two recent Blog postings show some indication that we are getting closer when talking about event processing:

• Carol-Ann Matignon from Fair Issac complains in her Blog that some people say that event processing = decision management and brings some counter examples where there are decision management applications that use batch processing.
• Paul Vincent from TIBCO wonders whether CEP = continuous analytics.
  

Actually, both are right. Event processing may have a role in decision management, there are some applications of decision management that are pure event processing, some in which event processing has some role, but not doing the entire trick, and some that are really batch oriented data management. Likewise continuous analytics can be done in response to event (raw or derived) or just periodically. Event processing may or may not have a role in deciding when to do this analytics (e.g. optimize the traffic light setting), the optimization itself is typically not event processing per se.

I think that it is very good to observe that event processing can play a role in many areas, likewise, it is also good to be clear about its possible role, and which are the cases in which it has value, and which are the cases it hasn't, I guess that we'll have to wait for the enlightenment phase, in the hype cycle terminology until there will be more universal clarity about the role and value of event processing. More - Later.

On conceptual and run-time EPN

Working now in my spare time on completing the second third of the EPIA book, so I'll have several postings related to the next three chapters of the books that are now in the "cleaning phase". The topic I'll discuss today deals with the concept of EPN (Event Processing Network) which is a major concept in our book. The approach we have taken in the book is to explain event processing through a meta-language that provide the various event processing concepts, and the event flow through a model based on event processing network. We are now also competing an editor that will enable the reader to play with the meta-language. However, this meta-language is not an executable language (at least not in this phase), and thus we also show the readers how the same application described in the meta-language is implemented using various executable languages of different language styles. The EPN described by the meta-language is a "conceptual EPN", it consist of logical EPAs, while the run-time EPN consist of run-time artifacts that implement the run-time instances of the EPAs.
The conceptual EPN can be mapped into physical implementation in various ways, as shown in this picture:



The traditional centralized implementation is that the entire EPN is being executed using a single run-time artifact, and the EPN describe the internal flow within this artifact.

When talking about distributed EPN, the EPN can be distributed according to several criteria:

• Segment partition: All the EPAs that relate to "platinum customers" are being executed by one run-time artifact, all the EPAs that relate to "gold customers" are being executed by another run-time artifact etc...
• Function partition: All the EPAs that perform a certain function are being executed by a unique run-time artifact
• Location partition: All the EPAs that relate to events created in a certain location.
  

These, of course, are just examples. The most distributed example, is, of course, a direct mapping of each EPA instance to run-time artifact.

The conceptual EPN is important for design and validation of the event processing application, while the run-time EPN is useful for control and management of the run-time.

More about EPNs and their components - in later postings.