More on the nodes in the event processing network

This is the illustration of the event processing network for the Fast Flower Delivery example that accompanies the EPIA book we are writing, don't spoil your eyes reading it. We'll provide the readers with an open source editor to the EPN meta-language we have developed (developed by Ronen Vaisenberg, a Ph.D. student in UCI). I'll have posting about it with more instructions in one of the next postings.

In this posting I would like to go back to the nodes in EPN and discuss why do we need multiple nodes and what are these nodes. One version is that there is a single type of node called EPA -- event processing agent, but my preference is to have multiple nodes, since they have different roles. Here are the types of nodes in our EPN model:

• Event Processing Agent (EPA): a node that gets one or more events as an input and produces one or more event as an output.
• Event producer: An entry point to the event processing system that emits events to the EPN.
• Event consumer: An exit point from the event processing system that receives events from the event processing system
• Event channel: A node that make routing decisions within the EPN, I have recently discussed event channels in this Blog.
• Global states: This is a node that holds various type of state (reference data, global variables)
• EPN node: An EPA can represent an EPN, and thus make the EPN recursive.
  

One of the frequent questions is whether an event producer which is also an event consumer is considered an EPA. In my opinion, the answer is no. Event producers and consumers operate on the borderline of event processing network, and is actually have have two nodes one for each of its roles. The information that a consumer is also a producer is important for traceability and lineage, but we'll discuss this aspect another time.

Christoph Emmesberger, whom I met in Trento, has proposed to extend the notion of channel to provide aggregation and filtering, this idea is rooted in pub/sub systems where event processing seems as an extension of some event processing functionality. However, when we deal with event processing system, I prefer to make a clear distinction between event processing agents that is doing -- filtering, aggregation, transformation and event pattern detection , and leave channel as a routing mechanism. thus we'll ensure proper separation of concerns, and have a cleaner model. We are now finishing the chapter about pattern detection, and I'll write about patterns soon.

On the boundaries of event processing

We are in the Passover vacation, to celebrate the biblical story of getting out from Egypt where the sea has split and people could move in the gap that was created, well -- I guess that at that time people just walked, but if it would have occurred today it might have looked like this.
Today I would like to write something about the "boundaries" of event processing, based on some discussions last week, related to writing a book about event processing. There are two issues related to the scope:

• Is pre-processing to emit event by the producer, and post-processing of events by the consumer are part of the event processing systems?
• Are pre-processing to obtain the event processing patterns that has to be monitored (i.e. using machine learning techniques) part of the event processing systems?

From the point of view of "event processing language", if we'll include the pre-processing and post-processing we'll have to extend the language to have the expressive power of any programming language, which will loose the focus on specific event processing functionality. Thus, while an application may require pre and post processing, this is typically outside the "event processing network". The main point of using "event processing language" and not hard-coding the event processing functionality in Java, C# or any other imperative general-purpose language is using higher level abstraction. As an analog, before the days of SQL we had to read from the database, loop over a record, and evaluate the conditions in hard-coded way, SQL did not provide anything we could not write in Cobol or PL/I (the languages of that time...), but just provided a more concise way to write it. The situation in event processing is similar, we can write something that is specified as:
" Match a pattern of events which is a conjunction of type E1, E2, E3 that refer to the same person and all occur within one hour since an event of type E0 for the same person, if there are several instances of E1, E2, E3 take the most recent of each at the point that the match occurred, and if there are multiple matches within this same time interval, ignore all but the first". Of course, one can write it in Java, but a language that enables to write this pattern in less than 1 minute is more cost-effective.

Back to the scope -- pre and post processing of events and patterns are not part of the event processing system, and typically done in different technologies. This does not say that they are not important, sometimes the pre-processing of events is more complicated than the event processing, especially since it is hard-code.

More on this - later

On Event Flow and Decoupling

This is a simulation of an anesthesia workstation, it can simulates various cases that creates a flow of events that are flowing inside this configuration, e.g. what happens when there is power failure.

I was recently asked if there is no contradiction between two concepts:

• The decoupling concept, each event processing agent is independent, it subscribes to some event, publishes derived events, and is independent in any other agent, furthermore, it is decoupled and does not know anything about the other agents.
  
• The Event Flow concept, in which there is explicit modeling of event flows
  

My answer is that there is not really a contradiction, since these two principles are in two different levels. The decoupling is in the execution level, event processing agents indeed do not need to communicate with one another, since there is no RPC or any other synchronous communication among them. The event flow concept exists in the modeling and management layers. In the modeling layer, there should be a view of the entire "event processing network" to enable that that the orchestra is playing together; in the management level, there should be a possibility to trace back the provenance of a certain decision or action, or trace forward the consequences of any event, however this still does not require to violate the decoupling in the execution layer, that's the beauty of model driven architecture... more - later.

Event Processing Platform (EPP) --- yes, but...

STAC, as cited in the popular Blog of Tim Bass, has determined that the correct name that should be used for event processing products is EPP (Event Processing Platforms) rather than CEP. I actually like the term EPP, actually I used this term before, so I should have copyrighted this name...

However, EPP should be used in the right meaning.

In event processing software there are two different things:

• Platforms that provide the "programming in the large" -- indeed a container in which different types of functionality can be plugged in.
• EPA implementation Software - that performs the actual event processing work - e.g. pattern matching, enrichment, filtering etc... This is the "programming in the small" (I called it "event processing engines", but not convinced that this is the best name)
  

In the EPTS glossary terminology, Platforms implement "event processing networks", while the other type implement various types of event processing agents.

These are not the same, there are vendors who provide platforms, but use other software to implement agents For example - BEA provided a platform, and used Esper for various functions, if I am not mistaken this is also true for Event Zero, IBM's Infosphere Streams is also a platform -- all are indeed platforms. Some products provide both the EPN platforms and various EPA implementations, some provide just the EPA implementation and runs on various platforms (or as a centralized stand alone engine).

So, while I agree that the EPN implementations are platforms, I am not sure that the EPA implementations are also platforms, and we probably may need a different name (engines ?, not sure)...

And one sentence about the term CEP. As I have written several times, I am not a big fan of this term at all, I am consistently talking about event processing and not about complex event processing as the name of the discipline that this Blog covers. However, this reminds me that once I have been a member in a Hebrew technical terminology committee, and one of the terms that came for discussion has been "real-time", for some strange reason, in Hebrew it was translated literally as "true time", and when it came to write the official glossary endorsed by the Israeli Academy of Hebrew Language, their representative who knew Hebrew linguistics, but not computer science, insisted the the Hebrew word should be a true translation of "real time", giving a long talk about "real numbers" and other real stuff. I argued that --- from linguistic point of view he is probably right, but, the scientifically wrong name is already well-known in the industry, and decision on another name would not be accepted by the public. After long discussion he agreed to include my wrong version as an alias to his true version. You can guess which of the two is still being used for "real-time" in Hebrew.
The moral of this story is that it may be too late to change names, since the name CEP has been accepted in the industry for any type of event processing system, whether or not it is scientifically accurate, and as somebody said once -- resistance is futile...

I'll continue to use "event processing", will use "event processing platform" for a platform, and still looking for a term for the "EPA implementation" (engines or otherwise). But -- my guess is that the people that use CEP to denote any type of EP will continue doing it, since this name may already penetrated to the ground. More - Later.

On Event Processing Networks

Back in my office, with the machine-made coffee; starting the day by reading some stuff on the Web, and first I've seen David Luckham's request to write some prediction about the CEP market in 2009. It seems that I've misplaced my crystal ball, which probably means that I am not in the prophecy business recently. While there are things that are beyond our control, I think that the approach taken by Paul Vincent to talk about challenges is more constructive.

I agree that the one of the challenges is to define the borders of the area -- like other areas that have determined clear definition of their scope -- and maybe partition to sub-types. There are other challenges of interoperability -- how connectivity to many producers and many consumers of various types can be achieved, and also interoperability between event processors that can send events to each other. I view the EPTS work groups that will be launched hopefully later this month (and those who continue from the pre-EPTS era) as vehicles for the community effort to advance in these areas: the use-cases work group in defining the various sub-types, the language analysis one in working on required functions, the interoperability analysis one on interoperability issues, meta-modeling on the modeling perspective, and of course the glossary and the reference architecture as pivots in defining terms and relationships among them. We shall not finish all work in 2009, but my challenge to the community is to achieve significant progress in all of these during 2009, and make it the year in which much of the discipline will be defined.

In addition, I have also read with interest Philip Howard's short article on "Event Processing Networks" (Below is Philip's picture on the Web)

I have received it in direct Email from David Tucker, Event Zero's CEO, and later also found it on David Luckham's site. Anybody who reads my Blog may realize that I view the EPN as the
basis of the conceptual an execution model for event processing. Anybody who reads Philip's article may infer that EPN is a new concept invented by Event Zero, and this is not really true; Though Event Zero is indeed one of the first companies to implement an EPN based solution.
The glossary defines EPN as: set of event processing agents (EPAs) and a set of event channels connecting them. The glossary definition is very general and there can be many implementations that fit this definition. One view of EPN is as a conceptual model and implement it using existing tools, another view of EPN is as an execution architecture. With the few implementations of EPN right now we see the known phenomenon of the "Babylon tower" that I have written about in the past -- each implementation chooses its own set of primitives (in this case -- agent types).


The benefits of the EPN model is in its relative simplicity, generality, and its natural support in distributed environment and parallel processing (not for free, some more wisdom is required here!). My view is that the concept of EPN should be in the center of the event processing community efforts mentioned before --- from the fundamental theory to the execution optimizations. I'll write more on that in later Blogs.

On Decisions and Event Processing

Saturday morning at home, after busy week and one resting day (our weekend is Friday and Saturday, tomorrow - Sunday, is again long and busy working day...) I am back in the Blogland.
Today I am in reactive mood, so will join a discussion thread that started by James Taylor who seems to have discovered that "event rules" and "decision rules" are not the same. Since this has been a reaction to a previous posting by Marco Siero, Marco actually agrees with him, saying that event rules and decision rules are not the same - saying that the "event rules" are in fact activation rules - saying WHEN a decision is needed. If they both agree then the thread of discussion can end.. -- nevertheless, I have some comments to this discussion:

1. The word "rules" is overloaded - there are many types of rules. Some rules deal with obtaining more facts (if the rules are based on logic it is called inference, and if the rules are based on data models it is called derivation), some with activating actions. Moreover, there are rules which are assertions about what is allowed and what is not (constraints) which are none of the above.

2. People have used the term "rules" to denote -- detect a pattern in events --- due to several reasons, one of them seems that rule has been the most closely known term that they could anchor at.

3. Today, when "event processing" is known by its own right, the use of term rules, when it is not needed is just confusing, and it is better to use the correct terms for event processing types.

4. I personally used the term "rules" for several years, but in the last three years have avoided doing it. It is difficult to get people change terminology, but sometimes it is required.

This, however, was just a comment on the term "rule", but let's get to the more interesting part -- as the title of the posting hints -- relationships between decisions and event processing.

The active database people have coined the term "active rule" which is also known as ECA rule, following its parts (event-condition-action). One can look at event processing network as an extension of ECA, in the sense that there is an entire processing done to get to the event that we actually want to react to, a raw event can be filtered, translated, aggregated, enriched, and
combined with other events that satisfy a certain pattern -- at then the event at the edge of the network (AKA situation) is the one that triggers the action.

The question whether an event processing application is doing decision or just notification that decision is needed, is not dependent on the event processing part - but on the type of action, if the action is notification -- notify somebody that the situation has been detected, I guess that it is what Marco meant in "activation rules", however, if the action is an automatic action (or several possible automatic actions based on conditions), then the result of the event processing is certainly a decision. There are cases in which the event processing ends in notification, e.g. most cases of BAM, and there are cases in which event processing ends in automated actions, e.g. send order to renew inventory, block a certain financial transaction and many other examples.

It also should be noted that the process of "event processing" may include decisions about the detection process itself -- examples: diagnosis, fraud detection - require application of decision components in order to determine what is the situation that occurred, which is also manifests itself in the decision.

James Taylor makes more interesting assertion: Event rules are not the same as the rules behind business decisions, even if the same language can be used to express both. One is tied to the events and the event handling system, one is not. I am not sure what is "tied" to the events, as said, the "action" takes part outside the event processing network, and it certainly can be a case of hybrid system, in which an event processing network detects what happened and the action attached to it is a "business rule" that decides what the reaction should be based on the situation. More- Later.

On Event Processing Network and Transaction Processing

It is a holiday period, time in which we have four holidays during three weeks, and is quite a lazy time here, with many people taking vacations (like the second part of December - beginning of January in the countries with christian majority), due to holidays and some other events I'll see my office in the coming week only on Tuesday, but working a bit from home now...

In an IBM internal Email exchange this week with a person who does not really understand event processing, this person has seen some illustration of EPN (Event Processing Network) and wondered -- this seems like regular transaction processing ? what is the difference ?

Indeed - from bird's eye view everything looks like directed graph, such as the one shown in the top of this page, both transactional flow and EPN as well as many other things are expressed using a directed graph, however there is a major difference in the semantics of the graph.

In order to refer to a concrete example, let's take an EPN example taken from an application of remote patient monitoring.






The semantics of EPN means that a node in a graph creates events and then these events are consumed by other nodes in the graph, for example the "enrich" node takes a blood pressure reading and enrich it with indication whether the patient is diabetic, thus creates a derived event; this derived event is consumed by the node that is looking for pattern to alert physician. Without going to the application's details too much - we may also state that unlike a control flow, the pattern detection node does not start its execution when all of its predecessors have finished, since the pattern may look at multiple blood pressure measurements of the same patient, it may exist for longer period relative to the enrich node that is created and measurements anytime that there is a blood pressure reading of a new patient, so the graph does not show the control flow, moreover, these two nodes don't know each other and communicate through a router (channel) node. So there are some differences between event processing network and transactional flow:

1. The EPN graph does not represent control flow, but event flow.
2. In a control flow graph, typically the relationship between predecessor and successor nodes are "finish to start" (either "meets" or "after" in the Allen's operators that I'll discuss in a seperate discussion) which means that the predecessor's node must terminate in order for the successor node to start, in EPN, this may not be the case.
3. EPN does not necessarily be atomic (one node in the EPN may fail, but others continue - no "atomic commitment protocol" (e.g. 2PC) is applied
4. It also may not be isolated -- a node can emit events, while still continue working; even if it fails later - its emitted events may still be valid, if atomicity is not required.
5. EPN can be restricted to behave in a transactional way - this is an interesting observation, as transaction support violates the decoupling principle, however there are cases in which it is required (again, deserves some more discussion). More - Later.

On latency in event processing network

Packing, on my way to a family vacation in Anatalya, Turkey.

Following an interesting discussion yesterday about performance metrics -- it turns out that definition of latency in event processing network is quite tricky. The reason is that an event can move in multiple paths over the network, in some it is filterred out, in some it just getting into internal state of agent since it does not complete a pattern, and in some it may complete a pattern and trigger reaction, thus there are various ways to define the metrics here. This is important since optimization has to take into account the goal function - what is being optimized. More discussion on this area -- after I'll return. I'll be out of touch for a week.