XTP, Distributed Event Processing, and Distruptive Technologies

Hello everybody. Another experience of novice blogger - I had to start moderating comments, since I got spam attack of comments - so comments will not be seen in real-time anymore - but keep sending them anyway :-)

I have heard couple of times in the last few months talks from Yefim Natis
from Gartner who promotes now the concept of XTP - "Extreme Transaction Processing"
Yefim claims that XTP is a disruptive technology, in the way that it will be a new wave in enterprise application servers, he mentions in particular two disruptive technologies: dynamic grid architectures and event-based programming models. My friend Alan Lundberg asks in a recent blog what is the difference between CEP and XTP -- the short answer is that XTP is a framework that should have CEP capabilities but some other capabilities such as: transaction management, highly distributed environment etc... It is reasonable to think that some CEP applications will run on XTP, and maybe for some it will be overkill, but this depends on the ability of XTP to scale down as well as scale up. Another friend, Tim Bass (the person who talked me into writing this blog), has also recent blog about distributed event processing in which he said that "CEP engines should operate as cooperating intelligent agents" - and this assertion is the essence of the "event-based programming model" for XTP. I'll drill down into it in later blogs, but just point out some of the principles:

• In the event-based programming model - events and their flow are the most basic primitive
• Events are flowing among agents - agent may be intelligent agents that are doing "smart" things, or just software agents doing predetermined computations.
• The wisdom of the system is both in the agents and in the network - the network itself is intelligent and getting decisions.

This is another view of the world than the current programming models that are based around objects (and recently services), and may act as a disruptive programming model towards them, since having event as a first class citizen changes things.... These are preliminary ideas, but I find this idea intriguing and fascinating, so I'll probably dig into it deeper.

More - Later.

Revisiting the "classification of event processing applications - part I"

The picture about is an attempt to classify styles of beards, my first beard has been "goatee and mustache", but after a short period I moved to full beard and persists for more than 30 years, unlike people who grow and remove beards, for me it is an integral part of my personality. Anyway, talking about pictures - for some strange reason the almighty Google has chosen the dog picture I've posted in one of the previous blog as the first answer to the query about "dog" in Google image points at this blog... the wonders of search technology :-)

Now - back to the event processing business, following a comment by PatternStorm (my dear colleague Claudi), I have checked the current definition "complex event processing" in the glossary is somewhat wider than I have remembered, thus, according to it - derived events are subset of complex events, where I thought that some derived events (like enrichment of a single event) are not complex event. Anyway, I am revisiting the classification done in the previous blog, and avoiding the terms "simple event processing" and "complex event processing" - since indeed it has nothing to do with the intuitive meaning of the terms : simple and complex. However, the three classes in terms of type of processing still exists - so I'll repeat them and try different names (but names are always a problem - so any help will be appreciated !).

• raw event processing: operations on events that do not change the events - just filter and route them.
• transformed event processing: aggregation and enrichment - this is a type of processing that transform the event. Examples: enrichment it is transformed by adding attributes taken from an external store (e.g. database), translation - create an event with different format / contain translated attributes...
• Pattern-based event processing -- detecting pattern - and derive events based on the events that participate in the detected pattern.

As said, this is just one of the classifications. More discussions about other classifications - later.

classification of event processing applications - part I

You may need a good eyesight to look at this slide, so let's skip it for now, and return to it later.

Paul Vincent, in his recent blog tells us that he is struggling with classification - and talks about "simple event simple processing" to "complex event complex processing". Well - here are some thoughts that may ease the struggle. You have noticed that Paul Vincent constantly writes about Complex Event Processing and I constantly write about Event Processing - are they synonyms and I am just omitting the C ? the answer is - no!, Complex Event Processing is a subset of Event processing, according to one of the classifications, but there are some other types of Event Processing. Back to the classification issue -- there are different dimensions by which we can classify "event processing applications" :

(1). By processing type - and CEP is part of this classification

(2). By goal - what they intend to achieve

(3). By non-functional requirements

(4). By type of events that are processed.

Maybe more classifications. So in this blog I'll concentrate in (1) - "by processing type".

The slide above is showing a relatively simple application that checks orders against supplies, in the following way:

• Supply may arrive in multiple ways - there is an aggregation function that aggregates all events about arrival of supply.
• When all the supply arrived it is matched against the original order - this is a "pattern matching" - in this case - sequence of two events (order, aggregated supply) with some matching condition
• If does not match - then - there is a "non matching" event reported and then enriched from a database in more details about the order
• The enriched event is then reported to the supplier, and some decision is being taken.
• A decision is taken by "supplier type" - thus there is some filtering and routing based on the type.

The functions that we have here are : aggregation, enrichment, pattern detection and filtering.

They belong to three different families:

• simple event processing: operations on events that do not change the events - just filter and route them.
• mediated event processing: aggregation and enrichment - this is a type of processing that transform the event. In enrichment it is transformed by adding attributes taken from an external store (e.g. database), while in aggregation - multiple events create one event with (in this case - sum of products in the different supplies).
• complex event processing -- detecting pattern - here the pattern is: order occurred and later all supplies arrive - but the quantities don't match. This is very simple pattern, but it is complex event processing, because it deals with complex events.

Some more observations:

• The border between mediated event processing and complex event processing (which both derive new events) is that mediated event processing does not detect a pattern, and does not need to keep the raw events in a state, however, it may be statefull (aggregation is statefull, but not complex event processing - the event that is kept at the end is simple.
• The names may not be very good - since pattern of complex event processing can be quite simple, while filter of simple event processing may be rather complex - however, the "complex" name became pervasive in the industry, and derives the others...

More about the other types of classifications - later

Seperation of concerns and Event Processing Language

In one of the previous blogs I have cited the "hammer and nail" blog written by Louie Lovas from Progress and said that I don't have the honor to know the author personally, so he has sent me a note introducing himself - very nice of him to do that ! Mr. Lovas recently has written another blog entitled: "Hitting the nail on the head", in this blog, the opinion is that the current trend is separation between the presentation layer, logic layer and data layer, and that using SQL for event processing is going backwards - since it mixes the business logic layer and data layer - if I understood the reasoning correctly. I agree with the end result of the author about the fact that SQL may be inappropriate language to express some event processing patterns, and I have written about it before, I also big fan of "separation of concerns" and other separations as a software engineering and architectural practice. However, these two things are independent, in the past, when we worked on active databases, this has been indeed under the concept that business logic is embedded inside the database logic, but, this has nothing to do with the API. We can still have separation to architectural layers on one hand, and share the same API, on the other hand; there are even some benefits in doing that - such as reducing the number of APIs. Thus - from the architecture point of view, I don't see any problem in using SQL-oriented language for event processing; the same (or similar) API can appear in various architecture layer (and in various roles).


the problem, as said is pragmatic, and stem from the SQL semantics and its fitness to event processing. In fact, in the course I am teaching we'll probably do with the students some experimentation on that.

More - Later

Teaching an event processing course

Since nobody ever took a picture of me teaching a class - I have brought the closest thing, a picture of me giving a talk in a conference. Anyway, this week I have started to teach an event processing course at the Technion. It is a strange situation now, since the faculty members are striking, they did not open the school year, but since I am an adjunct professor (the official title is: adjunct senior teaching associate) I am not striking, but they may close the campus at some point - so not sure what will happen. In any event, I have started the event processing course, and asked the students to read the blogs in this area as a background material. The course is intended to be a blend of theory and practice, where on one hand I am teaching the theory of event processing (my interpretation of it, to be exact), and one the other hand the students will be required (in teams) to implement a project that uses some CEP engine and some infrastructure around it to demonstrate an application prototype. Going to be interesting, and I'll provide more details about things related to the class as the course progresses. The discussions will also provide me more topics to blog around. One of the EPTS pending activities is to create a portal for teaching material around event processing. I know that some of my colleagues around the globe like Mani Chandy and Rainer von Ammon are also teaching event processing courses, and there may be perhaps some others. So will try to get some updates on the other courses as well. More - later.

Agnon, the dog, playing and downplaying

My literature teacher in the 7th grade has told the class one day a story about her days in as a university student; one of the literature classes in the university has been dedicated to Agnon, the only Hebrew writer who received Nobel Prize in Literature (Agnon's picture is in the left hand side). One of the famous stories of Agnon has a dog (such as the one in the right-hand side) as its hero; thus, the class of students at the university had discussed this story and had a debate about what does the dog represents, one group of students thought that dog is a symbol for Agnon itself, and one thought that the dog is a symbol for the entire Jewish people. Since Agnon was still alive those days, they have decided to send a couple of students to meet Agnon, set up a meeting in his house (well - Email questions were not on the radar at that time), and explained the dilemma - what does the dog represent? Agnon was somewhat amused to hear about the debate, and told them briefly - "I don't know what you are talking about, the dog represents a dog".

Going back for the childhood Reminiscences to the contemporary issues - I have read with interest the recent blog of Paul Vincent about "CEP vs. Business Rules" in which he mentioned me by reference (and not by name) saying: "Others seem to downplay the idea of rules for CEP". I have a substantial respect for Paul, and generally agree with most of the assertions he made in that blog. Let me add some more insights into this issue - I am using this blog to reflect about event processing, and make the point the "event processing" is a discipline that stands on its own feet, and is not a footnote to anything else - be it databases or business rules.

CEP functionality can indeed be implemented in either SQL or in an extension to business rule language as some of the vendors successfully demonstrated. It can also be implemented in half a dozen different ways (at least). Since I am trying to investigate "event processing" as a discipline, and not a specific implementation, thus is the play, and indeed I downplay specific implementations for now, and try to think on the fundamental issues. I may return in future blogs to talk about what type of implementations fit different types of functions/applications - which is a difficult question, and I don't really see that there is an issue of "CEP vs. Business rules" as these are conceptually orthogonal issues (and I agree that in practice there are various relations), and Paul's blog does a tremendous job of highlighting the differences and relations between the two.

For now - my dog is "event processing" and it represents "event processing" and not other concepts that are not fundamentally necessary to understand what "event processing" is.

More - Later.

Fallacies in implementation of event processing solutions

Back in Israel... the flight that has taken me back home suddenly landed in Athens, instead of continuing to its destination, it turns out that one of the passengers left a laptop on a vacant seat and went to sleep elsewhere in the aircraft, some passengers attracted the attention of the crew, and after some consultations, they decided to land in the nearest airport to have security people check the laptop, after landing the laptop owner woke up and retrieved his laptop, and the aircraft could take off again (well - not so fast, took almost an hour to take off again) - strange story (has been in several variations in all daily newspapers in Israel)... a fallacy in (human) event processing system. Talking about fallacies, recently Hans Glide in his blog talks about "failure scenarios" where EP solutions may fail to produce correct results. This reminds me of previous work I have done in the area of derived data using active databases, where in a relatively old paper I have looked at several possible fallacies in active databases - the analysis is also valid for event processing, here are some examples:

1. non deterministic execution due to conflicting rules.

2. Inability to reconstruct the derived data from the raw data since multiple rules derive the same data-element

3. Redundant updates in case of inter-connected derived updates, e.g. the same data-element is updated several times within one path

4. Derivations may result in infinite loops

The paper also shows that it is possible to eliminate these problems using a specific execution model.

Taking it back from active databases to event processing - in active databases, this paper has been in the framework of deriving data, while in the event processing cases we derive events - however, some variations of the same phenomena can occur in event processing - certainly non deterministic behavior can happen, and some other fallacies as well.

Hand Glide mentions that each event processing implementation project should pay attention to these - and indeed event processing application debug is more complex than the debug of regular applications due to inter-relations between rules/queries/derivations.

IMHO - it is a responsibility of the EP vendors to provide semantic debuggers that will enable doing it. Some are already around, but more work is done in this area, the users should not be on their own here... more about validation of EP applications - in one of the next posts (I am promising many follow-ups, but new topics keep coming)...