XTP, Distributed Event Processing, and Distruptive Technologies

Hello everybody. Another experience of novice blogger - I had to start moderating comments, since I got spam attack of comments - so comments will not be seen in real-time anymore - but keep sending them anyway :-)

I have heard couple of times in the last few months talks from Yefim Natis
from Gartner who promotes now the concept of XTP - "Extreme Transaction Processing"
Yefim claims that XTP is a disruptive technology, in the way that it will be a new wave in enterprise application servers, he mentions in particular two disruptive technologies: dynamic grid architectures and event-based programming models. My friend Alan Lundberg asks in a recent blog what is the difference between CEP and XTP -- the short answer is that XTP is a framework that should have CEP capabilities but some other capabilities such as: transaction management, highly distributed environment etc... It is reasonable to think that some CEP applications will run on XTP, and maybe for some it will be overkill, but this depends on the ability of XTP to scale down as well as scale up. Another friend, Tim Bass (the person who talked me into writing this blog), has also recent blog about distributed event processing in which he said that "CEP engines should operate as cooperating intelligent agents" - and this assertion is the essence of the "event-based programming model" for XTP. I'll drill down into it in later blogs, but just point out some of the principles:

• In the event-based programming model - events and their flow are the most basic primitive
• Events are flowing among agents - agent may be intelligent agents that are doing "smart" things, or just software agents doing predetermined computations.
• The wisdom of the system is both in the agents and in the network - the network itself is intelligent and getting decisions.

This is another view of the world than the current programming models that are based around objects (and recently services), and may act as a disruptive programming model towards them, since having event as a first class citizen changes things.... These are preliminary ideas, but I find this idea intriguing and fascinating, so I'll probably dig into it deeper.

More - Later.

Revisiting the "classification of event processing applications - part I"

The picture about is an attempt to classify styles of beards, my first beard has been "goatee and mustache", but after a short period I moved to full beard and persists for more than 30 years, unlike people who grow and remove beards, for me it is an integral part of my personality. Anyway, talking about pictures - for some strange reason the almighty Google has chosen the dog picture I've posted in one of the previous blog as the first answer to the query about "dog" in Google image points at this blog... the wonders of search technology :-)

Now - back to the event processing business, following a comment by PatternStorm (my dear colleague Claudi), I have checked the current definition "complex event processing" in the glossary is somewhat wider than I have remembered, thus, according to it - derived events are subset of complex events, where I thought that some derived events (like enrichment of a single event) are not complex event. Anyway, I am revisiting the classification done in the previous blog, and avoiding the terms "simple event processing" and "complex event processing" - since indeed it has nothing to do with the intuitive meaning of the terms : simple and complex. However, the three classes in terms of type of processing still exists - so I'll repeat them and try different names (but names are always a problem - so any help will be appreciated !).

• raw event processing: operations on events that do not change the events - just filter and route them.
• transformed event processing: aggregation and enrichment - this is a type of processing that transform the event. Examples: enrichment it is transformed by adding attributes taken from an external store (e.g. database), translation - create an event with different format / contain translated attributes...
• Pattern-based event processing -- detecting pattern - and derive events based on the events that participate in the detected pattern.

As said, this is just one of the classifications. More discussions about other classifications - later.

classification of event processing applications - part I

You may need a good eyesight to look at this slide, so let's skip it for now, and return to it later.

Paul Vincent, in his recent blog tells us that he is struggling with classification - and talks about "simple event simple processing" to "complex event complex processing". Well - here are some thoughts that may ease the struggle. You have noticed that Paul Vincent constantly writes about Complex Event Processing and I constantly write about Event Processing - are they synonyms and I am just omitting the C ? the answer is - no!, Complex Event Processing is a subset of Event processing, according to one of the classifications, but there are some other types of Event Processing. Back to the classification issue -- there are different dimensions by which we can classify "event processing applications" :

(1). By processing type - and CEP is part of this classification

(2). By goal - what they intend to achieve

(3). By non-functional requirements

(4). By type of events that are processed.

Maybe more classifications. So in this blog I'll concentrate in (1) - "by processing type".

The slide above is showing a relatively simple application that checks orders against supplies, in the following way:

• Supply may arrive in multiple ways - there is an aggregation function that aggregates all events about arrival of supply.
• When all the supply arrived it is matched against the original order - this is a "pattern matching" - in this case - sequence of two events (order, aggregated supply) with some matching condition
• If does not match - then - there is a "non matching" event reported and then enriched from a database in more details about the order
• The enriched event is then reported to the supplier, and some decision is being taken.
• A decision is taken by "supplier type" - thus there is some filtering and routing based on the type.

The functions that we have here are : aggregation, enrichment, pattern detection and filtering.

They belong to three different families:

• simple event processing: operations on events that do not change the events - just filter and route them.
• mediated event processing: aggregation and enrichment - this is a type of processing that transform the event. In enrichment it is transformed by adding attributes taken from an external store (e.g. database), while in aggregation - multiple events create one event with (in this case - sum of products in the different supplies).
• complex event processing -- detecting pattern - here the pattern is: order occurred and later all supplies arrive - but the quantities don't match. This is very simple pattern, but it is complex event processing, because it deals with complex events.

Some more observations:

• The border between mediated event processing and complex event processing (which both derive new events) is that mediated event processing does not detect a pattern, and does not need to keep the raw events in a state, however, it may be statefull (aggregation is statefull, but not complex event processing - the event that is kept at the end is simple.
• The names may not be very good - since pattern of complex event processing can be quite simple, while filter of simple event processing may be rather complex - however, the "complex" name became pervasive in the industry, and derives the others...

More about the other types of classifications - later