On CICS event processing

Recently, IBM has announced "CICS Event Processing". A full presentation explaining what it is about, is available on the Web. As I have written many times in the past, the processing part of events is just a part of a bigger picture, that includes: producing the events before the processing, route the events to the right processing elements, and consuming the events by consumers. In some cases, devising the event processing application is the easier part of the work, and the more difficult part is to connect it to the rest of the world. Since a substantial amount of the world transactions are going through CICS, which is a rather old, but still alive and kicking transaction processor, then it makes good sense to take it as a place for instrumentation, and emit events that can be sent either to further processing or directly to a consumer or a dashboard. The event processing part of CICS perform simple and mediated event processing, e.g. filtering, transformation, enrichment and routing. For pattern matching it sends the event to an event processing engine. I think that we'll see more of the producer side event processing support, that will reduce the need to write ad-hoc adapters and make it more cost-effective to use. We'll also see the complementary part - the consumer side, on which I'll write in a later date.

On messages and events

The smiling face belongs to Gregor Hohpe who is known as the co-author of the enterprise information patterns book. Gregor has conducted a BoF session in the EuroPLoP conference that I have written about in my previous Blog. The original book that Gregor has written has been about messaging oriented middleware, and now he is trying in his spare time to write the second volume - in the BoF he presented one chapter of the book talking about - identifying that something has gone wrong and handling it. This, by itself, is an interesting topic that I've been involved in the past on some related stuff, but as the title indicates the title of today's Blog is about messages and events. Hans Glide has recently dealt in his Blog in the issue of event processing vs. message processing, and part of the DEBS 2008 conference has been dedicated to messaging related research. Thus, this is a good timing to write a few lines about event and message processing --- being back home from all these travels.

Message is a transport mechanism, and as such message processing deals with the delivery of messages, relation to communication protocols, handling "not delivered messages etc..., one of the branches of messaging is also handling subscriptions when the protocol is pub/sub. The evolution of pub/sub has some similarity to that of event processing - started with subscriptions to every message published on a channel, moving to content-based subscription, which seem to be the focus of much work in the pub/sub research community. Content based subscription follows actually the ECA paradigm, where each message is treated as an event, thus, we see influence from the event world on the messaging world.

So - is event processing just fancy name to message processing ? -- not really, message processing is a lower level implementation that can serve also as infrastructure to event processing. An event can be represented by message, but not every message represents event. An event denotes something that happens, while I can send an old picture as a message and it is not an event at all. Thus the difference is in the level of architecture. Having said that, there are some interesting intersections: much of the simple and mediated event processing are shared with message processing (and indeed some of the patterns I am using to describe it are taken from Gregor's book).

The simple and mediated event processing are not new -- they indeed exist 15-20 years, and issue part of the functionality of event processing. The "detected patterns on history of multiple events" is the newer part in event processing, and it seems that it also a step in the evolution of pub/sub thinking.

Note that the pub/sub community calls itself "distributed event-based systems", and I indeed believe that they are part of the event processing community...

On CEP agent in EPN

This logo shows that we are not alone in the world of TLAs - there are other who use this acronyms, once I have made a list of acronyms of EDA which has shown that this is not a very good choice of acronym, but an acronym these days is context-sensitive. Anyway -- talking about "event processing networks", which we now try to clean its definitions - we have talked about SEP (simple event processing) agents, MEP (mediated event processing), CEP (complex event processing) and IEP (Intelligent event processing). The names can still change, I am not happy about any of them, but it seem to be relatively intuitive, and can be explained to non-technical people quite easily. Anyway, this time I'll concentrate on what is CEP agent. CEP agent have two basic phases:

1. Detect patterns over multiple events (these can be either multiple events of the same type, or multiple events from multiple event types - some are doing distinction here, but I don't see the rationale behind it) - this creates a collection of event-instances that satisfy the patterns.

2. Derive events as a function of the collection of events created in phase 1.

Some comments:

• There can be cases in which one of the two capabilities is used in a degenerated form. The pattern can be trivial (take all events without any pattern), but still a derivation (e.g. count, calculate the average of a certain attribute etc...) is required; on the other hand - the pattern can be complex, however the derivation is trivial (just a concatenation of all participating events).
• The term for the result of phase 1 - can be called "composite event" or "complex events", the two are not exactly synonyms, according to the glossary, as composite event is a syntactic term that is created by operators from primitive events, whereas complex event is a semantic term that includes aggregation of events that may not be obtained from patterns - however, the definition of phase 1 results is consistent with both.
• A CEP agent always lives within a context (temporal, spatial, semantic) -- most common type of context is a time-window, but the concept of context is much wider.
• The semantics of a CEP agent need to be fine-tuned in order to pick the instances of events participating in patterns relative to other possibilities which require to define policies - I'll talk more about semantics fine-tuning is a later postings.
• The result of a CEP agent is one or more derived events (consistent with other types of agents).

Why I prefer to use "event processing" with prefix, infix or suffix - a subjective tour of acronyms

Recently there has been more discussions about terms and acronyms, I am not sure that this is so important issue to spend much time on, but before moving to a more interesting points, I would like to provide some personal thoughts about acronyms in this area.


First, as you can see from the Blog's name, I prefer the term "event processing" with any prefix, infix, or suffix. The reason is that I view it as a name of a discipline and not as trend. Disciplines typically consist of two words: signal processing, information retrieval, machine learning, software engineering etc.. although there are exceptions. Three letter acronyms AKA TLA, are typically not names of core disciplines but of other things - protocols, architectures, trends etc..


Historically, when the first "event processing symposium" (which created EPTS) has been established we needed a name - the original founders were - David Luckham, Roy Schulte, Mark Palmer (from Progress Software) and myself. David, of course, thought that CEP is an appropriate name for the discipline, while Mark proposed ESP - "Event Stream Processing" since he did not like the word "complex" (read further about it). Roy and mysrelf proposed to take the part that both agree "event processing". Both David and Mark were not completely happy, but agreed, thus we advanced with the name "event processing symposium" and used "event processing" ever since.



Getting back to history - I have prefered to use the name "active technologies" being a veteran of the active database community, and although the autonomic computing community adopted the "active" term and had conferences named "active middleware services", this name actually did not get into the main stream, David Luckham used the term "complex event processing" in his famous book that used the term. The term "complex event processing" has ambigious meaning - one interpretation is that this is processing of complex events, where complex event is an event that consists of more than one event (analog to complex object), the other interpretation is that this is complex processing of events. I have started to use the term CEP in 2004 to differentiate such functionaity from "event correlation" in system management since there has been some confusion in IBM around this terms. I also made a modest contribution to get the name CEP known by giving a tutorial in ICWS in July 2004, attended by many people, whose common denominator has been tht they have not heard this term before. Anyhow - there are two school of thoughts around CEP


Interpretation one ("the monolithic approach") : CEP = EP, everything is a subset of CEP.
Interpreation two ("the layered approach") : EP is a collection of technologies, whereas CEP is one of them (a link in the chain). Some people takes the first interpretation, saying that "simple" event processing (whether it is simple event or simple processing) is a subset of complex event processing, the rational behind it that if an engine is capable of doing complex things it is surely capable of doing simple things. Interpreation two comes from Roy Schulte (Gartner) who introduced in December 2005 the following slide:

In this slide Roy Schulte talks about four types of processing (later he realized that the BPM one is of another category) - simple event processing (filter and route), mediate event processing (transform and enrich) and complex event processing (statefull pattern detector). This is consistent with a market view since there are products that do only simple event processing (messaging), other products who do mediated event processing (ESB) and CEP as the next layer as a stateful engine. I think that this approach is liked by those who are putting CEP on top of existing middleware, while the first ("monolithic") approach is liked by those who have stand-alone CEP engine. Anyway - the existence of this two approaches, and the fact that people may not understand that the other person is taking the second interpretation is causing a confusion.

Next acronym has been "event stream processing", the term "data stream manager" has been coined in Stanford in a similar meaning, but with SQL API, and continuing with other academic projects, and some descendent products (Coral8 is a descendent of the Stanford project). When Progress Software acquired Apama, Mark Palmer looked for an alternative word for CEP, since he was in the opinion that customers don't like anything labelled "complex", thus, he borowed the term "stream" although Apama's API is not SQL, and has not much to do with the academic stream projects and introduced the ESP term "Event Stream Processing" (which was dropped later). In response, David Luckham published an article to defend the "complex" word, starting with the words: "some people, I'm told, get scared when they hear the word complex, as in complex event processing.... start with the basic question, is life simple ? most people when asked about it will truthfully answer no...." and the rest you can read yourself. It seems that David has won this battle -- all vendors (including the SQL oriented ones) at some point or another have positioned themselves as CEP vendors, which also created some objections - by people who thought that it is important to diffrentiate between ESP and CEP, some saying that ESP is a subset of CEP, and some that these are completely different focus areas - as I have written before, there are many ways to define subsets of EP functionality, and I did not find any evidence that the one defined by this distinction (totally ordered events vs. partially ordered events) is the important one (in many applications we need both types for different purposes).

What other acronyms have flown around ? - well, Forrester at some point made a distinction between CEP and BEM (Business Event Management) that has been defined as - "a process of capturing real-time business events from multiple source and assigning them to the appropriate decision-maker for resolution based on the business context of the events". I have struggled to understand the distinction - maybe the fact that it deals with simple events, however, when they mention context - determining the context may by itself require CEP.

We, in IBM are using the term IEP (Intelligent Event Processing) to denote stochastic and intelligent reasoning beyond the deterministic pattern detection to CEP; this is consistent with the layer approach, the monolithic approach fans, view IEP as part of CEP.

The new term we heard this week from IBM is BEP (Business Event Processing) and this is intended to define event processing applications in which the business user can control the behaior (i.e. define and modify patterns without the help of a programmer), a topic I also discussed in the past.

Last but not least, some people in the academic community don't like the term "processing" which they think is too elementary and talk about "event-based computing" as the name of the discipline.
After this unusally long postings, my bottom lines are :

(1). The upcoming glossary should provide a consistent taxonomy of terms here - there is still much confusion about the names, and the glossary can be a good reference point,

(2). Personally, I still prefer to talk about types of functions and not about boundaries of names, however, I understand the importance of branding.

(3). I still prefer the name "event processing" without prefix, infix or suffix - and thus continue to use this name.

(4). Hopefully, this is the last posting I am writing on the *-E-P; E-*-P; E-P-* topic - I have more interesting topics to deal with.... more - later.

On situations

Hello from Zurich, got here in a train from Munich - and this is the famous train station of Zurich (in light), I have arrived in the dark, but has been in Zurich quite a lot of times before, tomorrow I am visiting the IBM Zurich Lab (which is actually located in Rüschlikon (a suburb of Zurich) home to four Noble prizes in physics. The hotel is 5 minutes walk from the train, but I forgot to print a map, and had to search a little. In the first half of the day I have participated in the continuation of the CITT conference, and earned my bread by giving a talk. After the talk I had some discussion with people about the term situation that I mentioned in the talk. I have borrowed this word from "situation calculus" in AI, and it fits well with terms in related domain such as: situation awareness. The discussion has been has been around one of the common misconceptions that situation is an alias to derived event. The answer is - no, these terms are not aliases ! Situation is some phenomenon that requires reaction (or notification). It is in the conceptual ontology of the consumer. Now - let's check the relationships with event:

(1). A raw event that is reported by a producer is a situation in the consumer's terms - in this case, only routing is required ("simple event processing").

(2). A raw event is reported by a producer has 1-1 correspodence to a situation in the consumer's terms, but more information or translation is required - in this case, the situation is created by mediated event processing.

(3). The situation occurs in a deterministic way once an event pattern is identified, and it is a functions of the events participating in the pattern - in this case the situation is created by complex event processing.

Note that the situation is not the derived event - but the derived event in this case is an indication that the situation is detected, note also that not all derived events are situations, derived events that are consumed by other agents, are not situations, only those that are consumed by event consumers.

However, situation detection may not be purely deterministic - and here we are getting to "intelligent event processing". Example: the pattern is only an approximation, example: the pattern is - "an event E1 happens at least 4 times within an hour". Now, if the event E1 happens 3 times and the pattern is an approximation, there is some probability that the situation did occur. This is even before looking at probabilistic events.

More about uncertainty in event processing - later.

IBM WebSphere CTO sees CEP as SOA's 'next big thing'

In Israel there are sometimes news items that are forbidden by the military censorship, but if broadcasted by foreign sources - one can quote the foreign source and broadcast it in Israel.

I have not referred in this blog to IBM plans in this area, since they have not announced yet by the proper channel, but I may quote the interview with IBM Websphere CTO, Jerry Cuomo

in which he stated that he sees CEP as SOA's next big thing -- and promised both new technology and additions to existing ones. Stay tuned for more information on IBM and Event Processing.

On the envelope for CEP

In two recent blogs - Mark Tsimelzon from Coral8 argued for CEP server vs. embedded CEP libraries, while Paul Vincent from TIBCO argued for the need for infrastructure stack outside the

CEP engine.

Since I avoid making product evaluations in this Blog, I'll talk about the principle.

Conceptually we have a model of "event producer" that produces the events, "event consumer" that consumes the event, and the "event processor" which processes the events and stands in the middle. There are two questions about envelopes:

(1). Is "event processor" an embedded capabilities inside applications or an server.

(2). If "event processor" is a server, is it independent server or part of a larger middleware ?

The answer, as anything is not binary (zero or one), there are cases in which there is a need to have event processing as embedded capabilities (e.g. inside pervasive devices), but Mark is right that the big majority of "event processors" are tend towards the server.

The second question is more interesting --- we have today both stand-alone server and servers that are part of a larger middleware, the rational behind being part of a larger middleware, stems from the fact that event processing is not isolated and has various relations with different applications in the enterprise, it is true that the loose-coupling nature of event driven architectures eases the task of separating it from the applications, but still the integration is the most costly part of building event processing applications, and means to ease the integration has already built into application integration middleware, and if the event processing server is a stand-alone one, there is a need to re-invent this integration, as Paul Vincent rightly say: every $ a CEP vendor spends on middleware integration is a $ less on interesting CEP functionality.

Furthermore, there are some event processing infrastructure and functions (pub/sub, routing - for simple event processing, and ESB mediations like - enrichment, transformation etc, that are already there). Thus, it seems that the ROI will be higher if event processors will be implemented on top of a "middleware stack".

An interesting observation is that from the point of view of application integration middlewares - event processing is becoming a key feature, and there are already some predictions that the standard event processing programming model (which is still not there!), will be the basis for application servers of the future, e.g. Gartner's XTP that I have once discussed and should discuss more.

CEP and the story of the captured traveller

Reading the recent posting of my friend Tim Bass entitled "CEP and the story of the Fish" I decided to answer with another story (from the other side of Asia) :

A traveller went in the jungle somewhere on the globe and unfortunately was captured by a tribe that is still using ancient weapons. He is brought to the chief, and the chief says - " You have trespassed into the tribe's territory, which is punishable by death, however, I am a very curious person, if you'll show me something I haven't seen before I'll let you go"; our unlucky traveller started to look in his pockets and the only meaningful thing he found was a lighter, so he took his chance, showing it to the chief saying: "this thing makes fire", however, since he was in under a big pressure, he pressed once - no fire, pressed twice - no fire, in the third time the lighter indeed has produced the promised fire, the chief did not hesitate and said "let him go", so our relieved traveller muttered to himself - "I knew that they have not seen a lighter", but surprisingly to him the chief said - "oh, I have seen many lighter, but a Zippo lighter that does not light in the first time I have never seen".

When someone disagrees with somebody else, it is very easy to assume that my point of view is right since I am smarter / knows more / more qualified / older / more experienced / generally always right etc... My preference is not to doubt the wisdom, experience or qualification of anybody that I am arguing / discussing / debating with, but make the arguments on the issue and not on the person who makes the arguments....

Enough introduction -- now for the main message of this posting, the term CEP (Complex Event Processing) has more or less agreed now in the industry to denote "computing that performs operations on complex events", where complex event is an "abstraction or aggregation of events". The term complex does not say that the processing is complex, but that it deals with complex events, as defined. Complex event processing is typically detecting predefined patterns that can be expressed by queries/rules/patterns/scripts and are deterministic in nature. Regardless if I think that this is the best term, I think that it is important to have common agreed terminology, otherwise we are confusing the industry, the customers (and sometimes ourselves). Now, Tim Bass claims that since event processing with stochastic/probabilistic/uncertain nature is more complex than what we call "complex event processing", we have to call this one - "complex event processing", and rename what we call "complex event processing" to be "simple event processing". Unfortunately, it is too late for that - and also not justified, again, since the "complex" in the "complex event processing" does not say that this is "complex processing of events" but that this is "processing of complex events" (very common misconception !). Bottom line: yes - there is another class of event processing capabilities that requires techniques from AI, machine learning, OR etc.. and that is not deterministic in nature; no - I don't think we should call it "complex event processing", we have suggested the term "intelligent event processing" which I have already referred to in previous posting , there are a variety of other postings that I have dedicated to terminology.

More - later

On simple events and simple event processing

This is a picture I have borrowed from Siemens, however, I'll use it talking about simple events and simple event processing. There is a constant confusion around the terms "simple" (and "complex") here due to the ambiguity of the phrase: simple (complex) event processing - does it mean: processing of simple (or complex) events? or does it mean simple (or complex) processing in events. In this posting, I am drilling down to the simplicity notion.

Let's start with simple event - I prefer to contrast simple event with composite event where the contrast is in the structure - composite event is a collection of simple events and simple event is an atomic event, nothing said about the processing yet.

Now - is there different processing for simple events and composite events ? - in principle no - there are some functions on collections that are not applicable for atomic events, but if we take a collection of simple events that has not been concatenated we can apply the same processing for them.

Thus - my preference is to attach the simple to the processing and not to the event type, and define simple event processing as simple type of processing, no matter what the event structure is. What are the characteristics of simple event processing ?

• processing is done on a single event - not looking on other events, but events are processed "one at a time".


• only types of processing possible are: filtering and routing


• filtering decides whether this event should be passed


• routing decides to whom this event should be passed

Basic pub/sub with filtering is simple event processing.

ECA (Event-Condition-Action) rule is also simple event processing -- this does not say that the event cannot be derived/complex/composite - but regardless of the history of the event, its structure, the reason why it is created, and its source the ECA rule still performs a simple event processing, and processes one event at a time, where the condition provides filtering, and the action is indeed routing to somewhere to do this action.

Furthermore, in many cases, the "simple event processing" is a preamble to the event processing network done by the event producer, or post-processing done by the consumer, however, it can still be part of the Event Processing Network.

More related concepts - in the next postings.

classification of event processing applications - part I

You may need a good eyesight to look at this slide, so let's skip it for now, and return to it later.

Paul Vincent, in his recent blog tells us that he is struggling with classification - and talks about "simple event simple processing" to "complex event complex processing". Well - here are some thoughts that may ease the struggle. You have noticed that Paul Vincent constantly writes about Complex Event Processing and I constantly write about Event Processing - are they synonyms and I am just omitting the C ? the answer is - no!, Complex Event Processing is a subset of Event processing, according to one of the classifications, but there are some other types of Event Processing. Back to the classification issue -- there are different dimensions by which we can classify "event processing applications" :

(1). By processing type - and CEP is part of this classification

(2). By goal - what they intend to achieve

(3). By non-functional requirements

(4). By type of events that are processed.

Maybe more classifications. So in this blog I'll concentrate in (1) - "by processing type".

The slide above is showing a relatively simple application that checks orders against supplies, in the following way:

• Supply may arrive in multiple ways - there is an aggregation function that aggregates all events about arrival of supply.
• When all the supply arrived it is matched against the original order - this is a "pattern matching" - in this case - sequence of two events (order, aggregated supply) with some matching condition
• If does not match - then - there is a "non matching" event reported and then enriched from a database in more details about the order
• The enriched event is then reported to the supplier, and some decision is being taken.
• A decision is taken by "supplier type" - thus there is some filtering and routing based on the type.

The functions that we have here are : aggregation, enrichment, pattern detection and filtering.

They belong to three different families:

• simple event processing: operations on events that do not change the events - just filter and route them.
• mediated event processing: aggregation and enrichment - this is a type of processing that transform the event. In enrichment it is transformed by adding attributes taken from an external store (e.g. database), while in aggregation - multiple events create one event with (in this case - sum of products in the different supplies).
• complex event processing -- detecting pattern - here the pattern is: order occurred and later all supplies arrive - but the quantities don't match. This is very simple pattern, but it is complex event processing, because it deals with complex events.

Some more observations:

• The border between mediated event processing and complex event processing (which both derive new events) is that mediated event processing does not detect a pattern, and does not need to keep the raw events in a state, however, it may be statefull (aggregation is statefull, but not complex event processing - the event that is kept at the end is simple.
• The names may not be very good - since pattern of complex event processing can be quite simple, while filter of simple event processing may be rather complex - however, the "complex" name became pervasive in the industry, and derives the others...

More about the other types of classifications - later