On CICS event processing

Recently, IBM has announced "CICS Event Processing". A full presentation explaining what it is about, is available on the Web. As I have written many times in the past, the processing part of events is just a part of a bigger picture, that includes: producing the events before the processing, route the events to the right processing elements, and consuming the events by consumers. In some cases, devising the event processing application is the easier part of the work, and the more difficult part is to connect it to the rest of the world. Since a substantial amount of the world transactions are going through CICS, which is a rather old, but still alive and kicking transaction processor, then it makes good sense to take it as a place for instrumentation, and emit events that can be sent either to further processing or directly to a consumer or a dashboard. The event processing part of CICS perform simple and mediated event processing, e.g. filtering, transformation, enrichment and routing. For pattern matching it sends the event to an event processing engine. I think that we'll see more of the producer side event processing support, that will reduce the need to write ad-hoc adapters and make it more cost-effective to use. We'll also see the complementary part - the consumer side, on which I'll write in a later date.

On Parallel processing and Event Processing - MapReduce

The "In Action" series of Manning Publications which hosts the "Event Processing in Action" book that Peter Niblett and myself are writing, also hosting another book "Hadoop in Action" that deals with the Hadoop, the open source MapReduce framework. In the framework of my series of postings about parallel processing and event processing, I would like today to drill down into MapReduce and make some observations about its possible relations with event processing.

The "Hadoop in Action" book describes MapReduce in the following way:

MapReduce programs are executed in two main phases, called mapping and reducing. Each phase is defined by a data processing function, and these functions are called mapper and reducer, respectively. In the mapping phase, MapReduce takes the input data and feeds each data element to the mapper. Afterward, in the reducing phase, the reducer processes all the outputs from the mapper and arrives at a final result. In general, the mapper is meant to filter and transform the input into something that the reducer can aggregate over.

In the MapReduce framework you write applications by specifying the mapper and reducer. It’s instructive to understanding the complete data flow:

1. The input to your application must be structured as a list of (key, value) pairs, list(). This input format may seem open-ended but is often quite simple in practice. The input format for processing many files is usually just list(). The input format for processing one big file, such as a log file, is something like list().
2. The list of (key, value) pairs is broken up and each individual (key, value) pair, , is processed by calling the map function of the mapper. In practice, the key k1 is often ignored by mapper. The mapper transforms each pair into a list of pairs. The details of this transformation largely determines what the MapReduce program does. Note that the (key, value) pairs can be processed in arbitrary order, and the transformation must be self contained in that its output is dependent only on one single (key, value) pair..
3. The output of all the mappers are (conceptually) aggregated into one giant list of pairs. All pairs sharing the same k2 are grouped together into a new (key, value) pair, . The framework then asks the reducer to process each one of these “aggregated” (key, value) pairs individually.
   

All Lisp fans, like myself, can trace the origins of the Map and Reduce functions, as written in the description the main objective of MapReduce is to partition large collection of data in order to provide aggregations in each individual partition.

MapReduce as a model for parallel processing has many fans, but also some non-believers, most notably two of the prominent database figures - David DeWitt and Mike Stonebraker, who attacked MapReduce on the basis of programming model, pefromenace issues, and general hyping of old ideas see posting 1, posting 2, (both of them from early 2008) and their new SIGMOD paper benchmark comparing MapReduce to parallel databases.

Anyway, let everybody read both sides. It seems that MapReduce is becoming a religion, and like any professional religions of the past, I keep being egnostic, don't take anything as a religion, this issue is not black and white...

Getting back to my favorite topic, event processing. The two questions that arise are:

1. Are the functionality that MapReduce is intended to support, is part of, or useful, in any way for event processing?
2. Are the requirements of event processing for parallel processing are best satisfied by MapReduce ?
   

The answer to the first question is definitely positive -- transforming, filtering and aggregation are part of the event processing story ("mediated event processing"). Moreover, as I mentioned before, it may be useful for pre-processing events, so that the more sophisticated processing is done over aggregations and not over raw events, but can also be used in any phase on derived events.

The answer to the second question is probably negative - MapReduce kind of processing can be useful for "set at a time" operations, since it has data orientation, getting input from files, and providing output on files, and probably not so much for "event at a time" which makes the calculations incrementally. Furthermore, the "key data" partition may be too simplistic for context oriented partition that may be based on predicate (e.g. customers whose age is > 40), of course, this can be bridged, by putting the segments inside the event payload, but this has its price in flexibility. Also, the partition in event processing network is typically not to take up a problem and divide it, and then merge the results, but creating parallel branches of computation that are independent, which is slightly different thing.

Bottom line -- MapReduce can help in some but bot all of the parallel processing requirements for event processing, for other things, we need something else. I guess that the question of cost-effectiveness is based on the specific applications' loads and focus.

More about parallel processing and event processing - later.

On messages and events

The smiling face belongs to Gregor Hohpe who is known as the co-author of the enterprise information patterns book. Gregor has conducted a BoF session in the EuroPLoP conference that I have written about in my previous Blog. The original book that Gregor has written has been about messaging oriented middleware, and now he is trying in his spare time to write the second volume - in the BoF he presented one chapter of the book talking about - identifying that something has gone wrong and handling it. This, by itself, is an interesting topic that I've been involved in the past on some related stuff, but as the title indicates the title of today's Blog is about messages and events. Hans Glide has recently dealt in his Blog in the issue of event processing vs. message processing, and part of the DEBS 2008 conference has been dedicated to messaging related research. Thus, this is a good timing to write a few lines about event and message processing --- being back home from all these travels.

Message is a transport mechanism, and as such message processing deals with the delivery of messages, relation to communication protocols, handling "not delivered messages etc..., one of the branches of messaging is also handling subscriptions when the protocol is pub/sub. The evolution of pub/sub has some similarity to that of event processing - started with subscriptions to every message published on a channel, moving to content-based subscription, which seem to be the focus of much work in the pub/sub research community. Content based subscription follows actually the ECA paradigm, where each message is treated as an event, thus, we see influence from the event world on the messaging world.

So - is event processing just fancy name to message processing ? -- not really, message processing is a lower level implementation that can serve also as infrastructure to event processing. An event can be represented by message, but not every message represents event. An event denotes something that happens, while I can send an old picture as a message and it is not an event at all. Thus the difference is in the level of architecture. Having said that, there are some interesting intersections: much of the simple and mediated event processing are shared with message processing (and indeed some of the patterns I am using to describe it are taken from Gregor's book).

The simple and mediated event processing are not new -- they indeed exist 15-20 years, and issue part of the functionality of event processing. The "detected patterns on history of multiple events" is the newer part in event processing, and it seems that it also a step in the evolution of pub/sub thinking.

Note that the pub/sub community calls itself "distributed event-based systems", and I indeed believe that they are part of the event processing community...

On CEP agent in EPN

This logo shows that we are not alone in the world of TLAs - there are other who use this acronyms, once I have made a list of acronyms of EDA which has shown that this is not a very good choice of acronym, but an acronym these days is context-sensitive. Anyway -- talking about "event processing networks", which we now try to clean its definitions - we have talked about SEP (simple event processing) agents, MEP (mediated event processing), CEP (complex event processing) and IEP (Intelligent event processing). The names can still change, I am not happy about any of them, but it seem to be relatively intuitive, and can be explained to non-technical people quite easily. Anyway, this time I'll concentrate on what is CEP agent. CEP agent have two basic phases:

1. Detect patterns over multiple events (these can be either multiple events of the same type, or multiple events from multiple event types - some are doing distinction here, but I don't see the rationale behind it) - this creates a collection of event-instances that satisfy the patterns.

2. Derive events as a function of the collection of events created in phase 1.

Some comments:

• There can be cases in which one of the two capabilities is used in a degenerated form. The pattern can be trivial (take all events without any pattern), but still a derivation (e.g. count, calculate the average of a certain attribute etc...) is required; on the other hand - the pattern can be complex, however the derivation is trivial (just a concatenation of all participating events).
• The term for the result of phase 1 - can be called "composite event" or "complex events", the two are not exactly synonyms, according to the glossary, as composite event is a syntactic term that is created by operators from primitive events, whereas complex event is a semantic term that includes aggregation of events that may not be obtained from patterns - however, the definition of phase 1 results is consistent with both.
• A CEP agent always lives within a context (temporal, spatial, semantic) -- most common type of context is a time-window, but the concept of context is much wider.
• The semantics of a CEP agent need to be fine-tuned in order to pick the instances of events participating in patterns relative to other possibilities which require to define policies - I'll talk more about semantics fine-tuning is a later postings.
• The result of a CEP agent is one or more derived events (consistent with other types of agents).

On Immutability of events

Israel is the land of milk and honey, but not the land of snow - snow is quite rare in Israel. This week the rare event of snow occurred in high mountains across the country - in the picture snow near one of the gates of the old city of Jerusalem. In the Carmel mountain ridge, where I live there was a little bit of snow, higher in the ridge, and some people went there and brought to the office bags full of snow... As I lived several years in the Philadelphia area in USA, I am personally less excited from snow - but it is a notable event here.

Today's topic relates to a question that Marco, fellow EP blogger, and the person behind rulecore, has asked about the previous posting, his question related to enrichment, but I'll extend it to the rest of the transformations : "event is something that is happened, and as such it is immutable, cannot be changed; Do transformations and enrichments break this model?".

The answer for this question is - no. Transformations do not break the model, and events are still immutable. According to the pure model - events cannot be altered or deleted, and when represented in an event store, it has to be an "append only" type of store.

Enrichment and transformations are, in fact, creation of new derived events, as a function of raw events. Thus, according to the pure model, transformed or enriched event is not the same event:

• It has a different type of event - with different structure.
• It has a different event-id.

An Enrichment example maybe - the event is: order, and it has an attribute that refers to customer. The enrichment function looks at the customer in some database, and fetches the values of : customer type (platinum, gold, silver, nobody) and customer_credit_limit.

The fact that there is a different event-id for the raw event and transformed event allows also the traceability to trace the transformation (maybe the problem is in wrong identification of the customer?).

There are some considerations that may, in practice, push towards not obeying to the pure model, and I'll talk about them some other time.

On Mediated Event Processing

I have mentioned the term "mediated event processing" in the past, but looking at previous postings - never explained exactly what it is.

So back to the educational mission of this Blog: mediations are exactly what they mean in messaging middleware or ESBs - transforming events. There are several types of mediators.

• Enrichment: Receives an event as an input, adds attributes as a result of look up in a data store (database, spreadsheet, file) --- there are various levels of sophistications in enrichment.
• Translation: Receives an event as an input, and translate to an event that is semantically equivalent. Example: XSL/T transformation (when the event is in XML format).
• Aggregation: Receives N events and creates a single event with one or more aggregated attributes, note - this is a statefull mediation, but unlike pattern detection in CEP, the original events are not kept, there is just incremental updates of the state, thus, the state is bounded.
• Split: Recieves a single event and creates M events - either identical clones or distinct - all of them functions of the input event.
• Composition: Aggregation + Split: N input event ---> M output events.

One of the interesting questions are is MEP a subset of CEP ? -- again - this is a matter of implementation - in monolithic stand-alone engines MEP can be done by CEP engine, when CEP engine is part of a middleware, the ESB mediations may be used for this with some twist, and in agent-based EPN, agent-types may be of each of the mediators types.

BTW - I have written before about the CITT meeting in Regensburg - if you want to have more details about the presentations are now on the CITT website. My presentation can be found also on this website, as well as some other interesting stuff.. More -Later.

Why I prefer to use "event processing" with prefix, infix or suffix - a subjective tour of acronyms

Recently there has been more discussions about terms and acronyms, I am not sure that this is so important issue to spend much time on, but before moving to a more interesting points, I would like to provide some personal thoughts about acronyms in this area.


First, as you can see from the Blog's name, I prefer the term "event processing" with any prefix, infix, or suffix. The reason is that I view it as a name of a discipline and not as trend. Disciplines typically consist of two words: signal processing, information retrieval, machine learning, software engineering etc.. although there are exceptions. Three letter acronyms AKA TLA, are typically not names of core disciplines but of other things - protocols, architectures, trends etc..


Historically, when the first "event processing symposium" (which created EPTS) has been established we needed a name - the original founders were - David Luckham, Roy Schulte, Mark Palmer (from Progress Software) and myself. David, of course, thought that CEP is an appropriate name for the discipline, while Mark proposed ESP - "Event Stream Processing" since he did not like the word "complex" (read further about it). Roy and mysrelf proposed to take the part that both agree "event processing". Both David and Mark were not completely happy, but agreed, thus we advanced with the name "event processing symposium" and used "event processing" ever since.



Getting back to history - I have prefered to use the name "active technologies" being a veteran of the active database community, and although the autonomic computing community adopted the "active" term and had conferences named "active middleware services", this name actually did not get into the main stream, David Luckham used the term "complex event processing" in his famous book that used the term. The term "complex event processing" has ambigious meaning - one interpretation is that this is processing of complex events, where complex event is an event that consists of more than one event (analog to complex object), the other interpretation is that this is complex processing of events. I have started to use the term CEP in 2004 to differentiate such functionaity from "event correlation" in system management since there has been some confusion in IBM around this terms. I also made a modest contribution to get the name CEP known by giving a tutorial in ICWS in July 2004, attended by many people, whose common denominator has been tht they have not heard this term before. Anyhow - there are two school of thoughts around CEP


Interpretation one ("the monolithic approach") : CEP = EP, everything is a subset of CEP.
Interpreation two ("the layered approach") : EP is a collection of technologies, whereas CEP is one of them (a link in the chain). Some people takes the first interpretation, saying that "simple" event processing (whether it is simple event or simple processing) is a subset of complex event processing, the rational behind it that if an engine is capable of doing complex things it is surely capable of doing simple things. Interpreation two comes from Roy Schulte (Gartner) who introduced in December 2005 the following slide:

In this slide Roy Schulte talks about four types of processing (later he realized that the BPM one is of another category) - simple event processing (filter and route), mediate event processing (transform and enrich) and complex event processing (statefull pattern detector). This is consistent with a market view since there are products that do only simple event processing (messaging), other products who do mediated event processing (ESB) and CEP as the next layer as a stateful engine. I think that this approach is liked by those who are putting CEP on top of existing middleware, while the first ("monolithic") approach is liked by those who have stand-alone CEP engine. Anyway - the existence of this two approaches, and the fact that people may not understand that the other person is taking the second interpretation is causing a confusion.

Next acronym has been "event stream processing", the term "data stream manager" has been coined in Stanford in a similar meaning, but with SQL API, and continuing with other academic projects, and some descendent products (Coral8 is a descendent of the Stanford project). When Progress Software acquired Apama, Mark Palmer looked for an alternative word for CEP, since he was in the opinion that customers don't like anything labelled "complex", thus, he borowed the term "stream" although Apama's API is not SQL, and has not much to do with the academic stream projects and introduced the ESP term "Event Stream Processing" (which was dropped later). In response, David Luckham published an article to defend the "complex" word, starting with the words: "some people, I'm told, get scared when they hear the word complex, as in complex event processing.... start with the basic question, is life simple ? most people when asked about it will truthfully answer no...." and the rest you can read yourself. It seems that David has won this battle -- all vendors (including the SQL oriented ones) at some point or another have positioned themselves as CEP vendors, which also created some objections - by people who thought that it is important to diffrentiate between ESP and CEP, some saying that ESP is a subset of CEP, and some that these are completely different focus areas - as I have written before, there are many ways to define subsets of EP functionality, and I did not find any evidence that the one defined by this distinction (totally ordered events vs. partially ordered events) is the important one (in many applications we need both types for different purposes).

What other acronyms have flown around ? - well, Forrester at some point made a distinction between CEP and BEM (Business Event Management) that has been defined as - "a process of capturing real-time business events from multiple source and assigning them to the appropriate decision-maker for resolution based on the business context of the events". I have struggled to understand the distinction - maybe the fact that it deals with simple events, however, when they mention context - determining the context may by itself require CEP.

We, in IBM are using the term IEP (Intelligent Event Processing) to denote stochastic and intelligent reasoning beyond the deterministic pattern detection to CEP; this is consistent with the layer approach, the monolithic approach fans, view IEP as part of CEP.

The new term we heard this week from IBM is BEP (Business Event Processing) and this is intended to define event processing applications in which the business user can control the behaior (i.e. define and modify patterns without the help of a programmer), a topic I also discussed in the past.

Last but not least, some people in the academic community don't like the term "processing" which they think is too elementary and talk about "event-based computing" as the name of the discipline.
After this unusally long postings, my bottom lines are :

(1). The upcoming glossary should provide a consistent taxonomy of terms here - there is still much confusion about the names, and the glossary can be a good reference point,

(2). Personally, I still prefer to talk about types of functions and not about boundaries of names, however, I understand the importance of branding.

(3). I still prefer the name "event processing" without prefix, infix or suffix - and thus continue to use this name.

(4). Hopefully, this is the last posting I am writing on the *-E-P; E-*-P; E-P-* topic - I have more interesting topics to deal with.... more - later.

On terminology again - BEP and CEP

There were several postings in the last few days from my esteemed colleagues - David Luckham, Tim Bass and Pual Vincent (in his comment to Tim's posting) all referred to the term "Business Event Processing" (BEP) that has been mentioned by Sandy Carter, VP of SOA and Websphere marketing in IBM. There were several references to this term relative to CEP.

The terminology issue is sometimes confusing, so let me clarify here: as I have discussed the
*-EP phenomenon in the past, so I'll dedicate this posting to clarification.

Event Processing, in general, is bigger in scope from Complex Event Processing; CEP deals with the detection of patterns over the a collection of events ("event cloud"). Event processing starts from stateless filtering and pub/sub, advances to mediated event processing - transformation, enrichment, validation etc, then to CEP, and to IEP (Intelligent Event Processing) that adds stochastic reasoning.

BEP (Business Event Processing) is an event processing applied to business applications. To Paul Vincnet's question - are there "non business" EP - the answer is -- yes, event processing can be used for individual person in smart homes; some people also see IT system management as a "non business" application, and differentiate between business and IT. What is the relations between BEP an CEP ? BEP is thus -- the entire EP applied to business application; CEP is subset of EP. More - later.

On situations

Hello from Zurich, got here in a train from Munich - and this is the famous train station of Zurich (in light), I have arrived in the dark, but has been in Zurich quite a lot of times before, tomorrow I am visiting the IBM Zurich Lab (which is actually located in Rüschlikon (a suburb of Zurich) home to four Noble prizes in physics. The hotel is 5 minutes walk from the train, but I forgot to print a map, and had to search a little. In the first half of the day I have participated in the continuation of the CITT conference, and earned my bread by giving a talk. After the talk I had some discussion with people about the term situation that I mentioned in the talk. I have borrowed this word from "situation calculus" in AI, and it fits well with terms in related domain such as: situation awareness. The discussion has been has been around one of the common misconceptions that situation is an alias to derived event. The answer is - no, these terms are not aliases ! Situation is some phenomenon that requires reaction (or notification). It is in the conceptual ontology of the consumer. Now - let's check the relationships with event:

(1). A raw event that is reported by a producer is a situation in the consumer's terms - in this case, only routing is required ("simple event processing").

(2). A raw event is reported by a producer has 1-1 correspodence to a situation in the consumer's terms, but more information or translation is required - in this case, the situation is created by mediated event processing.

(3). The situation occurs in a deterministic way once an event pattern is identified, and it is a functions of the events participating in the pattern - in this case the situation is created by complex event processing.

Note that the situation is not the derived event - but the derived event in this case is an indication that the situation is detected, note also that not all derived events are situations, derived events that are consumed by other agents, are not situations, only those that are consumed by event consumers.

However, situation detection may not be purely deterministic - and here we are getting to "intelligent event processing". Example: the pattern is only an approximation, example: the pattern is - "an event E1 happens at least 4 times within an hour". Now, if the event E1 happens 3 times and the pattern is an approximation, there is some probability that the situation did occur. This is even before looking at probabilistic events.

More about uncertainty in event processing - later.

On the envelope for CEP

In two recent blogs - Mark Tsimelzon from Coral8 argued for CEP server vs. embedded CEP libraries, while Paul Vincent from TIBCO argued for the need for infrastructure stack outside the

CEP engine.

Since I avoid making product evaluations in this Blog, I'll talk about the principle.

Conceptually we have a model of "event producer" that produces the events, "event consumer" that consumes the event, and the "event processor" which processes the events and stands in the middle. There are two questions about envelopes:

(1). Is "event processor" an embedded capabilities inside applications or an server.

(2). If "event processor" is a server, is it independent server or part of a larger middleware ?

The answer, as anything is not binary (zero or one), there are cases in which there is a need to have event processing as embedded capabilities (e.g. inside pervasive devices), but Mark is right that the big majority of "event processors" are tend towards the server.

The second question is more interesting --- we have today both stand-alone server and servers that are part of a larger middleware, the rational behind being part of a larger middleware, stems from the fact that event processing is not isolated and has various relations with different applications in the enterprise, it is true that the loose-coupling nature of event driven architectures eases the task of separating it from the applications, but still the integration is the most costly part of building event processing applications, and means to ease the integration has already built into application integration middleware, and if the event processing server is a stand-alone one, there is a need to re-invent this integration, as Paul Vincent rightly say: every $ a CEP vendor spends on middleware integration is a $ less on interesting CEP functionality.

Furthermore, there are some event processing infrastructure and functions (pub/sub, routing - for simple event processing, and ESB mediations like - enrichment, transformation etc, that are already there). Thus, it seems that the ROI will be higher if event processors will be implemented on top of a "middleware stack".

An interesting observation is that from the point of view of application integration middlewares - event processing is becoming a key feature, and there are already some predictions that the standard event processing programming model (which is still not there!), will be the basis for application servers of the future, e.g. Gartner's XTP that I have once discussed and should discuss more.

classification of event processing applications - part I

You may need a good eyesight to look at this slide, so let's skip it for now, and return to it later.

Paul Vincent, in his recent blog tells us that he is struggling with classification - and talks about "simple event simple processing" to "complex event complex processing". Well - here are some thoughts that may ease the struggle. You have noticed that Paul Vincent constantly writes about Complex Event Processing and I constantly write about Event Processing - are they synonyms and I am just omitting the C ? the answer is - no!, Complex Event Processing is a subset of Event processing, according to one of the classifications, but there are some other types of Event Processing. Back to the classification issue -- there are different dimensions by which we can classify "event processing applications" :

(1). By processing type - and CEP is part of this classification

(2). By goal - what they intend to achieve

(3). By non-functional requirements

(4). By type of events that are processed.

Maybe more classifications. So in this blog I'll concentrate in (1) - "by processing type".

The slide above is showing a relatively simple application that checks orders against supplies, in the following way:

• Supply may arrive in multiple ways - there is an aggregation function that aggregates all events about arrival of supply.
• When all the supply arrived it is matched against the original order - this is a "pattern matching" - in this case - sequence of two events (order, aggregated supply) with some matching condition
• If does not match - then - there is a "non matching" event reported and then enriched from a database in more details about the order
• The enriched event is then reported to the supplier, and some decision is being taken.
• A decision is taken by "supplier type" - thus there is some filtering and routing based on the type.

The functions that we have here are : aggregation, enrichment, pattern detection and filtering.

They belong to three different families:

• simple event processing: operations on events that do not change the events - just filter and route them.
• mediated event processing: aggregation and enrichment - this is a type of processing that transform the event. In enrichment it is transformed by adding attributes taken from an external store (e.g. database), while in aggregation - multiple events create one event with (in this case - sum of products in the different supplies).
• complex event processing -- detecting pattern - here the pattern is: order occurred and later all supplies arrive - but the quantities don't match. This is very simple pattern, but it is complex event processing, because it deals with complex events.

Some more observations:

• The border between mediated event processing and complex event processing (which both derive new events) is that mediated event processing does not detect a pattern, and does not need to keep the raw events in a state, however, it may be statefull (aggregation is statefull, but not complex event processing - the event that is kept at the end is simple.
• The names may not be very good - since pattern of complex event processing can be quite simple, while filter of simple event processing may be rather complex - however, the "complex" name became pervasive in the industry, and derives the others...

More about the other types of classifications - later