Wednesday, January 30, 2008

On Mediated Event Processing



I have mentioned the term "mediated event processing" in the past, but looking at previous postings - never explained exactly what it is.

So back to the educational mission of this Blog: mediations are exactly what they mean in messaging middleware or ESBs - transforming events. There are several types of mediators.

  • Enrichment: Receives an event as an input, adds attributes as a result of look up in a data store (database, spreadsheet, file) --- there are various levels of sophistications in enrichment.
  • Translation: Receives an event as an input, and translate to an event that is semantically equivalent. Example: XSL/T transformation (when the event is in XML format).
  • Aggregation: Receives N events and creates a single event with one or more aggregated attributes, note - this is a statefull mediation, but unlike pattern detection in CEP, the original events are not kept, there is just incremental updates of the state, thus, the state is bounded.
  • Split: Recieves a single event and creates M events - either identical clones or distinct - all of them functions of the input event.
  • Composition: Aggregation + Split: N input event ---> M output events.

One of the interesting questions are is MEP a subset of CEP ? -- again - this is a matter of implementation - in monolithic stand-alone engines MEP can be done by CEP engine, when CEP engine is part of a middleware, the ESB mediations may be used for this with some twist, and in agent-based EPN, agent-types may be of each of the mediators types.

BTW - I have written before about the CITT meeting in Regensburg - if you want to have more details about the presentations are now on the CITT website. My presentation can be found also on this website, as well as some other interesting stuff.. More -Later.


Monday, January 28, 2008

Why I prefer to use "event processing" with prefix, infix or suffix - a subjective tour of acronyms




Recently there has been more discussions about terms and acronyms, I am not sure that this is so important issue to spend much time on, but before moving to a more interesting points, I would like to provide some personal thoughts about acronyms in this area.


First, as you can see from the Blog's name, I prefer the term "event processing" with any prefix, infix, or suffix. The reason is that I view it as a name of a discipline and not as trend. Disciplines typically consist of two words: signal processing, information retrieval, machine learning, software engineering etc.. although there are exceptions. Three letter acronyms AKA TLA, are typically not names of core disciplines but of other things - protocols, architectures, trends etc..


Historically, when the first "event processing symposium" (which created EPTS) has been established we needed a name - the original founders were - David Luckham, Roy Schulte, Mark Palmer (from Progress Software) and myself. David, of course, thought that CEP is an appropriate name for the discipline, while Mark proposed ESP - "Event Stream Processing" since he did not like the word "complex" (read further about it). Roy and mysrelf proposed to take the part that both agree "event processing". Both David and Mark were not completely happy, but agreed, thus we advanced with the name "event processing symposium" and used "event processing" ever since.



Getting back to history - I have prefered to use the name "active technologies" being a veteran of the active database community, and although the autonomic computing community adopted the "active" term and had conferences named "active middleware services", this name actually did not get into the main stream, David Luckham used the term "complex event processing" in his famous book that used the term. The term "complex event processing" has ambigious meaning - one interpretation is that this is processing of complex events, where complex event is an event that consists of more than one event (analog to complex object), the other interpretation is that this is complex processing of events. I have started to use the term CEP in 2004 to differentiate such functionaity from "event correlation" in system management since there has been some confusion in IBM around this terms. I also made a modest contribution to get the name CEP known by giving a tutorial in ICWS in July 2004, attended by many people, whose common denominator has been tht they have not heard this term before. Anyhow - there are two school of thoughts around CEP


Interpretation one ("the monolithic approach") : CEP = EP, everything is a subset of CEP.
Interpreation two ("the layered approach") : EP is a collection of technologies, whereas CEP is one of them (a link in the chain). Some people takes the first interpretation, saying that "simple" event processing (whether it is simple event or simple processing) is a subset of complex event processing, the rational behind it that if an engine is capable of doing complex things it is surely capable of doing simple things. Interpreation two comes from Roy Schulte (Gartner) who introduced in December 2005 the following slide:









In this slide Roy Schulte talks about four types of processing (later he realized that the BPM one is of another category) - simple event processing (filter and route), mediate event processing (transform and enrich) and complex event processing (statefull pattern detector). This is consistent with a market view since there are products that do only simple event processing (messaging), other products who do mediated event processing (ESB) and CEP as the next layer as a stateful engine. I think that this approach is liked by those who are putting CEP on top of existing middleware, while the first ("monolithic") approach is liked by those who have stand-alone CEP engine. Anyway - the existence of this two approaches, and the fact that people may not understand that the other person is taking the second interpretation is causing a confusion.

Next acronym has been "event stream processing", the term "data stream manager" has been coined in Stanford in a similar meaning, but with SQL API, and continuing with other academic projects, and some descendent products (Coral8 is a descendent of the Stanford project). When Progress Software acquired Apama, Mark Palmer looked for an alternative word for CEP, since he was in the opinion that customers don't like anything labelled "complex", thus, he borowed the term "stream" although Apama's API is not SQL, and has not much to do with the academic stream projects and introduced the ESP term "Event Stream Processing" (which was dropped later). In response, David Luckham published an article to defend the "complex" word, starting with the words: "some people, I'm told, get scared when they hear the word complex, as in complex event processing.... start with the basic question, is life simple ? most people when asked about it will truthfully answer no...." and the rest you can read yourself. It seems that David has won this battle -- all vendors (including the SQL oriented ones) at some point or another have positioned themselves as CEP vendors, which also created some objections - by people who thought that it is important to diffrentiate between ESP and CEP, some saying that ESP is a subset of CEP, and some that these are completely different focus areas - as I have written before, there are many ways to define subsets of EP functionality, and I did not find any evidence that the one defined by this distinction (totally ordered events vs. partially ordered events) is the important one (in many applications we need both types for different purposes).

What other acronyms have flown around ? - well, Forrester at some point made a distinction between CEP and BEM (Business Event Management) that has been defined as - "a process of capturing real-time business events from multiple source and assigning them to the appropriate decision-maker for resolution based on the business context of the events". I have struggled to understand the distinction - maybe the fact that it deals with simple events, however, when they mention context - determining the context may by itself require CEP.

We, in IBM are using the term IEP (Intelligent Event Processing) to denote stochastic and intelligent reasoning beyond the deterministic pattern detection to CEP; this is consistent with the layer approach, the monolithic approach fans, view IEP as part of CEP.


The new term we heard this week from IBM is BEP (Business Event Processing) and this is intended to define event processing applications in which the business user can control the behaior (i.e. define and modify patterns without the help of a programmer), a topic I also discussed in the past.

Last but not least, some people in the academic community don't like the term "processing" which they think is too elementary and talk about "event-based computing" as the name of the discipline.
After this unusally long postings, my bottom lines are :


(1). The upcoming glossary should provide a consistent taxonomy of terms here - there is still much confusion about the names, and the glossary can be a good reference point,

(2). Personally, I still prefer to talk about types of functions and not about boundaries of names, however, I understand the importance of branding.

(3). I still prefer the name "event processing" without prefix, infix or suffix - and thus continue to use this name.

(4). Hopefully, this is the last posting I am writing on the *-E-P; E-*-P; E-P-* topic - I have more interesting topics to deal with.... more - later.

Friday, January 25, 2008

On terminology again - BEP and CEP


There were several postings in the last few days from my esteemed colleagues - David Luckham, Tim Bass and Pual Vincent (in his comment to Tim's posting) all referred to the term "Business Event Processing" (BEP) that has been mentioned by Sandy Carter, VP of SOA and Websphere marketing in IBM. There were several references to this term relative to CEP.

The terminology issue is sometimes confusing, so let me clarify here: as I have discussed the
*-EP phenomenon in the past, so I'll dedicate this posting to clarification.

Event Processing, in general, is bigger in scope from Complex Event Processing; CEP deals with the detection of patterns over the a collection of events ("event cloud"). Event processing starts from stateless filtering and pub/sub, advances to mediated event processing - transformation, enrichment, validation etc, then to CEP, and to IEP (Intelligent Event Processing) that adds stochastic reasoning.

BEP (Business Event Processing) is an event processing applied to business applications. To Paul Vincnet's question - are there "non business" EP - the answer is -- yes, event processing can be used for individual person in smart homes; some people also see IT system management as a "non business" application, and differentiate between business and IT. What is the relations between BEP an CEP ? BEP is thus -- the entire EP applied to business application; CEP is subset of EP. More - later.

Wednesday, January 23, 2008

Welcome to the AptSoft team to IBM


Burlington, Mass. - I have never been there - but I have learned from its official website that they hold town meetings which is nice. Besides town meetings, it is also the home of the AptSoft company. Today it was announced that AptSoft has gained a new home and becomes part of the IBM Websphere organization. Getting from a start-up to a big company is a big step (I have been in both types of organizations and know the difference...), and sometimes a caltural shock... it is also a big step for those who believed that IBM should be a player in the CEP space,
making the first announcement about complex event processing product with this acquisition.

While I still need more education about the AptSoft product, it seems on first impression (what I have seen in the Gartner EPS conference in their booth) that they have a thinking competible with what I posted about the fact that CEP patterns should be defined, in many cases, by a business user and not by a developer, and their thinking is geared towards this direction -- this is a good value that they bring to the table.
Meanwhile -- to Steve Lyons, David Martin and the rest of the AptSoft team - warm welcome to IBM, and happy blue-washing... see you around in the corridors of the blue giant.

Tuesday, January 22, 2008

On The WHEN question - in event processing

In one of the previous postings I have talked about various notions of real-time, from the example of the human event processing that I have given yesterday, it seems that there is importance in some cases to react in "hard real-time", e.g. if we'll assume that instead of human drivers the cars will be driven by a computerized system, then this system will have a lot of hard real-time constraints, especially when related to safety, based on events that received from sensors about the environment. An inheritance from active databases is the distinction between "immidate action" and "deferred action", however these are inaccurate terms-

Immediate typically means:

  • As soon as possible based on best effort,

However sometimes it means :

  • Hard or soft real time constraints on the latency.

Deferred may mean:

  • As soon as possible after the end of some context (e.g. at the end of a time window)

But it may also means:

  • Exactly in 5:00PM, Anytime between 7:00 - 9:00 PM etc..

Indeed - not all events are processed immediately, the best example is an absent pattern based on time-out, in which the fact that the event did not happen is processed at the end of the time-out .

In some cases we also want to apply event patterns to historical event - which is known as "retrospective processing"; this will be discussed in another opportunity.

Monday, January 21, 2008

Unplanned events - again


This (more or less) how my MPV car looked like at the beginning of this day, unfortuantely, it does not look like this now -- while driving in a highway, there was some traffic congestion, and the traffic slowed down, however, the driver behind me has not detected this event and proceeded to drive full speed - as a result he crashed his small car into my car, totally destroyed his car, but made also substantial damage to the back of my car. So - I had an absent event, did not make the meeting I was driving to, had to wait for police, and then almost two hours for a replacement car from the leasing company -- so wasted much of the day. Luckily for me, I have detected that with the momentum of the crash, I am approaching very quickly a track before me -- and succeeded to stop the car before getting into the track -- otherwise, I may not be sitting at home and typing now -- so people also need to process events in real-time and not in batch...

Sunday, January 20, 2008

On ECA and E*C*A*


Back on the ridge - Haifa is a collection of ridges on the Carmel mountain, my residence neigborhood Ramat Begin is seen here from bird eye's view. Anyway - still have some backlog from discussions last week in the CITT conference. One notion is about ECA (Event-Condition-Action) and its role in Event Processing. A few months ago in the VLDB conference I've met
Umesh Dayal, HP Fellow and a person with inspiring work, and we had a short discussion whether the area of "active databases" we have both involved in the past has failed, Umesh was in the opinion that while it has not become mainstrem in database products, as we hoped, but the notion of ECA (that has been coined by Umesh and his colleagues in the HiPAC project) has survived and had a substantial impact in various technologies (not necessarily in the database area). ECA stands for Event - Condition - Action. ECA still has role in event processing, however, this role is more similar to the role of rules in Event Processing - and mainly used in the edge of the EPN network, where the event is being consumed, ECA rules can control the action that is being trigerred at the consumer side. Interestingly enough we can charachterize the "event processing network" itself also using the same three letters - with stars - E*C*A* , where the interpretation is:
  • E* - zero or more events that a single agent see (AKA "event cloud")
  • C* - zero or more contexts that the agent is associated with
  • A* - zero or more agents are activated.

The EPN routes events (both input event from a producer, and derived events from the EPN agents) according to context to activate agents.