On bitter pills, hype-cycles, and event processing

My friend, Tim Bass, has decided to feed us with bitter pills about the state of the CEP market today - should we swallow ? on the other hand - The TIBCO CEP philosopher Paul Vincent is telling us that CEP is not a hype, and can eliminate the "fall" side of the hype cycle should we believe him?

To give you full disclosure - I don't take pills - sweet, bitter or otherwise, and let my immune system protecting me, without help -- maybe with age I'll have to desert this policy, but not out of free will. I also think that while CEP is real, there is some hype phenomenon around it - and the hype cycle is a very smart idea - representing the laws of nature.

So - a few subjective assertions about the state of the event processing market:

1. Event Processing applications are real, there are cases in which they provide a significant value (and ROI) for customers, for various reasons (still owe you postings about the different ROI I found).
2. Even the non-visionary spreadsheet-oriented vendors identify revenues (or losses if they'll not jump into the CEP wagon), thus, it seems that the size of the market starting to hit a critical size. The list of "reference customers" is some indication, but does not provide a good one about the state of the market - some small companies go to the press with any signing on customer agreement to do pilot, while in bigger companies only mega-deals are reported this way; besides there are customers who are not ready to get public exposing even in titles what they are doing.
3. We are still in the first generation of technologies - and first generations are typically the prelude, the thinking about an area is changing, and requirements are identified much faster than the ability of vendors to cope with them (so they start with hacking around them). To provide an analogy - we are now in a similar situation to the database technologies in the early 1960-ies, before the relational database has been invented, even before DBTG proposal. There were some products - but their utilization was limited.
4. While there is a real value to customers, there are also some of the hype phenomenon around it, which leads to the "hype cycle" as a predictor --- we are still in the climbing side of the hype curve - but at some point we'll see some more understanding around what EP applications may do -- and those vendors who will not be able to cope with the relative rapid changes that will be required -- will stay out of the picture, which is also a normal things. We have today more CEP vendors than DB vendors, so we are not yet in steady state.
5. Tim Bass complains about looking at the history -- saying we should focus on applications and history has no importance -- Contrary to that, and looking at other disciplines, there need to be a standard basis of the discipline, otherwise it become a collection of ad-hoc efforts that don't reuse the type of thinking. Standard thinking comes from unified theory - and we have not found yet the equivalent of the "relational model" in databases. In order to do it we need to go back to the basics in various disciplines, and see what can be pulled together - thus the work like David Luckham's survey about the history and the contributing disciplines is very important to get people from these disciplines to work together - like the Dagstuhl seminar or the upcoming DEBS conference

So what is the bottom line of all these assertions ---

first - I am optimistic, I view EP as a disrupting technology for the computing industry, and it has an important role in architectures, applications and conceptions of enterprise computing.

second -- the first generation is just the start, a lot of work to be done to get to the "hype cycle plateau" and beyond...

Since one of my declared hats is a catalyst - I prefer to use a constructive tone -- saying -- here is what should be done in order to get there, and also put a modest contribution to make it happen -- is more constructive from swallowing pills. In one of my next postings - I'll continue this postings by providing views on "what are the most important things to be done".
more - later.

On Event Processing Web Sites

In the last few days there is a debate in the community around the launch of Marco's wiki entitled "event processing wiki" - one opinion said - we already have a portal - David Luckham's site that contains a discussion forum and there will be more visibility to postings on a more known site. The other opinion is "let a thousands flowers bloom" an indication for the vitality of a community is in multiple sites, wikis, blogs etc.

You may ask yourself - what is the Rabbi's picture doing here? do I think that this is a religious question, or have I become religious in my old age. The question is - none of the above - this discussion reminded me an old Jewish story (nothing to do with Rabbi Malkior, an ex-minister and member of the Israeli parlaiment, whose picture you can see) :

An husband and wife came in front of the Rabbi so that he can judge in a dispute among them. The Rabbi listens to the wife and says - you are right, then the husband told the Rabbi his claimes and the Rabbi said - you are right too. After they have gone away - the Rabbi's assistant asks him - how is it possible that they both have been right, they told you contradicting things - and the Rabbi answered patiently - you are also right, of course.

This, more or less, reflects my opinion on the discussion. I totally agree that we, as a community, should not have a central control, and communities are distributed by nature and not centralized. I also agree that posting on a known an popular portal has more visibility then on not-yet-established one, so I'll leave it to the "market forces".

Talking about portals - there is another interesting EP portal now, dedicated mostly to scientific papers in this discipline, and include references to many articles: event based.org

Bottom line: At least cross-reference...

On Killer Applications

My friend Tim Bass, the popular blogger, is the person who implored me into writing my own Blog again and again, until I decided to give it a try. Tim, whom I always enjoy to hear talking, even when we agree to disagree, has notified us in his Blog that he is going to participate in a Webinar about "BAM as a killer application for CEP" . While the Webinar has not yet occurred, and I don't have a clue what they are going to say - I would like to dedicate this posting to killer applications and EP, since this is not the first time that I've heard assertions about killer applications. To start the discussion, let's look at three assertions:

1. BI is a killer application of databases
2. Electronic commerce is a killer application of the World Wide Web
3. BAM is a killer application of CEP

Before looking at assertion number 3, let's look at the previous two assertions:

• Databases are used for many purposed, certainly BI uses databases, which implement data warehouses, but most of the uses in databases today are on operational systems, master data management etc - and BI is just one application. The opposite holds, one cannot do BI, without storing historical data, thus databases is a killer technology for BI.
• Electronic commerce is certainly a growing area, and maybe there are people whose main usage of the web is electronic selling or buying, but today I have entered the Internet several times, none of them has been in order to buy. Actually, my own buying from the internet (I am buying books and songs - as I listen to music in the background when I am not in a meeting, so I have a collection of around 1500 songs now - and growing - only legal downloads!) issue a very small part of my use of the Internet, this is true for most persons, however, the converse holds - it is difficult to hold electronic commerce without the Web, so the Web is a killer technology for EC.

Now, back to the assertion that we inspect - "CEP is a killer application of BAM" - this would have been true if most of the CEP usages were BAM applications, Wikipedia description of BAM states: The goals of Business Activity Monitoring are to provide real time information about the status and results of various operations, processes, and transactions

Many of the BAM products concentrate around displaying Key Performance Indicators on dashboards, but even if we extend the notion of BAM, it is still observation on operations according to predefined measurements (typically aggregative ones).

Now the question - whether most CEP users are doing it through BAM ? according to my observation on the CEP market, the answer is - NO. BAM is an important application of CEP, but there are others - the early adopter application - algorithmic trading - is not really BAM - it is more RTE type - it makes decisions and not presents observations, system and network management applications are also mostly not BAM - they are diagnosis, attempting to find root cause for problems and not display measurements, and the same is true for information dissemination systems that don't monitor anything, and predictive systems that don't have measurements. Thinking about a sample of CEP applications I have looked at recently, there is certainly some that are of BAM type, but it is not the majority. CEP is being used for different purposes, and has different ROI to different people, I'll write more on the different ROI's in one of the next postings, thus, like databases, it does not have a single killer application.

The interesting question is if CEP is a killer technology for BAM or any other application type ? but - I have written enough for today. BTW - speaking about BAM, I have noted the posting of James Taylor - "why are enterprise application so dumb?" - doubting the benefits of presenting observation to humans, instead of taking automated decisions - food for thought. More Later

On Event Processing and Now.

After some more technical postings, back to macro-level issue. David Luckham has written recently in his website about the history of CEP (I'll refer to the content of this article in another posting). All indications are the event processing is not a new thing, however, as some people indicate, there are a lot of interest, events, maybe hype around it NOW - the question is what is new ?



The first observation is that unlike the past, today there are commercial "on the shelf" products whose main purpose is to provide event processing capabilities, while in the past there were event processing capabilities in other types of products (simulation, databases, network and system management, middleware, real-time systems etc..), there is also a start of event processing as a discipline. What are the reasons for the interest now - some of what happened in the last few years that supported the shift from hard-coded event processing functionality to COTS are (based on discussions with people in multiple industries):

• Some contemporary applications which are event-driven by nature - such as: compliance with regulations, the need to detect frauds as two examples - have become pervasive in multiple industries
• The increasing complexity of inter-process integration that is simplified with event-driven interaction
• The need for flexibility and agility to gain market advantage in different areas - thus, move away from hard-coded solutions.
• The substantial growth in the number of available events - e.g. since RFID technology became pervasive.
• Some industry trends like - BAM, RTE, "on demand" - that are also based on responding to events.
• The drive to save expenses in back offices by automating exception handling - trends like STP.

I am sure that there are more of these.

Thus, while event processing functionality is not new - "event processing" as a first class citizen in the computing world - with its own dedicated products, community and emerging discipline is new. More - Later

On Immutability of events

Israel is the land of milk and honey, but not the land of snow - snow is quite rare in Israel. This week the rare event of snow occurred in high mountains across the country - in the picture snow near one of the gates of the old city of Jerusalem. In the Carmel mountain ridge, where I live there was a little bit of snow, higher in the ridge, and some people went there and brought to the office bags full of snow... As I lived several years in the Philadelphia area in USA, I am personally less excited from snow - but it is a notable event here.

Today's topic relates to a question that Marco, fellow EP blogger, and the person behind rulecore, has asked about the previous posting, his question related to enrichment, but I'll extend it to the rest of the transformations : "event is something that is happened, and as such it is immutable, cannot be changed; Do transformations and enrichments break this model?".

The answer for this question is - no. Transformations do not break the model, and events are still immutable. According to the pure model - events cannot be altered or deleted, and when represented in an event store, it has to be an "append only" type of store.

Enrichment and transformations are, in fact, creation of new derived events, as a function of raw events. Thus, according to the pure model, transformed or enriched event is not the same event:

• It has a different type of event - with different structure.
• It has a different event-id.

An Enrichment example maybe - the event is: order, and it has an attribute that refers to customer. The enrichment function looks at the customer in some database, and fetches the values of : customer type (platinum, gold, silver, nobody) and customer_credit_limit.

The fact that there is a different event-id for the raw event and transformed event allows also the traceability to trace the transformation (maybe the problem is in wrong identification of the customer?).

There are some considerations that may, in practice, push towards not obeying to the pure model, and I'll talk about them some other time.

On Mediated Event Processing

I have mentioned the term "mediated event processing" in the past, but looking at previous postings - never explained exactly what it is.

So back to the educational mission of this Blog: mediations are exactly what they mean in messaging middleware or ESBs - transforming events. There are several types of mediators.

• Enrichment: Receives an event as an input, adds attributes as a result of look up in a data store (database, spreadsheet, file) --- there are various levels of sophistications in enrichment.
• Translation: Receives an event as an input, and translate to an event that is semantically equivalent. Example: XSL/T transformation (when the event is in XML format).
• Aggregation: Receives N events and creates a single event with one or more aggregated attributes, note - this is a statefull mediation, but unlike pattern detection in CEP, the original events are not kept, there is just incremental updates of the state, thus, the state is bounded.
• Split: Recieves a single event and creates M events - either identical clones or distinct - all of them functions of the input event.
• Composition: Aggregation + Split: N input event ---> M output events.

One of the interesting questions are is MEP a subset of CEP ? -- again - this is a matter of implementation - in monolithic stand-alone engines MEP can be done by CEP engine, when CEP engine is part of a middleware, the ESB mediations may be used for this with some twist, and in agent-based EPN, agent-types may be of each of the mediators types.

BTW - I have written before about the CITT meeting in Regensburg - if you want to have more details about the presentations are now on the CITT website. My presentation can be found also on this website, as well as some other interesting stuff.. More -Later.

Why I prefer to use "event processing" with prefix, infix or suffix - a subjective tour of acronyms

Recently there has been more discussions about terms and acronyms, I am not sure that this is so important issue to spend much time on, but before moving to a more interesting points, I would like to provide some personal thoughts about acronyms in this area.


First, as you can see from the Blog's name, I prefer the term "event processing" with any prefix, infix, or suffix. The reason is that I view it as a name of a discipline and not as trend. Disciplines typically consist of two words: signal processing, information retrieval, machine learning, software engineering etc.. although there are exceptions. Three letter acronyms AKA TLA, are typically not names of core disciplines but of other things - protocols, architectures, trends etc..


Historically, when the first "event processing symposium" (which created EPTS) has been established we needed a name - the original founders were - David Luckham, Roy Schulte, Mark Palmer (from Progress Software) and myself. David, of course, thought that CEP is an appropriate name for the discipline, while Mark proposed ESP - "Event Stream Processing" since he did not like the word "complex" (read further about it). Roy and mysrelf proposed to take the part that both agree "event processing". Both David and Mark were not completely happy, but agreed, thus we advanced with the name "event processing symposium" and used "event processing" ever since.



Getting back to history - I have prefered to use the name "active technologies" being a veteran of the active database community, and although the autonomic computing community adopted the "active" term and had conferences named "active middleware services", this name actually did not get into the main stream, David Luckham used the term "complex event processing" in his famous book that used the term. The term "complex event processing" has ambigious meaning - one interpretation is that this is processing of complex events, where complex event is an event that consists of more than one event (analog to complex object), the other interpretation is that this is complex processing of events. I have started to use the term CEP in 2004 to differentiate such functionaity from "event correlation" in system management since there has been some confusion in IBM around this terms. I also made a modest contribution to get the name CEP known by giving a tutorial in ICWS in July 2004, attended by many people, whose common denominator has been tht they have not heard this term before. Anyhow - there are two school of thoughts around CEP


Interpretation one ("the monolithic approach") : CEP = EP, everything is a subset of CEP.
Interpreation two ("the layered approach") : EP is a collection of technologies, whereas CEP is one of them (a link in the chain). Some people takes the first interpretation, saying that "simple" event processing (whether it is simple event or simple processing) is a subset of complex event processing, the rational behind it that if an engine is capable of doing complex things it is surely capable of doing simple things. Interpreation two comes from Roy Schulte (Gartner) who introduced in December 2005 the following slide:

In this slide Roy Schulte talks about four types of processing (later he realized that the BPM one is of another category) - simple event processing (filter and route), mediate event processing (transform and enrich) and complex event processing (statefull pattern detector). This is consistent with a market view since there are products that do only simple event processing (messaging), other products who do mediated event processing (ESB) and CEP as the next layer as a stateful engine. I think that this approach is liked by those who are putting CEP on top of existing middleware, while the first ("monolithic") approach is liked by those who have stand-alone CEP engine. Anyway - the existence of this two approaches, and the fact that people may not understand that the other person is taking the second interpretation is causing a confusion.

Next acronym has been "event stream processing", the term "data stream manager" has been coined in Stanford in a similar meaning, but with SQL API, and continuing with other academic projects, and some descendent products (Coral8 is a descendent of the Stanford project). When Progress Software acquired Apama, Mark Palmer looked for an alternative word for CEP, since he was in the opinion that customers don't like anything labelled "complex", thus, he borowed the term "stream" although Apama's API is not SQL, and has not much to do with the academic stream projects and introduced the ESP term "Event Stream Processing" (which was dropped later). In response, David Luckham published an article to defend the "complex" word, starting with the words: "some people, I'm told, get scared when they hear the word complex, as in complex event processing.... start with the basic question, is life simple ? most people when asked about it will truthfully answer no...." and the rest you can read yourself. It seems that David has won this battle -- all vendors (including the SQL oriented ones) at some point or another have positioned themselves as CEP vendors, which also created some objections - by people who thought that it is important to diffrentiate between ESP and CEP, some saying that ESP is a subset of CEP, and some that these are completely different focus areas - as I have written before, there are many ways to define subsets of EP functionality, and I did not find any evidence that the one defined by this distinction (totally ordered events vs. partially ordered events) is the important one (in many applications we need both types for different purposes).

What other acronyms have flown around ? - well, Forrester at some point made a distinction between CEP and BEM (Business Event Management) that has been defined as - "a process of capturing real-time business events from multiple source and assigning them to the appropriate decision-maker for resolution based on the business context of the events". I have struggled to understand the distinction - maybe the fact that it deals with simple events, however, when they mention context - determining the context may by itself require CEP.

We, in IBM are using the term IEP (Intelligent Event Processing) to denote stochastic and intelligent reasoning beyond the deterministic pattern detection to CEP; this is consistent with the layer approach, the monolithic approach fans, view IEP as part of CEP.

The new term we heard this week from IBM is BEP (Business Event Processing) and this is intended to define event processing applications in which the business user can control the behaior (i.e. define and modify patterns without the help of a programmer), a topic I also discussed in the past.

Last but not least, some people in the academic community don't like the term "processing" which they think is too elementary and talk about "event-based computing" as the name of the discipline.
After this unusally long postings, my bottom lines are :

(1). The upcoming glossary should provide a consistent taxonomy of terms here - there is still much confusion about the names, and the glossary can be a good reference point,

(2). Personally, I still prefer to talk about types of functions and not about boundaries of names, however, I understand the importance of branding.

(3). I still prefer the name "event processing" without prefix, infix or suffix - and thus continue to use this name.

(4). Hopefully, this is the last posting I am writing on the *-E-P; E-*-P; E-P-* topic - I have more interesting topics to deal with.... more - later.