This is a blog describing some thoughts about issues related to event processing and thoughts related to my current role. It is written by Opher Etzion and reflects the author's own opinions
Thursday, January 3, 2008
IBM WebSphere CTO sees CEP as SOA's 'next big thing'
Monday, December 31, 2007
Has BAM failed ?

- The sin - BAM solutions were developed outside the true religion of BI;
- The punishment - BAM failed;
- The redemption - Embedded BI applications have included the concept of BAM -- and lived happily ever after.
In order to analyze this claim - let's first define what "BAM solutions" are. BAM - Business activity monitoring (or management) stands for a class of applications that observe the behavior of business activities (business processes, transactions, applications etc..), in a non politically correct terminology (well, we are lagging behind USA, we have not invented political correctness yet) I can call it "the big brother is watching", a term that is sometimes used is "observation" - the essence is -- look at business activity from the outside, and try to find situations that are of interest - can be major situations like - threats and opportunities, but most of them are micro issues that need to be observed, maybe fixed, maybe just watched closely.
Now for the question of failure or success -- Bill Gassman from Gartner claims in his talks that 65% of the BAM solutions are industry specific, and general purpose software accounts only to 35% of the BAM solutions. Indeed, while the notion of observation is general, what we would like to observe and how is very much domain dependent. Performance management that trace Key Performance Indicators is one flavor that has been generalized, but many of the BAM solutions are not about tracing aggregates or measurements, but about tracing individual situations in the business activities: delivery has failed, to meet the deadline, customer has complained and not handled well, medical treatment has been done outside the protocol for that disease - there are examples from all industries for it. Event Processing is a useful technology in cases where the BAM solution is event-driven, and its input is event. Some of the BAM situations are detected on-line, some of them are detected in retrospect - and I'll talk about retrospective event processing in another posting. General BAM systems are useful for some cases - and are less useful for others.
So - my 2 cents opinion about Colin White's claims:
- The concept of BAM has not failed - there are many success stories, however the area has not reached maturity. General purpose BAM are effective for some of the BAM market, but many BAM solutions are more industry-specific -- event processing is useful both for the general-purpose BAM market and for the industry-specific BAM market.
- And as for BAM on top of BI -- according to Gassman this accounts for 10% of the BAM solutions market, so while it has certainly its own variety of application, the majority of the BAM market is not a BI market.
P.S - the picture above is of a skateboard called BAM -- and I hope it sells well - click for a free advertisement to skateboards
More - Later.
Sunday, December 30, 2007
Event Processing for Business Intelligence
From the various descriptions of Business Intelligence found on the Web, I have chosen the one by Hakki Erbug as a starting point.
As noted in several previous postings, IMHO, event processing is a set of technologies that have multiple usages, and are not really strongly associated with a single type of application. Business Intelligence is similar in the fact that it combines various technologies, but different in the fact that it is focused around area of using data for decision making in various ways. In the event processing case, decision making is one of the possible usages, but not the only. This posting will briefly survey how event processing can enrich business intelligence, and in fact, in the Gartner EPS summit I have seen several BI vendors that are looking at EP as a natural capability to complement their products.
Going on Erburg's illustration clockwise:
1. "Active Data Warehouse" - While traditional warehouses are being updated in batch, the notion of active data warehouse makes the warehouse update itself an event-driven action. The rationale is: when a certain event (raw or derived) occurs, a decision has to be taken, however, the decision relies on a data warehouse, thus, an update of the data warehouse should occur before making this decision. The update can be of the same event that happens or of some collection of data that has still not been updated in the data warehouse and is needed for the decision making. There can be some time constraints associated with the decision (and in turn with the warehouse updates). The time constraints are not necessarily micro-seconds, the constraints can be minutes or hours, but they are typically well-defined.
2. ETL and mediated event processing: ETL has some functional similarity with mediated event processing, it is also about transformation. We see mediated event processing and ETL getting closer to one another, where difference may be in quality of service. In the future there may be a case that ETL will become a specific case of mediated event processing (of course ETL folks may say the same from the opposite direction).
3. Real-time analytics: While analytics (simulation, optimization, mining...) has been used for a while in the decision making part of BI, in the event-driven world, the reaction to an event in some cases is temporally-bound, which means that there is a real-time constraint, or upper limit on the requested reaction time. This provides new way of thinking about analytics - while without time constraints an optimization should strive to get the "best result" (or if heuristics satisfy some approximation condition), in real-time analytics the optimization strives to get "the best result that can be obtained in T time-units as specified (e.g. 18 seconds)". How does event processing play in real-time analytics? it may play in a simulation mode - scenarios are created and simulated events are emitted - they in turn may create simulated derived events which determine the situations of this simulation. This is, of course, in addition to the fact that in an event-driven universe, the entire BI cycle is event-driven and relates to the event and its context.
To conclude -- event processing is a natural step in the BI capabilities, and thus I expect BI suites to support the event-driven flavor... This - again, does not say that BI is the ONLY use of event processing. I still need also to refer to the issue - "has BAM failed because it was not based on BI techniques" as claimed in the article that triggered my discussion on the BI topic - stay tuned.
Thursday, December 27, 2007
On Business Intelligence and Event Processing
Anyway, coming back, I saw that the blog world of event processing is also alive these days, but this time I would like to give a few comments to "operational analytics: yesterday today and tomorrow" by Colin White
In this article Mr. White has some interesting assertions (at least, my own interpretation of these assertions):
(1). CEP is a buzzword which stands for a kind of operational analytics that should be used in extreme cases.
(2). BAM has failed since it has been independent of BI
(3). Hint at the fate of CEP (or ESP, it is not clear from his article what is the difference) will be the same if not be part of BI.
While I am sure that Mr. White is a big expert on BI, it seems that he also falls into the "hammer and nail" trap that has been discussed by myself and several other bloggers in this area. So here is some preliminary responses to his assertions:
(1). CEP is a technology, it has roots in multiple disciplines, and some of it has roots in BI, but there is a distance between this and the assertion that it is part of BI. CEP has different uses that may not be even connected to BI (e.g. network management diagnostics or situational awareness), here again we get back to the issue of motivation for using CEP - the consistent view of database people is that the only reason to use CEP is extreme latency/throughput otherwise one can use the right religion of SQL databases, I think that this issue has been discussed also in the EP blogs that there are multiple reasons for using CEP and the high throughput / low latency is one, but not necessarily event the dominant one.
(2). As far as the "BAM has failed" -- is that a fact or a wishful thinking ? in the Gartner BAM and EP summit we have heard some success stories of BAM, and saw some prospering BAM products. While there are synergies and relationships between BAM and BI - I wonder what are the success / failure criteria used to derive this assertion ?
(3). I have already stated my opinion about the question - whether event processing is a footnote to databases
While I spent many years as part of the database community, I am in the opinion that event processing is a discipline by its own right, with some interaction and intersection with the database area, as well as other areas (business rules, distributed computing, middleware, software agents).
This is a preliminary reaction - mostly giving some comments to the article, which does not free me from writing a "positive" article about - the relationships between event processing and business intelligence, and I'll do so in one of the next postings - more later.
Wednesday, December 26, 2007
EPTS plans for 2008

This picture has been taken in the EPTS meeting in Orlando in September 2007, the meeting itself has been described by Paul Vincent in his blog : first day, second and third day.
Saturday, December 22, 2007
On the envelope for CEP
Thursday, December 20, 2007
On - "one size fits all" and Event Processing
- I don't believe that ESP vs. CEP is the right type of partition in this area;
- There may be a need to have various implementation under one roof (the heterogeneous framework approach),
For the first point -- what is the right type of partition ? this is a multi-dimensional questions and we still have to learn more to know the most useful combinations.
One of the important dimensions is the "reason for use" dimension, and here in an internal IBM study we got to five different reasons to use, I'll write about it in one of the next postings.
EPTS has recently launched a workgroup that tries to identify these classifications by doing a comprehansive survey of use cases that will be compared using the same template. A team that consists of Tao Lin (SAP), Dieter Gawlick (Oracle) and Pedro Bizzaro (University of Coimbra, Portugal) is working on this template, and a larger team will handle this survey and analysis -- the end result - a collaborative white paper about the state of the practice in event processing is expected somewhere in the second quarter of 2008. Stay tuned.
More - Later.