The one week school break for children ends tomorrow, and today I went with my two younger daughters to see Avatar, a very ambitious movie with a lot of advanced 3D graphics. The downside is that the film spans over 3 hours without a break -- could cut 1 hour easily, but nevertheless the graphics is very impressive, the plot is kind of paraphrase on other movies.
This is a blog describing some thoughts about issues related to event processing and thoughts related to my current role. It is written by Opher Etzion and reflects the author's own opinions
Saturday, December 19, 2009
More on the ecosystem of event processing systems
The one week school break for children ends tomorrow, and today I went with my two younger daughters to see Avatar, a very ambitious movie with a lot of advanced 3D graphics. The downside is that the film spans over 3 hours without a break -- could cut 1 hour easily, but nevertheless the graphics is very impressive, the plot is kind of paraphrase on other movies.
Tuesday, February 5, 2008
On Killer Applications
- BI is a killer application of databases
- Electronic commerce is a killer application of the World Wide Web
- BAM is a killer application of CEP
Before looking at assertion number 3, let's look at the previous two assertions:
- Databases are used for many purposed, certainly BI uses databases, which implement data warehouses, but most of the uses in databases today are on operational systems, master data management etc - and BI is just one application. The opposite holds, one cannot do BI, without storing historical data, thus databases is a killer technology for BI.
- Electronic commerce is certainly a growing area, and maybe there are people whose main usage of the web is electronic selling or buying, but today I have entered the Internet several times, none of them has been in order to buy. Actually, my own buying from the internet (I am buying books and songs - as I listen to music in the background when I am not in a meeting, so I have a collection of around 1500 songs now - and growing - only legal downloads!) issue a very small part of my use of the Internet, this is true for most persons, however, the converse holds - it is difficult to hold electronic commerce without the Web, so the Web is a killer technology for EC.
Now, back to the assertion that we inspect - "CEP is a killer application of BAM" - this would have been true if most of the CEP usages were BAM applications, Wikipedia description of BAM states: The goals of Business Activity Monitoring are to provide real time information about the status and results of various operations, processes, and transactions
Many of the BAM products concentrate around displaying Key Performance Indicators on dashboards, but even if we extend the notion of BAM, it is still observation on operations according to predefined measurements (typically aggregative ones).
Now the question - whether most CEP users are doing it through BAM ? according to my observation on the CEP market, the answer is - NO. BAM is an important application of CEP, but there are others - the early adopter application - algorithmic trading - is not really BAM - it is more RTE type - it makes decisions and not presents observations, system and network management applications are also mostly not BAM - they are diagnosis, attempting to find root cause for problems and not display measurements, and the same is true for information dissemination systems that don't monitor anything, and predictive systems that don't have measurements. Thinking about a sample of CEP applications I have looked at recently, there is certainly some that are of BAM type, but it is not the majority. CEP is being used for different purposes, and has different ROI to different people, I'll write more on the different ROI's in one of the next postings, thus, like databases, it does not have a single killer application.
The interesting question is if CEP is a killer technology for BAM or any other application type ? but - I have written enough for today. BTW - speaking about BAM, I have noted the posting of James Taylor - "why are enterprise application so dumb?" - doubting the benefits of presenting observation to humans, instead of taking automated decisions - food for thought. More Later
Sunday, February 3, 2008
On Event Processing and Now.
After some more technical postings, back to macro-level issue. David Luckham has written recently in his website about the history of CEP (I'll refer to the content of this article in another posting). All indications are the event processing is not a new thing, however, as some people indicate, there are a lot of interest, events, maybe hype around it NOW - the question is what is new ?
The first observation is that unlike the past, today there are commercial "on the shelf" products whose main purpose is to provide event processing capabilities, while in the past there were event processing capabilities in other types of products (simulation, databases, network and system management, middleware, real-time systems etc..), there is also a start of event processing as a discipline. What are the reasons for the interest now - some of what happened in the last few years that supported the shift from hard-coded event processing functionality to COTS are (based on discussions with people in multiple industries):
- Some contemporary applications which are event-driven by nature - such as: compliance with regulations, the need to detect frauds as two examples - have become pervasive in multiple industries
- The increasing complexity of inter-process integration that is simplified with event-driven interaction
- The need for flexibility and agility to gain market advantage in different areas - thus, move away from hard-coded solutions.
- The substantial growth in the number of available events - e.g. since RFID technology became pervasive.
- Some industry trends like - BAM, RTE, "on demand" - that are also based on responding to events.
- The drive to save expenses in back offices by automating exception handling - trends like STP.
I am sure that there are more of these.
Thus, while event processing functionality is not new - "event processing" as a first class citizen in the computing world - with its own dedicated products, community and emerging discipline is new. More - Later
Monday, January 14, 2008
CITT BAM-BPM-CEP-EDA conference Day 1

This is another picture from Regensburg - a picturesque medieval town on the Donau. Today I have attended the first day of the CITT conference organized and navigated by my good friend Rainer von Ammon .
Saturday, January 5, 2008
On Trifecta and Event Processing
I had to admit that I don't have any clue what a "Trifecta" is, and rushed to wikipedia
So today I have learned about horse races and promised some more postings (not promising when) -- more, later.
Monday, December 31, 2007
Has BAM failed ?

- The sin - BAM solutions were developed outside the true religion of BI;
- The punishment - BAM failed;
- The redemption - Embedded BI applications have included the concept of BAM -- and lived happily ever after.
In order to analyze this claim - let's first define what "BAM solutions" are. BAM - Business activity monitoring (or management) stands for a class of applications that observe the behavior of business activities (business processes, transactions, applications etc..), in a non politically correct terminology (well, we are lagging behind USA, we have not invented political correctness yet) I can call it "the big brother is watching", a term that is sometimes used is "observation" - the essence is -- look at business activity from the outside, and try to find situations that are of interest - can be major situations like - threats and opportunities, but most of them are micro issues that need to be observed, maybe fixed, maybe just watched closely.
Now for the question of failure or success -- Bill Gassman from Gartner claims in his talks that 65% of the BAM solutions are industry specific, and general purpose software accounts only to 35% of the BAM solutions. Indeed, while the notion of observation is general, what we would like to observe and how is very much domain dependent. Performance management that trace Key Performance Indicators is one flavor that has been generalized, but many of the BAM solutions are not about tracing aggregates or measurements, but about tracing individual situations in the business activities: delivery has failed, to meet the deadline, customer has complained and not handled well, medical treatment has been done outside the protocol for that disease - there are examples from all industries for it. Event Processing is a useful technology in cases where the BAM solution is event-driven, and its input is event. Some of the BAM situations are detected on-line, some of them are detected in retrospect - and I'll talk about retrospective event processing in another posting. General BAM systems are useful for some cases - and are less useful for others.
So - my 2 cents opinion about Colin White's claims:
- The concept of BAM has not failed - there are many success stories, however the area has not reached maturity. General purpose BAM are effective for some of the BAM market, but many BAM solutions are more industry-specific -- event processing is useful both for the general-purpose BAM market and for the industry-specific BAM market.
- And as for BAM on top of BI -- according to Gassman this accounts for 10% of the BAM solutions market, so while it has certainly its own variety of application, the majority of the BAM market is not a BI market.
P.S - the picture above is of a skateboard called BAM -- and I hope it sells well - click for a free advertisement to skateboards
More - Later.
Sunday, December 30, 2007
Event Processing for Business Intelligence
From the various descriptions of Business Intelligence found on the Web, I have chosen the one by Hakki Erbug as a starting point.
As noted in several previous postings, IMHO, event processing is a set of technologies that have multiple usages, and are not really strongly associated with a single type of application. Business Intelligence is similar in the fact that it combines various technologies, but different in the fact that it is focused around area of using data for decision making in various ways. In the event processing case, decision making is one of the possible usages, but not the only. This posting will briefly survey how event processing can enrich business intelligence, and in fact, in the Gartner EPS summit I have seen several BI vendors that are looking at EP as a natural capability to complement their products.
Going on Erburg's illustration clockwise:
1. "Active Data Warehouse" - While traditional warehouses are being updated in batch, the notion of active data warehouse makes the warehouse update itself an event-driven action. The rationale is: when a certain event (raw or derived) occurs, a decision has to be taken, however, the decision relies on a data warehouse, thus, an update of the data warehouse should occur before making this decision. The update can be of the same event that happens or of some collection of data that has still not been updated in the data warehouse and is needed for the decision making. There can be some time constraints associated with the decision (and in turn with the warehouse updates). The time constraints are not necessarily micro-seconds, the constraints can be minutes or hours, but they are typically well-defined.
2. ETL and mediated event processing: ETL has some functional similarity with mediated event processing, it is also about transformation. We see mediated event processing and ETL getting closer to one another, where difference may be in quality of service. In the future there may be a case that ETL will become a specific case of mediated event processing (of course ETL folks may say the same from the opposite direction).
3. Real-time analytics: While analytics (simulation, optimization, mining...) has been used for a while in the decision making part of BI, in the event-driven world, the reaction to an event in some cases is temporally-bound, which means that there is a real-time constraint, or upper limit on the requested reaction time. This provides new way of thinking about analytics - while without time constraints an optimization should strive to get the "best result" (or if heuristics satisfy some approximation condition), in real-time analytics the optimization strives to get "the best result that can be obtained in T time-units as specified (e.g. 18 seconds)". How does event processing play in real-time analytics? it may play in a simulation mode - scenarios are created and simulated events are emitted - they in turn may create simulated derived events which determine the situations of this simulation. This is, of course, in addition to the fact that in an event-driven universe, the entire BI cycle is event-driven and relates to the event and its context.
To conclude -- event processing is a natural step in the BI capabilities, and thus I expect BI suites to support the event-driven flavor... This - again, does not say that BI is the ONLY use of event processing. I still need also to refer to the issue - "has BAM failed because it was not based on BI techniques" as claimed in the article that triggered my discussion on the BI topic - stay tuned.
Friday, September 14, 2007
In-Line vs. Observations
Anyway, last post I have discussed RTE vs. BAM, and succeeded to confuse some people, so it needs some clarification. First - as said before, event processing is a collection of functions, that derive, detect pattern, enrich, transform, and some other functions that take events as input, processes the events, and produces (possibly other) events as output.
The distinction is - where, relative to running transactions, the event processing is done. In-Line event processing means that the event processing is done inside a transaction, not observing the transaction from the outside. In this case, the event creation may be part of the transaction (e.g. an application emits event, but the result actions belong to the same transaction), or a starting-point for a transaction. To give a concrete example (repeating the same example from the previous post) - some application emits an event saying "cancel order" (this by itself may or may not be a result of event processing function), this should be atomic with the execution of the cancellation, if cancellation is not possible, then some compensation is needed in the producer application. Likewise, an event can start a transaction that spans all its descendants and actions, if an action in the consumer cannot execute, or there is some validation issue, the entire thread of the "event processing network" should abort/compensate, this is in contrast to "observation mode" -- observe if there is some anomaly, and if there is - notify somebody. In the In-Line case, the transaction can change its course due to event processing, in the observation case - the impact is indirect. Of course, in some cases we have combinations of all. There are two issues that should be discussed: what is the relationship between "observation" mode of event processing to what the market knows as BAM ? another question --- what is the semantic meaning of rollback or compensation when talking about events? more discussion on each of these issues - later. Happy New Year.