On Trifecta and Event Processing

Reading the educated blog of my friend Tim Bass
I had to admit that I don't have any clue what a "Trifecta" is, and rushed to wikipedia

for help to find out that in horse races - it is a bet on the first three horses. I must confess to my ignorance - I have never watched a horse race, neither have I gambled on horses. I have made a major technological gamble though, around 10 years ago, to bet around "complex event processing" (without using this name and without knowing on the three others who have worked in parallel on the same topics - will have at some point postings about the history), and after suffering some frustration periods, it seems that the reality also slowly getting there, and I still hope it will be the "next big thing" - but we have some work to do in order to get there -- again, a topic for another posting. Now the question posed both by Tim Bass, and by Joe McKendrick

is - whether CEP is depended on SOA, and in turn - does it require the coupling of SOA and BPM - which is by itself a problematic concept.

My answer is -- CEP is an horse that is indepdent in any other horse, donkey, mule or tiger. As I noted in earlier postings, SOA and EDA are orthogonal - SOA is about modularization or componentization of the enterprise, and hence it IT systems, introducing modules as "services". Now let's talk about the relations between SOA, CEP and BPM.

(1). Services in SOA can be producers and consumers for "Event Processing Agents"; as a producer - the service is instrumented to emit events, as a consumer it can be notified or triggered by an event; Services are one type of consumers and producers but not the only type - every application can be consumer or producer, thus there is a loose coupling.

(2). In SOA environment there is an ESB (Enterprise Service Bus), whose functions have some overlap with event processing (especially mediated event processing) - can be a natural host for the "event processing network" (I'll write more on ESBs in another posting). However, again Event Processing Agents can also run outside ESB.

(3). Workflows (BPM) are special type of consumers and producers. In the SOA world, BPM orchestrate services, and thus, BPM can emit events about the status of the workflow, while as a consumer it can add/modify/delete one or more workflow instances. Some of the Event Processing applications (e.g. some types of BAM applications) are based on business processing - however, even in SOA environment this is only one type of consumer and producer, and there are other types (e.g. RTE applications which process events in-line and not in observation mode).

Bottom line --- Event Processing can have different interactions with SOA, and when IBM's announcements in this area will be available you'll realize that there are different entry points. Event processing can also work in legacy and non-SOA environment.

So today I have learned about horse races and promised some more postings (not promising when) -- more, later.

IBM WebSphere CTO sees CEP as SOA's 'next big thing'

In Israel there are sometimes news items that are forbidden by the military censorship, but if broadcasted by foreign sources - one can quote the foreign source and broadcast it in Israel.

I have not referred in this blog to IBM plans in this area, since they have not announced yet by the proper channel, but I may quote the interview with IBM Websphere CTO, Jerry Cuomo

in which he stated that he sees CEP as SOA's next big thing -- and promised both new technology and additions to existing ones. Stay tuned for more information on IBM and Event Processing.

Has BAM failed ?

Last posting in the year 2007 - we don't celebrate new year as an holiday, but since the civilized world is using the Gregorian calendar and not the Hebrew calendar we tend to celebrate holidays according the Hebrew calendar and do everything else by the Gregorian calendar.

Anyway - this posting is another in the series of responses to Colin White's article. This time I'll refer to one sentence in that article: This is somewhat analogous to the way business activity monitoring (BAM) solutions were developed independently from the existing business intelligence environment. BAM failed, and the solution was to instead build embedded BI applications that supported the concept of BAM, but which also were integrated with traditional BI processing. The claim is simple:

• The sin - BAM solutions were developed outside the true religion of BI;
• The punishment - BAM failed;
• The redemption - Embedded BI applications have included the concept of BAM -- and lived happily ever after.

In order to analyze this claim - let's first define what "BAM solutions" are. BAM - Business activity monitoring (or management) stands for a class of applications that observe the behavior of business activities (business processes, transactions, applications etc..), in a non politically correct terminology (well, we are lagging behind USA, we have not invented political correctness yet) I can call it "the big brother is watching", a term that is sometimes used is "observation" - the essence is -- look at business activity from the outside, and try to find situations that are of interest - can be major situations like - threats and opportunities, but most of them are micro issues that need to be observed, maybe fixed, maybe just watched closely.

Now for the question of failure or success -- Bill Gassman from Gartner claims in his talks that 65% of the BAM solutions are industry specific, and general purpose software accounts only to 35% of the BAM solutions. Indeed, while the notion of observation is general, what we would like to observe and how is very much domain dependent. Performance management that trace Key Performance Indicators is one flavor that has been generalized, but many of the BAM solutions are not about tracing aggregates or measurements, but about tracing individual situations in the business activities: delivery has failed, to meet the deadline, customer has complained and not handled well, medical treatment has been done outside the protocol for that disease - there are examples from all industries for it. Event Processing is a useful technology in cases where the BAM solution is event-driven, and its input is event. Some of the BAM situations are detected on-line, some of them are detected in retrospect - and I'll talk about retrospective event processing in another posting. General BAM systems are useful for some cases - and are less useful for others.

So - my 2 cents opinion about Colin White's claims:

• The concept of BAM has not failed - there are many success stories, however the area has not reached maturity. General purpose BAM are effective for some of the BAM market, but many BAM solutions are more industry-specific -- event processing is useful both for the general-purpose BAM market and for the industry-specific BAM market.
• And as for BAM on top of BI -- according to Gassman this accounts for 10% of the BAM solutions market, so while it has certainly its own variety of application, the majority of the BAM market is not a BI market.

P.S - the picture above is of a skateboard called BAM -- and I hope it sells well - click for a free advertisement to skateboards

More - Later.

Event Processing for Business Intelligence

From the various descriptions of Business Intelligence found on the Web, I have chosen the one by Hakki Erbug as a starting point.



As noted in several previous postings, IMHO, event processing is a set of technologies that have multiple usages, and are not really strongly associated with a single type of application. Business Intelligence is similar in the fact that it combines various technologies, but different in the fact that it is focused around area of using data for decision making in various ways. In the event processing case, decision making is one of the possible usages, but not the only. This posting will briefly survey how event processing can enrich business intelligence, and in fact, in the Gartner EPS summit I have seen several BI vendors that are looking at EP as a natural capability to complement their products.



Going on Erburg's illustration clockwise:



1. "Active Data Warehouse" - While traditional warehouses are being updated in batch, the notion of active data warehouse makes the warehouse update itself an event-driven action. The rationale is: when a certain event (raw or derived) occurs, a decision has to be taken, however, the decision relies on a data warehouse, thus, an update of the data warehouse should occur before making this decision. The update can be of the same event that happens or of some collection of data that has still not been updated in the data warehouse and is needed for the decision making. There can be some time constraints associated with the decision (and in turn with the warehouse updates). The time constraints are not necessarily micro-seconds, the constraints can be minutes or hours, but they are typically well-defined.

2. ETL and mediated event processing: ETL has some functional similarity with mediated event processing, it is also about transformation. We see mediated event processing and ETL getting closer to one another, where difference may be in quality of service. In the future there may be a case that ETL will become a specific case of mediated event processing (of course ETL folks may say the same from the opposite direction).

3. Real-time analytics: While analytics (simulation, optimization, mining...) has been used for a while in the decision making part of BI, in the event-driven world, the reaction to an event in some cases is temporally-bound, which means that there is a real-time constraint, or upper limit on the requested reaction time. This provides new way of thinking about analytics - while without time constraints an optimization should strive to get the "best result" (or if heuristics satisfy some approximation condition), in real-time analytics the optimization strives to get "the best result that can be obtained in T time-units as specified (e.g. 18 seconds)". How does event processing play in real-time analytics? it may play in a simulation mode - scenarios are created and simulated events are emitted - they in turn may create simulated derived events which determine the situations of this simulation. This is, of course, in addition to the fact that in an event-driven universe, the entire BI cycle is event-driven and relates to the event and its context.

To conclude -- event processing is a natural step in the BI capabilities, and thus I expect BI suites to support the event-driven flavor... This - again, does not say that BI is the ONLY use of event processing. I still need also to refer to the issue - "has BAM failed because it was not based on BI techniques" as claimed in the article that triggered my discussion on the BI topic - stay tuned.