Showing posts with label BAM. Show all posts
Showing posts with label BAM. Show all posts

Saturday, December 19, 2009

More on the ecosystem of event processing systems


The one week school break for children ends tomorrow, and today I went with my two younger daughters to see Avatar, a very ambitious movie with a lot of advanced 3D graphics. The downside is that the film spans over 3 hours without a break -- could cut 1 hour easily, but nevertheless the graphics is very impressive, the plot is kind of paraphrase on other movies.

In my previous posting I've written about event processing functions, one of the frequently asked questions is whether event processing has a value by its own right, or as part of a larger ecosystem. Let's look at another question: DBMS software deal with update, retrieve, store, organize, backup and restore data, it also supports concurrency control, transaction management and other stuff. DBMS itself does not have value, the value is in the use of data.

In a similar way, event processing software deals with transferring, filtering, transforming, pattern matching, and situation discovering, it also provides execution control and support of some non functional properties, but the aim is to receive event process them and derive more events. The value is not in derivation of events, but in the way it is used.

In order to complete the cycle we need two other types of software: event producers that produce the raw events, and event consumers that consume the derived events and execute the actions triggered by these events -- the consumers are those who are driving the value.

The producer can be any type of software or sensors. One example of event producer I've written about in the past is the "EDA extension of CICS" that I have written about before. It is an example in which a software is instrumented to produce events. We are seeing more software of this type in the sensor area, and other areas as well.

Consumers are the part of ecosystem that utilize the events flowing from the event processing systems. I've posted before about the consumer chapter in EPIA, There are many ways to consume events, they can also be connected with other types of enterprise software: Events can drive decisions, and can serve as input to business rules; events can drive KPI (Key Performance Indicators) metrics and can be input to BAM systems, it can have various actions related to BPM systems, events can drive real-time analytics and be input to BI systems, it can post on Twitter, and update Ambient Orbs (see the posting about the EPIA chapter), or if it is part of a "smart house" it can turn off lights or control the temperature using some actuators.

Like DBMS that requires somebody to feed the data, and somebody else to use it, event processing requires its ecosystem, we'll see more of the ecosystem software both in the producer side (like the CICS example) or in the consumer side (like BAM software). More on event processing ecosystem later.

Tuesday, February 5, 2008

On Killer Applications


My friend Tim Bass, the popular blogger, is the person who implored me into writing my own Blog again and again, until I decided to give it a try. Tim, whom I always enjoy to hear talking, even when we agree to disagree, has notified us in his Blog that he is going to participate in a Webinar about "BAM as a killer application for CEP" . While the Webinar has not yet occurred, and I don't have a clue what they are going to say - I would like to dedicate this posting to killer applications and EP, since this is not the first time that I've heard assertions about killer applications. To start the discussion, let's look at three assertions:
  1. BI is a killer application of databases
  2. Electronic commerce is a killer application of the World Wide Web
  3. BAM is a killer application of CEP

Before looking at assertion number 3, let's look at the previous two assertions:

  • Databases are used for many purposed, certainly BI uses databases, which implement data warehouses, but most of the uses in databases today are on operational systems, master data management etc - and BI is just one application. The opposite holds, one cannot do BI, without storing historical data, thus databases is a killer technology for BI.
  • Electronic commerce is certainly a growing area, and maybe there are people whose main usage of the web is electronic selling or buying, but today I have entered the Internet several times, none of them has been in order to buy. Actually, my own buying from the internet (I am buying books and songs - as I listen to music in the background when I am not in a meeting, so I have a collection of around 1500 songs now - and growing - only legal downloads!) issue a very small part of my use of the Internet, this is true for most persons, however, the converse holds - it is difficult to hold electronic commerce without the Web, so the Web is a killer technology for EC.

Now, back to the assertion that we inspect - "CEP is a killer application of BAM" - this would have been true if most of the CEP usages were BAM applications, Wikipedia description of BAM states: The goals of Business Activity Monitoring are to provide real time information about the status and results of various operations, processes, and transactions

Many of the BAM products concentrate around displaying Key Performance Indicators on dashboards, but even if we extend the notion of BAM, it is still observation on operations according to predefined measurements (typically aggregative ones).

Now the question - whether most CEP users are doing it through BAM ? according to my observation on the CEP market, the answer is - NO. BAM is an important application of CEP, but there are others - the early adopter application - algorithmic trading - is not really BAM - it is more RTE type - it makes decisions and not presents observations, system and network management applications are also mostly not BAM - they are diagnosis, attempting to find root cause for problems and not display measurements, and the same is true for information dissemination systems that don't monitor anything, and predictive systems that don't have measurements. Thinking about a sample of CEP applications I have looked at recently, there is certainly some that are of BAM type, but it is not the majority. CEP is being used for different purposes, and has different ROI to different people, I'll write more on the different ROI's in one of the next postings, thus, like databases, it does not have a single killer application.

The interesting question is if CEP is a killer technology for BAM or any other application type ? but - I have written enough for today. BTW - speaking about BAM, I have noted the posting of James Taylor - "why are enterprise application so dumb?" - doubting the benefits of presenting observation to humans, instead of taking automated decisions - food for thought. More Later

Sunday, February 3, 2008

On Event Processing and Now.



After some more technical postings, back to macro-level issue. David Luckham has written recently in his website about the history of CEP (I'll refer to the content of this article in another posting). All indications are the event processing is not a new thing, however, as some people indicate, there are a lot of interest, events, maybe hype around it NOW - the question is what is new ?



The first observation is that unlike the past, today there are commercial "on the shelf" products whose main purpose is to provide event processing capabilities, while in the past there were event processing capabilities in other types of products (simulation, databases, network and system management, middleware, real-time systems etc..), there is also a start of event processing as a discipline. What are the reasons for the interest now - some of what happened in the last few years that supported the shift from hard-coded event processing functionality to COTS are (based on discussions with people in multiple industries):
  • Some contemporary applications which are event-driven by nature - such as: compliance with regulations, the need to detect frauds as two examples - have become pervasive in multiple industries
  • The increasing complexity of inter-process integration that is simplified with event-driven interaction
  • The need for flexibility and agility to gain market advantage in different areas - thus, move away from hard-coded solutions.
  • The substantial growth in the number of available events - e.g. since RFID technology became pervasive.
  • Some industry trends like - BAM, RTE, "on demand" - that are also based on responding to events.
  • The drive to save expenses in back offices by automating exception handling - trends like STP.

I am sure that there are more of these.

Thus, while event processing functionality is not new - "event processing" as a first class citizen in the computing world - with its own dedicated products, community and emerging discipline is new. More - Later

Monday, January 14, 2008

CITT BAM-BPM-CEP-EDA conference Day 1



This is another picture from Regensburg - a picturesque medieval town on the Donau. Today I have attended the first day of the CITT conference organized and navigated by my good friend Rainer von Ammon .
The conference is held in an old historical building, and like the VLDB that was held in a historical building in Vienna, the architect of that building also is in the opinion that people love going up and down the stairs multiple times... Anyway -- I have learned several interesting things - one that is not really related to the conference-- one of the speakers revealed that there is a formula to calculate the value of any digit in pi without calculating the other digits - and indeed after some search in the internet (wikipedia does not seem to know about it) I have found it .
Other than that - some discussion about creating a consortium for an EP European project, some presentations about "fraud detection", a keynote address was given by Stefan Reid from Forrester Research - the topic has been - " ESB Open Source Challenge", and he discussed what is ESB and is it mature enough to be an "open source" topic. His ESB definition has been too broad to my taste (i.e. he sees business process modeling and monitoring as part of ESB, I think that they are not really ESB functions, and architecturally they are in another category). He also said that "in the end of 2008 any ESB that will not include CEP will be obsolete" - and I obviously like this statement. I think that ESB (in the SOA case) is the major (but not the only) carrier of CEP (but I'll discuss ESB in a seperate posting). Later in the day Rainer brought his entire class of students and they presented BPM oriented projects. The last presentation was done via Skype from Redwood City - two of Rainer's students that are doing a project in Oracle has presented their project, while the two CEP Oracle champions - Dieter Gawlick and Shailendra Mishra (my EPTS partners), have presented the upcoming Oracle CEP. Well - there is some room to improve the skype technology.... There was also in the middle a panel about BPM patterns - and somebody made the observation that people talk about technical patterns, while there is no enough work on business patterns - this is true in a way for EP as well, but I'll discuss patterns in another posting. Tomorrow - second day, where I'll have to earn my trip by giving my keynote talk entitled: "event processing as part of enterprise architecture" - well, since I am working in a "just in time" mode - I'd better finish this presentation.
PS.
Tim Bass complained in his popular Blog about the lack of possibility to read blogs and from China -- I have checked in the log of my blog (thanks to google analytics) -- it seems that there are (not many but some) people form China that are reading my blog - I even have one reader from Xiamen (most are from Beijing), so probably there is a way to overcome it. I have done a business trip to China (Beijing only) in 2004, and it has been very interesting experience, indeed when I watched CNN in my hotel, and they started to talk about demonstrations for democracy in Hong-Kong, there has been a sudden communication obstacle that has been resolved three minutes later, but this has probably been a conicidence.
I am also glad for Tim that democracy has been restored in Thailand after some period of military control.... things are fragile in our universe... Back to my presentation now - more later.

Saturday, January 5, 2008

On Trifecta and Event Processing


Reading the educated blog of my friend Tim Bass
I had to admit that I don't have any clue what a "Trifecta" is, and rushed to wikipedia
for help to find out that in horse races - it is a bet on the first three horses. I must confess to my ignorance - I have never watched a horse race, neither have I gambled on horses. I have made a major technological gamble though, around 10 years ago, to bet around "complex event processing" (without using this name and without knowing on the three others who have worked in parallel on the same topics - will have at some point postings about the history), and after suffering some frustration periods, it seems that the reality also slowly getting there, and I still hope it will be the "next big thing" - but we have some work to do in order to get there -- again, a topic for another posting. Now the question posed both by Tim Bass, and by Joe McKendrick
is - whether CEP is depended on SOA, and in turn - does it require the coupling of SOA and BPM - which is by itself a problematic concept.
My answer is -- CEP is an horse that is indepdent in any other horse, donkey, mule or tiger. As I noted in earlier postings, SOA and EDA are orthogonal - SOA is about modularization or componentization of the enterprise, and hence it IT systems, introducing modules as "services". Now let's talk about the relations between SOA, CEP and BPM.
(1). Services in SOA can be producers and consumers for "Event Processing Agents"; as a producer - the service is instrumented to emit events, as a consumer it can be notified or triggered by an event; Services are one type of consumers and producers but not the only type - every application can be consumer or producer, thus there is a loose coupling.
(2). In SOA environment there is an ESB (Enterprise Service Bus), whose functions have some overlap with event processing (especially mediated event processing) - can be a natural host for the "event processing network" (I'll write more on ESBs in another posting). However, again Event Processing Agents can also run outside ESB.
(3). Workflows (BPM) are special type of consumers and producers. In the SOA world, BPM orchestrate services, and thus, BPM can emit events about the status of the workflow, while as a consumer it can add/modify/delete one or more workflow instances. Some of the Event Processing applications (e.g. some types of BAM applications) are based on business processing - however, even in SOA environment this is only one type of consumer and producer, and there are other types (e.g. RTE applications which process events in-line and not in observation mode).
Bottom line --- Event Processing can have different interactions with SOA, and when IBM's announcements in this area will be available you'll realize that there are different entry points. Event processing can also work in legacy and non-SOA environment.

So today I have learned about horse races and promised some more postings (not promising when) -- more, later.

Monday, December 31, 2007

Has BAM failed ?


Last posting in the year 2007 - we don't celebrate new year as an holiday, but since the civilized world is using the Gregorian calendar and not the Hebrew calendar we tend to celebrate holidays according the Hebrew calendar and do everything else by the Gregorian calendar.
Anyway - this posting is another in the series of responses to Colin White's article. This time I'll refer to one sentence in that article: This is somewhat analogous to the way business activity monitoring (BAM) solutions were developed independently from the existing business intelligence environment. BAM failed, and the solution was to instead build embedded BI applications that supported the concept of BAM, but which also were integrated with traditional BI processing. The claim is simple:
  • The sin - BAM solutions were developed outside the true religion of BI;
  • The punishment - BAM failed;
  • The redemption - Embedded BI applications have included the concept of BAM -- and lived happily ever after.

In order to analyze this claim - let's first define what "BAM solutions" are. BAM - Business activity monitoring (or management) stands for a class of applications that observe the behavior of business activities (business processes, transactions, applications etc..), in a non politically correct terminology (well, we are lagging behind USA, we have not invented political correctness yet) I can call it "the big brother is watching", a term that is sometimes used is "observation" - the essence is -- look at business activity from the outside, and try to find situations that are of interest - can be major situations like - threats and opportunities, but most of them are micro issues that need to be observed, maybe fixed, maybe just watched closely.

Now for the question of failure or success -- Bill Gassman from Gartner claims in his talks that 65% of the BAM solutions are industry specific, and general purpose software accounts only to 35% of the BAM solutions. Indeed, while the notion of observation is general, what we would like to observe and how is very much domain dependent. Performance management that trace Key Performance Indicators is one flavor that has been generalized, but many of the BAM solutions are not about tracing aggregates or measurements, but about tracing individual situations in the business activities: delivery has failed, to meet the deadline, customer has complained and not handled well, medical treatment has been done outside the protocol for that disease - there are examples from all industries for it. Event Processing is a useful technology in cases where the BAM solution is event-driven, and its input is event. Some of the BAM situations are detected on-line, some of them are detected in retrospect - and I'll talk about retrospective event processing in another posting. General BAM systems are useful for some cases - and are less useful for others.

So - my 2 cents opinion about Colin White's claims:

  • The concept of BAM has not failed - there are many success stories, however the area has not reached maturity. General purpose BAM are effective for some of the BAM market, but many BAM solutions are more industry-specific -- event processing is useful both for the general-purpose BAM market and for the industry-specific BAM market.
  • And as for BAM on top of BI -- according to Gassman this accounts for 10% of the BAM solutions market, so while it has certainly its own variety of application, the majority of the BAM market is not a BI market.

P.S - the picture above is of a skateboard called BAM -- and I hope it sells well - click for a free advertisement to skateboards

More - Later.

Sunday, December 30, 2007

Event Processing for Business Intelligence



From the various descriptions of Business Intelligence found on the Web, I have chosen the one by Hakki Erbug as a starting point.



As noted in several previous postings, IMHO, event processing is a set of technologies that have multiple usages, and are not really strongly associated with a single type of application. Business Intelligence is similar in the fact that it combines various technologies, but different in the fact that it is focused around area of using data for decision making in various ways. In the event processing case, decision making is one of the possible usages, but not the only. This posting will briefly survey how event processing can enrich business intelligence, and in fact, in the Gartner EPS summit I have seen several BI vendors that are looking at EP as a natural capability to complement their products.



Going on Erburg's illustration clockwise:



1. "Active Data Warehouse" - While traditional warehouses are being updated in batch, the notion of active data warehouse makes the warehouse update itself an event-driven action. The rationale is: when a certain event (raw or derived) occurs, a decision has to be taken, however, the decision relies on a data warehouse, thus, an update of the data warehouse should occur before making this decision. The update can be of the same event that happens or of some collection of data that has still not been updated in the data warehouse and is needed for the decision making. There can be some time constraints associated with the decision (and in turn with the warehouse updates). The time constraints are not necessarily micro-seconds, the constraints can be minutes or hours, but they are typically well-defined.

2. ETL and mediated event processing: ETL has some functional similarity with mediated event processing, it is also about transformation. We see mediated event processing and ETL getting closer to one another, where difference may be in quality of service. In the future there may be a case that ETL will become a specific case of mediated event processing (of course ETL folks may say the same from the opposite direction).

3. Real-time analytics: While analytics (simulation, optimization, mining...) has been used for a while in the decision making part of BI, in the event-driven world, the reaction to an event in some cases is temporally-bound, which means that there is a real-time constraint, or upper limit on the requested reaction time. This provides new way of thinking about analytics - while without time constraints an optimization should strive to get the "best result" (or if heuristics satisfy some approximation condition), in real-time analytics the optimization strives to get "the best result that can be obtained in T time-units as specified (e.g. 18 seconds)". How does event processing play in real-time analytics? it may play in a simulation mode - scenarios are created and simulated events are emitted - they in turn may create simulated derived events which determine the situations of this simulation. This is, of course, in addition to the fact that in an event-driven universe, the entire BI cycle is event-driven and relates to the event and its context.

To conclude -- event processing is a natural step in the BI capabilities, and thus I expect BI suites to support the event-driven flavor... This - again, does not say that BI is the ONLY use of event processing. I still need also to refer to the issue - "has BAM failed because it was not based on BI techniques" as claimed in the article that triggered my discussion on the BI topic - stay tuned.

Friday, September 14, 2007

In-Line vs. Observations

I am back in business after spilling water on my laptop (total-loss)... Fortunately, the hard-disk was not damaged, and I after getting the excellent IBM support to work in holiday, I have (another) laptop, not recommended to spill - water, coffee or anything else on computers...

Anyway, last post I have discussed RTE vs. BAM, and succeeded to confuse some people, so it needs some clarification. First - as said before, event processing is a collection of functions, that derive, detect pattern, enrich, transform, and some other functions that take events as input, processes the events, and produces (possibly other) events as output.

The distinction is - where, relative to running transactions, the event processing is done. In-Line event processing means that the event processing is done inside a transaction, not observing the transaction from the outside. In this case, the event creation may be part of the transaction (e.g. an application emits event, but the result actions belong to the same transaction), or a starting-point for a transaction. To give a concrete example (repeating the same example from the previous post) - some application emits an event saying "cancel order" (this by itself may or may not be a result of event processing function), this should be atomic with the execution of the cancellation, if cancellation is not possible, then some compensation is needed in the producer application. Likewise, an event can start a transaction that spans all its descendants and actions, if an action in the consumer cannot execute, or there is some validation issue, the entire thread of the "event processing network" should abort/compensate, this is in contrast to "observation mode" -- observe if there is some anomaly, and if there is - notify somebody. In the In-Line case, the transaction can change its course due to event processing, in the observation case - the impact is indirect. Of course, in some cases we have combinations of all. There are two issues that should be discussed: what is the relationship between "observation" mode of event processing to what the market knows as BAM ? another question --- what is the semantic meaning of rollback or compensation when talking about events? more discussion on each of these issues - later. Happy New Year.