Some thoughts on data mining and event processing - take one

Somebody with whom I've talked to last week said with some irony that in order to get attention today - no matter what you do, you have to say that it has something to do with clouds, and that it performs some kind of analytics. Well - it is a clear bright day today here without any cloud, so I'll delay the discussion about clouds to rainy day.   As for analytics, there are various types of analytics, today I'll write something about data mining and event processing.  There are two sides here: what event processing can do for data mining, and what data mining can do for event processing.  Let's focus the discussion now about the second issue.  The answer seems easy,  an event processing application is modeled by event processing network, that consists of event processing agents, which in most current implementations are implemented by rules/queries or some other constructs.  Today the application is being composed manually using some authoring tool -  however, there is a frequently asked question, can the computer somehow use magic to compose the application itself -- this is a natural candidate problem for data mining.    The achievements of data mining in event processing until today are somewhat modest, but there still might be a promise there.


So - let's go further and explore the potential.   We can look at three types of functions  that an event processing application may be assisted by data mining:

1.  Case in which we are looking for anomalies in general, the data mining can assist in identifying that we have anomaly now,  this is actually a different type of application in which there are no preset patterns.
2. Case of  detect of trends/thresholds oriented, where the thresholds can be adjusted by data mining
3. Case of pattern detection - where the patterns can be determined by data mining.

The first type is a classification issue - and this can be done by some types of learning of what is normal behavior. 
The second type -- learning thresholds also has some known methods in data mining.
The main problem is the third one -- learning patterns.   There are several difficulties there, the first one is the intent;  data mining typically discovers events that happen together, this by itself may not be of interest, since the aim of patterns are to detect situations that require reaction, thus there is some additional semantic knowledge here that is not captured by data mining without providing additional informations, furthermore the pattern may occur very rarely, such that it will not be captured within the existing data; another difficulty is the richness of pattern types and the various variations of patterns, so looking at the space of large possibilities.  
Successes in this area were typically limited to a certain type of pattern within a certain temporal window -- for example, there was some work that I familiar with to mine sequence of two events within a given temporal window, this again belongs to events that happen together, where a human has to go over all combination and decide whether they create an interesting situation.     


Bottom line -- no magic bullet,  but any breakthrough in this area will be helpful  

On the boundaries of event processing

We are in the Passover vacation, to celebrate the biblical story of getting out from Egypt where the sea has split and people could move in the gap that was created, well -- I guess that at that time people just walked, but if it would have occurred today it might have looked like this.
Today I would like to write something about the "boundaries" of event processing, based on some discussions last week, related to writing a book about event processing. There are two issues related to the scope:

• Is pre-processing to emit event by the producer, and post-processing of events by the consumer are part of the event processing systems?
• Are pre-processing to obtain the event processing patterns that has to be monitored (i.e. using machine learning techniques) part of the event processing systems?

From the point of view of "event processing language", if we'll include the pre-processing and post-processing we'll have to extend the language to have the expressive power of any programming language, which will loose the focus on specific event processing functionality. Thus, while an application may require pre and post processing, this is typically outside the "event processing network". The main point of using "event processing language" and not hard-coding the event processing functionality in Java, C# or any other imperative general-purpose language is using higher level abstraction. As an analog, before the days of SQL we had to read from the database, loop over a record, and evaluate the conditions in hard-coded way, SQL did not provide anything we could not write in Cobol or PL/I (the languages of that time...), but just provided a more concise way to write it. The situation in event processing is similar, we can write something that is specified as:
" Match a pattern of events which is a conjunction of type E1, E2, E3 that refer to the same person and all occur within one hour since an event of type E0 for the same person, if there are several instances of E1, E2, E3 take the most recent of each at the point that the match occurred, and if there are multiple matches within this same time interval, ignore all but the first". Of course, one can write it in Java, but a language that enables to write this pattern in less than 1 minute is more cost-effective.

Back to the scope -- pre and post processing of events and patterns are not part of the event processing system, and typically done in different technologies. This does not say that they are not important, sometimes the pre-processing of events is more complicated than the event processing, especially since it is hard-code.

More on this - later

More on Event Pattern Detection and Discovery

One cannot ignore these days the change of president in the USA, something with affects the entire universe. One minority the Mr. Obama belongs to is the minority of left-handed people, as can be clearly seen in the picture, while four of the last USA presidents were left handed (which make his fifth in the last seven presidents), conference rooms in the USA or university classes - all of them have desks only for the right handed majority. Here is a picture of a left handed desk,
I am sure that the USA president has much more urgent items on his agenda, however, unlike his predecessors he may also do something for the deprived minority of left-handed people. BTW - the situation in Israel (whose current prime-minister is, surprise...a left-handed person) is somewhat better. We, Left handed people may be a small minority (around 10% of the population) but our collective impact on humanity is unproportionately huge -- starting from Alexander the great, Julius Caesar, Napoleon, Queen Victoria, Lewis Carrol, Mark Twain, Escher, Michelangelo, Leonardo da Vinci and many more....well enough of that for now.

Going back to one of my previous posts
that has explained the difference between event pattern detection and event pattern discovery.
In the wake of some questions, here is more about the relationship between these two terms:

• Event pattern detection is performed for patterns that are known in advance, the pattern detection in done "on-line" when the event occur.
• Event pattern discovery is performed typically off-line, it can use machine learning techniques on past events in some cases, it can also use some natural language understanding technique to derive pattern from legal documents (e.g. regulations) in other cases.
  
• A pattern discovery creates patterns that are detected on-line by pattern detection, so they are complementary techniques.
• In some cases there is continuous discovery, and thus the patterns are updated in a dynamic way, however, still the discovery feeds the detection part on-line, and the respective roles are preserved.
• Last but not least, the discovery process may use simulation techniques that use detection of simulated events in order to check assumptions about patterns.
  

Typically, event processing products contain event pattern detection capabilities, in one form or another. The event pattern discovery is considered as add-on, typically using techniques that are not particular to event processing.

On some EP related conferences

As I said all I had to say about the EDA vs. CEP issue, and does not wish again to get inside discussion of "what is CEP", I'll write today about some conferences in the event processing area. In the top of this page I've put the logo of DEBS 2009 which became the flagship research conference of the event processing community. The conference that has become ACM conference will include, like last year, industrial track that will contain - experience reports, architectures, interesting applications, and anything else of interest from the industrial perspective. We'll also try to solicit intermediate reports of the EPTS work-groups. Follow the call for papers in the conference website.

Another conference that announced extension in submission : the AAAI symposium on intelligent event processing - submission extended to November 13th.

Happy conferencing.

event processing meets artificial intelligence

Bedford, MA, USA.

In the EPTS symposium last week, Alan Lundberg from TIBCO, who moderated the "business panel" made the analogy to AI, especially to "Experts systems", saying that there was a hype in the beginning, and people believed it will solve many of the world problems, and in the reality, it did not recover from sliding down in the hype cycle, this triggered the (somewhat surprising to some) response of Brenda Michelson, that actually EP is under-hyped, and its place in the hype-cycle is much lower in the climbing phase than the Gartner analysts draw, this is the diagram that Brenda presented with "event processing" in orange, way below SOA (in blue), BPM (in red), and Web 2.0 (in green).

Anyway - this is not the topic of today's Blog, but going back to the AI issue. The term AI is interesting, in the sense that it has spawned several disciplines (e.g. robotics, image processing, information retrieval, data mining and more) which are based on AI principles, but when they mature they stop being AI and become disciplines of their own. This is the same phenomenon we have for philosophy - the mother of all arts and sciences - many disciplines has emerged from philosophy, but when they depart, they are not considered as philosophy anymore. Event processing as a young discipline, is a descendent of multi disciplines as stated in the past, AI is certainly one of them.

What are the current topics in which AI touches event processing?

1. Modeling: the basic term "situation" and "context" have been taken from AI (situation calculus), conceptual modeling is important for design of EP applications, AI techniques can help here

2. Discovery: Prediction of events, mining of patterns - these are all derivatives of machine learning in AI.

3. Reasoning: Defining precise semantics of both event processing languages and execution models. Evidently from the recent discussions in the community, this becomes an important topic - again, precise reasoning of both the regular case of event processing, and the extended case of handling uncertain events.

As my colleague Guy Sharon described in the research session of the EPTS meeting, we in IBM Haifa Research Lab (together with some colleagues in IBM Watson Research Center) are engaged in the "Intelligent Event Processing" project that concentrates now on the discovery aspects, however, the idea is to extend the activity probably through collaborative work with the academia, as part of this collaboration we are organizing the "Intelligent Event Processing" workshop which will take place as one of the AAAI spring symposium series that will take at Stanford University, March 2009. The idea is to have the EP community meet the AI community and create partnerships to deal with these issues... so - target this conference for paper submission and/or attendance. More - later.

On embedded intelligence within event processing application

In the previous post I have referred to the term "Intelligent Event Processing" - one question that I have asked - is this a new term ? how does it related to the other "X event processing" terms ? -- I am not sure if the term "intelligent event processing" will stick around, I would say that a better way to explain what it is may be - "embedded intelligence in event processing".

If we look at event processing architectures

- there are: producers who produce events, consumers who consume the processing results, and the EPN (Event processing Network) in the middle, which really does the processing. So where are intelligent techniques can help - here are some (real) examples:

1. In the producer - the producer has a video stream of all cars that pass below the camera, an intelligent process (using image processing techniques) isolates the license plate number of the car, and send it for further processing (security, traffic violation, billing etc..).

2. In the "meta-data" composition -- a "pattern detection" node typically looks at pre-defined patterns and attempts to detect them in run-time. In current applications the patterns are entered by the developers or users. In some cases the patterns are "moving target" like in - fraud detection -- if the patterns for fraud are discovered they are of little value, and thus in the other side of the law - people are constantly looking for new loopholes, thus, intelligent techniques, such as machine learning are used to refresh the patterns that are looked for in run-time. The run-time does not change - same pattern-detection mechanism, just different sources of where these patterns come from.

3. Intelligent nodes within the EPN -- in some cases the process of derivation of new events cannot be expressed as derivation expression and need some intelligent derivation process - e.g. a heuristic algorithm to determine the traffic light policies based on traffic events.


There are many more examples - like creating predicted events and more -- but this was more to give some flavor. Is it useful -- yes, it is useful for a variety of applications. Does every CEP application need embedded intelligence -- not really. More - later.

On Intelligent Event Processing - AAAI symposium

This is a picture of the beautiful Stanford University, I am happy to inform today that the 2009 spring symposia of AAAI has accepted our proposal to hold a symposium about "Intelligent Event Processing" that I am organizing together with two young and energetic colleagues - Nenad Stojanovic and Adrian Paschke. This will be an attempt to get the AI community involved in event processing, and help realize the vision of "intelligent event processing". How can AI help event processing: there are several ways, and all of them are reflected in the symposium description, here are the important ones in my opinion:

Event Processing Modeling - using AI techniques: advanced logics, semantic

nets etc..

Event Pattern Discovery/mining

Event Prediction

Reasoning with uncertain events

Event-based reasoning under real-time constraints

The list of topics on the site contqins some additional topics as well. As one of our missions taken (within the EPTS) is to help accelerate the advancement of the event processing area, and using intelligent techniques may help some types of applications (again -- getting back to the elephant metaphor of my past postings - just a leg, not all applications !), the dialogue with the AI community and some projects already launched is a step in that direction. We'll post "call for papers" on the epts site, soon.

On EP and Analytics

Recently, some of my fellow-bloggers have occupied themselves with the question whether analytics are integral necessary part from CEP, and without it CEP does not really exist. My good friend Tim Bass went further in his current Blog and called the current state of the practice in CEP as Snake Oil , well - here I return to my previous posting with the metaphor of a group of blind people touching an elephant and each of them touches a different side of an elephant and each is confident that he knows what an elephant is, and furthermore he is the only one who knows what an elephant is. One of the benefits (there also some shortcomings...) of working in a large company is the exposure to many types of applications that are very far apart, and yet, all of them can share the same infrastructure (with some variations), talking specifically about the issue of EP and analytics - there are some applications that you cannot really think of without using analytics like "fraud detection" - where we need always to look at patterns that were unknown before, and have been used when our software has blocked the previous set of gaps, thus if the blind person touches the right back leg of the elephant in the "fraud detection" side, he sees analytics as a must. However - likewise there are plenty of other application - probably on other legs, back or trunk that do not require analytics - some simple examples: A physician sets individual alerts based on combination of test results/monitors reading - when to alert the physician or nurse; exception handling in manufacturing process - where the exceptions types are well-defined (and translated again to some combination of events); monitoring security regulation of different persons that need to be accompanied to various places within a plant according to the tag type, and checking if an autorized escorting person exists within the required distance, which monitors regulation and does not require any analytics, this is a small sample of CEP applications I have noticed recently.

Personally I think that analytics are very important, and we'll see more and more applications in which they are required, like the fraud detection one I've already mentioned, smart auditing etc... As stated before, I think that the combination of EP and some intelligent techniques (like: machine learning, prediction, handling uncertain information), which I call "Intelligent Event Processing" is very important for going forward, and we intend to reach to the AI community by doing a first Intelligent Event Processing conference as one of the AAAI spring symposia - stay tunde to note on that,

Having said that, I cannot say that this is the majority of applications, currently most CEP applications do not require analytics, here I join the opinion of Hans Glide on this issue.



This assertion returns me to the picture on the top of this blog - showing the half-full glass syndrom. Some people look at the empty half glass and some look at the full half glass. First - let's take the optimistic "full half" approach -- indications show that there are CEP products that serve as platform to build applications that bring real value to real customers, these applications are in a wide variery of areas, in variety of products, some of the type of "time series" streams, and some are not (next week I'll give a tutorial in DEBS 2008 on patterns, so this will be an opportunity to blog about types of patterns). The fact that they exist, is an indication that the EP indusrty is not just promoting "snake oil"
It is more interesting, as a scientist, to look at the empty half glass - which I think is even more than half. I think that the EP discipline is in its infancny, it already walks some steps, can say a few words, but we need to invest more until it will run, dance, sing and write Blogs... However, as every parent can testify, huge progress happens since from being a newborn until getting to the state I've described, so I don't underestimate the work done so far, and the traction achieved in the market, and think the entire community believe that we just scratched the surface of the potential. More - later.

On the three meanings of CEP

There is an old Jewish story on two people who had some dispute, and decided to go to the Rabbi and ask his opinion. The Rabbi listened to the first person and told him: you are right, then he listened to the second person and also told him: you are right. The Rabbi's assistant who has been present asked him: Rabbi, how can they both be right ? and received the obvious answer: you are also right.

Recently, a dispute between two opinionated persons - Hans Glide and Tim Bass has stormed the network, and somehow got also to my Blog. Somehow both of them understood that what I've written is consistent with their view - which encourages me to change career and become a Rabbi, but I think that there are some skills I lack - so anyway, I'll stay on event processing thinking.

While the dispute started around POSETS, the last posting by Tim Bass in the CEP forum re-focuses the discussion around - what is CEP ? so I'll stay with this topic for a while, since I think that there are (at least) three different interpretations of what CEP is - and this is a source of a lot of confusion. Thus I'll try to explain the three interpretations.

Interpretation One: the glossary interpretation -- complex event is the processing of complex events, where complex event is an abstraction or aggregation of events. According to this interpretation, a software function can be defined as CEP if it involves the processing of multiple events - typically collecting or matching some pattern among multiple events; The test for CEP according to this interpretation is typically support of a state that accumulate the relevant events - since they typically arrive over time. This definition does not say anything about the number of events, number of event types, whether they are totally ordered or not, or whether causality relation is supported or not - these are all attributes of specific implementations.

Interpretation Two: Event Processing = Complex Event Processing. This is a common practice in the commercial work to label all event processing functions as CEP. This spans from support of CEP (according to the interpretation one) and other event processing functions (such as: routing, enrichment, transformation) that don't satisfy the CEP test (since they are stateless and deal with a single event at a time). It can get to an absurd that some product that does not support CEP at all according to interpretation one, calls itself CEP. This is the most confusing interpretation, IMHO.

Interpretation Three: Complex event processing is event processing that has some complexity associated with it. According to Tim Bass - hidden causality and Markov Processes are vital for something to be defined as CEP. This really says that it CEP must involve uncertain events, causality that need to be discovered (by mining and other techniques), and the general usage of CEP is to predict something according to analysis of recent (past) events. According to Interpretation three, indeed the products that current products that call themselves CEP, do not satisfy this criteria, and thus are not CEP.

My opinion: as stated in the past, the term CEP has some inherent ambiguity, therefore I always thought it is confusing term. As far as my own taste in terminology - I prefer Interpretation One, saying that CEP is a subset of EP functions that deal with "complex events", it also seems that this is the closest to glossary definition. Interpretation two is confusing, as it turns CEP from a well-defined term to a marketing buzzword, and thus there is no test for what it is. Interpretation three is interesting, there are certainly applications that require prediction and various usages of stochastic processing and use of AI techniques (machine learning and others) in event processing. Hidden causalities is an important term, and I'll refer to it in another posting, since it has some pragmatic difficulty to obtain. However, I prefer to stick with the concept that CEP is processing of complex events, and not complex processing of events, and for interpretation one, we don't really need (necessarily) to apply AI techniques, this is just one type of application, there are a variety of applications that does not require it, and that detect predefined patterns on events is sufficient.

So the terminology that I personally prefer is:

Interpretation One = Complex Event Processing

Interpretation Two = Event Processing

Interpretation Three = Intelligent Event Processing.

Why I prefer to use "event processing" with prefix, infix or suffix - a subjective tour of acronyms

Recently there has been more discussions about terms and acronyms, I am not sure that this is so important issue to spend much time on, but before moving to a more interesting points, I would like to provide some personal thoughts about acronyms in this area.


First, as you can see from the Blog's name, I prefer the term "event processing" with any prefix, infix, or suffix. The reason is that I view it as a name of a discipline and not as trend. Disciplines typically consist of two words: signal processing, information retrieval, machine learning, software engineering etc.. although there are exceptions. Three letter acronyms AKA TLA, are typically not names of core disciplines but of other things - protocols, architectures, trends etc..


Historically, when the first "event processing symposium" (which created EPTS) has been established we needed a name - the original founders were - David Luckham, Roy Schulte, Mark Palmer (from Progress Software) and myself. David, of course, thought that CEP is an appropriate name for the discipline, while Mark proposed ESP - "Event Stream Processing" since he did not like the word "complex" (read further about it). Roy and mysrelf proposed to take the part that both agree "event processing". Both David and Mark were not completely happy, but agreed, thus we advanced with the name "event processing symposium" and used "event processing" ever since.



Getting back to history - I have prefered to use the name "active technologies" being a veteran of the active database community, and although the autonomic computing community adopted the "active" term and had conferences named "active middleware services", this name actually did not get into the main stream, David Luckham used the term "complex event processing" in his famous book that used the term. The term "complex event processing" has ambigious meaning - one interpretation is that this is processing of complex events, where complex event is an event that consists of more than one event (analog to complex object), the other interpretation is that this is complex processing of events. I have started to use the term CEP in 2004 to differentiate such functionaity from "event correlation" in system management since there has been some confusion in IBM around this terms. I also made a modest contribution to get the name CEP known by giving a tutorial in ICWS in July 2004, attended by many people, whose common denominator has been tht they have not heard this term before. Anyhow - there are two school of thoughts around CEP


Interpretation one ("the monolithic approach") : CEP = EP, everything is a subset of CEP.
Interpreation two ("the layered approach") : EP is a collection of technologies, whereas CEP is one of them (a link in the chain). Some people takes the first interpretation, saying that "simple" event processing (whether it is simple event or simple processing) is a subset of complex event processing, the rational behind it that if an engine is capable of doing complex things it is surely capable of doing simple things. Interpreation two comes from Roy Schulte (Gartner) who introduced in December 2005 the following slide:

In this slide Roy Schulte talks about four types of processing (later he realized that the BPM one is of another category) - simple event processing (filter and route), mediate event processing (transform and enrich) and complex event processing (statefull pattern detector). This is consistent with a market view since there are products that do only simple event processing (messaging), other products who do mediated event processing (ESB) and CEP as the next layer as a stateful engine. I think that this approach is liked by those who are putting CEP on top of existing middleware, while the first ("monolithic") approach is liked by those who have stand-alone CEP engine. Anyway - the existence of this two approaches, and the fact that people may not understand that the other person is taking the second interpretation is causing a confusion.

Next acronym has been "event stream processing", the term "data stream manager" has been coined in Stanford in a similar meaning, but with SQL API, and continuing with other academic projects, and some descendent products (Coral8 is a descendent of the Stanford project). When Progress Software acquired Apama, Mark Palmer looked for an alternative word for CEP, since he was in the opinion that customers don't like anything labelled "complex", thus, he borowed the term "stream" although Apama's API is not SQL, and has not much to do with the academic stream projects and introduced the ESP term "Event Stream Processing" (which was dropped later). In response, David Luckham published an article to defend the "complex" word, starting with the words: "some people, I'm told, get scared when they hear the word complex, as in complex event processing.... start with the basic question, is life simple ? most people when asked about it will truthfully answer no...." and the rest you can read yourself. It seems that David has won this battle -- all vendors (including the SQL oriented ones) at some point or another have positioned themselves as CEP vendors, which also created some objections - by people who thought that it is important to diffrentiate between ESP and CEP, some saying that ESP is a subset of CEP, and some that these are completely different focus areas - as I have written before, there are many ways to define subsets of EP functionality, and I did not find any evidence that the one defined by this distinction (totally ordered events vs. partially ordered events) is the important one (in many applications we need both types for different purposes).

What other acronyms have flown around ? - well, Forrester at some point made a distinction between CEP and BEM (Business Event Management) that has been defined as - "a process of capturing real-time business events from multiple source and assigning them to the appropriate decision-maker for resolution based on the business context of the events". I have struggled to understand the distinction - maybe the fact that it deals with simple events, however, when they mention context - determining the context may by itself require CEP.

We, in IBM are using the term IEP (Intelligent Event Processing) to denote stochastic and intelligent reasoning beyond the deterministic pattern detection to CEP; this is consistent with the layer approach, the monolithic approach fans, view IEP as part of CEP.

The new term we heard this week from IBM is BEP (Business Event Processing) and this is intended to define event processing applications in which the business user can control the behaior (i.e. define and modify patterns without the help of a programmer), a topic I also discussed in the past.

Last but not least, some people in the academic community don't like the term "processing" which they think is too elementary and talk about "event-based computing" as the name of the discipline.
After this unusally long postings, my bottom lines are :

(1). The upcoming glossary should provide a consistent taxonomy of terms here - there is still much confusion about the names, and the glossary can be a good reference point,

(2). Personally, I still prefer to talk about types of functions and not about boundaries of names, however, I understand the importance of branding.

(3). I still prefer the name "event processing" without prefix, infix or suffix - and thus continue to use this name.

(4). Hopefully, this is the last posting I am writing on the *-E-P; E-*-P; E-P-* topic - I have more interesting topics to deal with.... more - later.

On terminology again - BEP and CEP

There were several postings in the last few days from my esteemed colleagues - David Luckham, Tim Bass and Pual Vincent (in his comment to Tim's posting) all referred to the term "Business Event Processing" (BEP) that has been mentioned by Sandy Carter, VP of SOA and Websphere marketing in IBM. There were several references to this term relative to CEP.

The terminology issue is sometimes confusing, so let me clarify here: as I have discussed the
*-EP phenomenon in the past, so I'll dedicate this posting to clarification.

Event Processing, in general, is bigger in scope from Complex Event Processing; CEP deals with the detection of patterns over the a collection of events ("event cloud"). Event processing starts from stateless filtering and pub/sub, advances to mediated event processing - transformation, enrichment, validation etc, then to CEP, and to IEP (Intelligent Event Processing) that adds stochastic reasoning.

BEP (Business Event Processing) is an event processing applied to business applications. To Paul Vincnet's question - are there "non business" EP - the answer is -- yes, event processing can be used for individual person in smart homes; some people also see IT system management as a "non business" application, and differentiate between business and IT. What is the relations between BEP an CEP ? BEP is thus -- the entire EP applied to business application; CEP is subset of EP. More - later.

More thoughts on Rules in the context of Event Processing

I spent today (and tomorrow) in the IBM Hursley Lab - my second home in the last couple of years, this is a picture of the "Hursley House" (well - from the back side) - a countryside English manor that serves today as place for meetings and conferences, and the office of the Lab Director - besides this building there is a complex of connected building with multiple systems for room numbering that can provide in-door navigation challenges. I'll write today about some ideas that came out from discussions in the CITT meeting earlier this week about the role of rule technology. I still hold my opinion that although it is possible to take a technology that has been developed for one purpose and "hack" it to use it to other purpose, it may not be the most natural/effective/efficient way to do. This is true for SQL as well as rules when we are talking about pattern detection in complex event processing (I am working now on tutorial on the issue of patterns). However, this does not say that rule technology does not have a place in event processing in general. Here are some places it can be used:

(1). Decision-based routing in event processing networks.

(2). Transformation of events.

(3). Validation of events.

(4). Orchestration rules.

(5). Intelligent Event Processing.

Note that different type of rules are being used for the different cases -

for routing and transformation - it is typically - if-then rules/decision trees/decision tables.

for validation - constraint oriented rules.

for orchestration - ECA rules (note that orchestration rules are in the domain of the consumer that receives an event from the event processing network and has to decide what to do with it).

for intelligent event processing - all type of rules - deductive, inductive, abductive, rules with uncertainty - can play in different cases.

More about ECA rules and event processing - soon.

On situations

Hello from Zurich, got here in a train from Munich - and this is the famous train station of Zurich (in light), I have arrived in the dark, but has been in Zurich quite a lot of times before, tomorrow I am visiting the IBM Zurich Lab (which is actually located in Rüschlikon (a suburb of Zurich) home to four Noble prizes in physics. The hotel is 5 minutes walk from the train, but I forgot to print a map, and had to search a little. In the first half of the day I have participated in the continuation of the CITT conference, and earned my bread by giving a talk. After the talk I had some discussion with people about the term situation that I mentioned in the talk. I have borrowed this word from "situation calculus" in AI, and it fits well with terms in related domain such as: situation awareness. The discussion has been has been around one of the common misconceptions that situation is an alias to derived event. The answer is - no, these terms are not aliases ! Situation is some phenomenon that requires reaction (or notification). It is in the conceptual ontology of the consumer. Now - let's check the relationships with event:

(1). A raw event that is reported by a producer is a situation in the consumer's terms - in this case, only routing is required ("simple event processing").

(2). A raw event is reported by a producer has 1-1 correspodence to a situation in the consumer's terms, but more information or translation is required - in this case, the situation is created by mediated event processing.

(3). The situation occurs in a deterministic way once an event pattern is identified, and it is a functions of the events participating in the pattern - in this case the situation is created by complex event processing.

Note that the situation is not the derived event - but the derived event in this case is an indication that the situation is detected, note also that not all derived events are situations, derived events that are consumed by other agents, are not situations, only those that are consumed by event consumers.

However, situation detection may not be purely deterministic - and here we are getting to "intelligent event processing". Example: the pattern is only an approximation, example: the pattern is - "an event E1 happens at least 4 times within an hour". Now, if the event E1 happens 3 times and the pattern is an approximation, there is some probability that the situation did occur. This is even before looking at probabilistic events.

More about uncertainty in event processing - later.

Event Processing for Business Intelligence

From the various descriptions of Business Intelligence found on the Web, I have chosen the one by Hakki Erbug as a starting point.



As noted in several previous postings, IMHO, event processing is a set of technologies that have multiple usages, and are not really strongly associated with a single type of application. Business Intelligence is similar in the fact that it combines various technologies, but different in the fact that it is focused around area of using data for decision making in various ways. In the event processing case, decision making is one of the possible usages, but not the only. This posting will briefly survey how event processing can enrich business intelligence, and in fact, in the Gartner EPS summit I have seen several BI vendors that are looking at EP as a natural capability to complement their products.



Going on Erburg's illustration clockwise:



1. "Active Data Warehouse" - While traditional warehouses are being updated in batch, the notion of active data warehouse makes the warehouse update itself an event-driven action. The rationale is: when a certain event (raw or derived) occurs, a decision has to be taken, however, the decision relies on a data warehouse, thus, an update of the data warehouse should occur before making this decision. The update can be of the same event that happens or of some collection of data that has still not been updated in the data warehouse and is needed for the decision making. There can be some time constraints associated with the decision (and in turn with the warehouse updates). The time constraints are not necessarily micro-seconds, the constraints can be minutes or hours, but they are typically well-defined.

2. ETL and mediated event processing: ETL has some functional similarity with mediated event processing, it is also about transformation. We see mediated event processing and ETL getting closer to one another, where difference may be in quality of service. In the future there may be a case that ETL will become a specific case of mediated event processing (of course ETL folks may say the same from the opposite direction).

3. Real-time analytics: While analytics (simulation, optimization, mining...) has been used for a while in the decision making part of BI, in the event-driven world, the reaction to an event in some cases is temporally-bound, which means that there is a real-time constraint, or upper limit on the requested reaction time. This provides new way of thinking about analytics - while without time constraints an optimization should strive to get the "best result" (or if heuristics satisfy some approximation condition), in real-time analytics the optimization strives to get "the best result that can be obtained in T time-units as specified (e.g. 18 seconds)". How does event processing play in real-time analytics? it may play in a simulation mode - scenarios are created and simulated events are emitted - they in turn may create simulated derived events which determine the situations of this simulation. This is, of course, in addition to the fact that in an event-driven universe, the entire BI cycle is event-driven and relates to the event and its context.

To conclude -- event processing is a natural step in the BI capabilities, and thus I expect BI suites to support the event-driven flavor... This - again, does not say that BI is the ONLY use of event processing. I still need also to refer to the issue - "has BAM failed because it was not based on BI techniques" as claimed in the article that triggered my discussion on the BI topic - stay tuned.

CEP and the story of the captured traveller

Reading the recent posting of my friend Tim Bass entitled "CEP and the story of the Fish" I decided to answer with another story (from the other side of Asia) :

A traveller went in the jungle somewhere on the globe and unfortunately was captured by a tribe that is still using ancient weapons. He is brought to the chief, and the chief says - " You have trespassed into the tribe's territory, which is punishable by death, however, I am a very curious person, if you'll show me something I haven't seen before I'll let you go"; our unlucky traveller started to look in his pockets and the only meaningful thing he found was a lighter, so he took his chance, showing it to the chief saying: "this thing makes fire", however, since he was in under a big pressure, he pressed once - no fire, pressed twice - no fire, in the third time the lighter indeed has produced the promised fire, the chief did not hesitate and said "let him go", so our relieved traveller muttered to himself - "I knew that they have not seen a lighter", but surprisingly to him the chief said - "oh, I have seen many lighter, but a Zippo lighter that does not light in the first time I have never seen".

When someone disagrees with somebody else, it is very easy to assume that my point of view is right since I am smarter / knows more / more qualified / older / more experienced / generally always right etc... My preference is not to doubt the wisdom, experience or qualification of anybody that I am arguing / discussing / debating with, but make the arguments on the issue and not on the person who makes the arguments....

Enough introduction -- now for the main message of this posting, the term CEP (Complex Event Processing) has more or less agreed now in the industry to denote "computing that performs operations on complex events", where complex event is an "abstraction or aggregation of events". The term complex does not say that the processing is complex, but that it deals with complex events, as defined. Complex event processing is typically detecting predefined patterns that can be expressed by queries/rules/patterns/scripts and are deterministic in nature. Regardless if I think that this is the best term, I think that it is important to have common agreed terminology, otherwise we are confusing the industry, the customers (and sometimes ourselves). Now, Tim Bass claims that since event processing with stochastic/probabilistic/uncertain nature is more complex than what we call "complex event processing", we have to call this one - "complex event processing", and rename what we call "complex event processing" to be "simple event processing". Unfortunately, it is too late for that - and also not justified, again, since the "complex" in the "complex event processing" does not say that this is "complex processing of events" but that this is "processing of complex events" (very common misconception !). Bottom line: yes - there is another class of event processing capabilities that requires techniques from AI, machine learning, OR etc.. and that is not deterministic in nature; no - I don't think we should call it "complex event processing", we have suggested the term "intelligent event processing" which I have already referred to in previous posting , there are a variety of other postings that I have dedicated to terminology.

More - later

On CEP and IEP

Ambidexterity is a good property for a boxer, he can decide when is better to attack with his right hand, and when to attack with the left hand (I am part of the left-handed minority, should write sometimes a post about being left-handed in the Right-hand people's world). Likewise, there are problems in the event processing space that can be solved by deterministic means (rules, queries, scripts, patterns --- chose your favorite religion), and problems that are solved by stochastic means -- using probabilistic networks, machine learning etc.. (AKA IEP - Intelligent Event Processing). When there is a pattern that need to be traced , to check compliance with regulations, and the pattern is well-defined - then a deterministic approach should be used; when there is a need to dynamically change the traffic lights policies to have minimal waiting time of vehicle, there is a need to predict the traffic in the next few minutes - this is a non deterministic problem and require some stochastic tool (BTW - my student, Elad Margalit, is looking at the traffic lights issue as his M.Sc. thesis). Event Processing Platforms should include various types of functionality - which brings to another discussion on the "actor/agent" architecture - which I'll refer to in one of the next posts. more -later