On Event Processing Network and Situations - the semantic bridge

One of the challenges in building a semantic meta-language that captures event processing behavior is to bridge the gap between terms that are in different domains. I have written before about Situations following the CITT meeting, in which this term has been discussed. In fact, we have used the term "situation" in AMiT , we also called its core component the "situation manager" We used the term situation to denote a meta-data entity the defines combination of pattern and derivation, but I must admit that this has been a mismatch of terms, although there is a strong correlation among these terms.

First - let's go back to the operational world, in this world events are flowing within event processing networks, in the EPN there are agents of various types: simple (filtering), mediating (transformation, enrichment, aggregation, splitting), complex (pattern detection) and intelligent (uncertainty handling, decision heuristics... ), in the roots of the network there are producers, and in the leaves of the network there are consumers. This is an operational view of what is done, the term "situation" is in fact, not an operational term, but a semantic term, in the consumers' terminology and can be defined as circumstances that require reaction.

How can we move from the operational world to the semantic world - we have two cases here:

• Deterministic case: there is an exact mapping between concepts in the operational world and the situation concept;
• Approximate case: there is only approximate mapping.

In order to understand it, let's take two examples:

Example 1 - toll violation (determinstic, simple)

• The cirumstance that require reaction is the case that somebody crosses a highway toll booth without paying (in Israel the only toll highway is completely automatic, but we'll assume that it occurs elsewhere, and that the toll is not applicable between 9PM - 6AM and in weekends).
• Getting it to the operational domain - there are two cases - one: go in the EZ pass lane and don't pay, two: go in a manual lane and somehow succeed to cross the barrier (obstacle?) without paying.
• The action in both cases: apply camera to capture the license plate, SMS the picture to officer on duty in the other side of the bridge.

From the EPN perspective, we have events of car cross a certain section of the road (the raw event), the EZ pass reading is an attribute of this event, and if no EZ pass it gets a "null" value. There is context information which is --- temporal (it is in hour and day where toll is in effect), spatial (the location of EZ pass lane), note: sometimes people mix the notion of context with the notion of situation, I have explained the difference in the past. Within this context a filter agent that looks for null value in the EZ pass reading is applied, if the filter agent evaluates to true then the situation phrased has been applied in deterministic way, and indeed the edge going out of the filter agent is going directly to a consumer (the camera snapshot may or may not be considered as part of the EPN). This is a case of "simple event processing", stateless filtering, whose output is a situation. This gives a counter example to the misconception that situation is closely related with complex event processing [I can continue the example to the other case, but I think that you got the point by now]

Example 2 - Angry Customer (approximate, complex)

The setting is a call center, the situation is -- detect an angry customer -- refer him or her to the "angry customers officer".

Here the life is not that easy, a human agent can detect angry customers by the tone of their voice (or electronic message), but this does not include all cases of angry customers, so we can look at some pattern saying -- a customer that applied 3rd time in a single day is an angry customer, and then we need to have a "pattern detection" agent that detects the pattern "3rd instances of application" where the context partition refers to the same date, same customer, same product. In this case also a leaf edge is mapped to a situation, but there are two differences from the previous case:

1. The agent is now complex event processing agent since it detects pattern in mu;tliple agents;

2. The edge represents the situation in an approximate way, which means that it can have false positives (the CEP pattern is satisfied but the customer is not really angry, just asked for a lot of information to install the product), or false negatives (the customer called twice, and does not talk in an aggressive tone, yet he is furious).

In some case it also makes sense to associate "certainty factor" with the leaf edge, approximating the measure of belief in the fact that this edge represents the situation. I'll leave the discussion about uncertain situations to another time.

On event Driven BPEL

WS-BPEL has become a leading standard in the area of business process management, and since event processing is typically part of a bigger picture, it often interacts with business process management systems. One of the success factors is to make the environment ready for event processing, i.e. being able to get the right events, and being able to perform the right actions, while the event processing network has detected a situation (in one of the next postings I'll discuss how situation is related to the EPN world).

My student, Alex Kofman, has been doing recently a M.Sc. thesis looking from the BPEL perspective, he has determined what need to be done in order to extend BPEL to be "event driven", and proposed detailed modifications to the standard. This is intended to enable easy interaction between event processing system and BPM systems that plays as producer and consumer from the event processing viewpoint, while from the BPM viewpoint, input events can serve in decisions about orchestration, while output events can provide decisions external to the BPM based on the BPM state.

Here are some of the main points raised in this thesis:

1. Ability of BPM subscribe/unsubscribe to events.
2. Ability of BPM to publish events.
3. Ability to invalidate one or more processes as a result of an event (partially exists).
4. Ability to start a new process as a result of an event (partially exists)
5. Ability to invoke a task upon occurrence of an event
6. Adapt process execution upon occurrence of an event
7. Support the definition of an event processing pattern requested by the BPM engine, as a "callback" to an event processing system.
This in general will raise the level of abstraction and make the integration easier...

More about this project, later -- the thesis is now being written, Alex gave presentation about it last week in the department seminar, so this is a good time to start communicating these ideas.

On Event Processing Description Language

This is a logo of the UK Geologists celebrated their 15o anniversary. I have more modest accomplishment, the Blog dashboard claims that this is my 150th entry into this Blog, that have started almost a year ago (in the one year anniversary, I'll look for some statistics about it)..

In one of the first postings to this Blog I have talked about meta-language for describing event processing behavior as a possible candidate for first standard. Recently, I have been working further on this idea, and soon I'll be able to start talking more about the details (those who heard my tutorial in DEBS 2008 could get a sneak preview. The language is a semantic language whose roots are coming from two directions: the "outside in" direction and the "inside out" direction.

The "outside in" direction is a result of requirement survey that has been done internally in IBM in nine industries, by interviewing IBM domain experts (such as industry CTOs) and in some industries also selected customers. The "inside out" direction is a result of looking at existing event processing languages (both from products and from academic projects) and try to find the union (but also learn something from the interaction), we are now in the process of doing the "inside out" part, while the goal is not to compare language, but to learn from them - it is still interesting to look at different languages and at different assumptions that are reflected in features in the language (example: if the language assumes that the input is a time series, then counting events is equivalent to creating fixed time interval, while in other cases where events arrive in a sporadic/chaotic way these concepts are totally orthogonal). One interesting question is the "effective" expressive power of languages, which is somewhat different from the theoretical expressive power, since - given a specific requirements, with some (or a lot of..) hacking, sometimes adding code, one can achieve this goal, so in comparing language one should use a subjective goal -- how easy to express it / is it a natural language to express it / can a typical developer understand determine how to express it ? maybe it can be translated to a quantitative measures -- length of solution, development time etc...

While the "pattern detection" is certainly the most challenging part of the meta-language, it is by no means the only part - it also includes sections around: event definition, transformation, enrichment etc.. This work also entails various interesting topics to report in this Blog -- more later.

On the first class citizens of enterprise computing

These are the citizens of hawktown, chosen to start this Blog posting. The issue of first class citizens has been discussed recently by Paul Vincent in a recent Blog posting that also cites Forrester's Blog on the four classical elements : Process, Services, Events and information.

This is interesting, since it presents events as first class citizens in the computing world. Paul asks whether CEP is a fifth element and proposes the following diagram.

In the introduction class that I am giving to my students in the event processing course I am showing a slide that explains how events relate to the rest of the universe, interestingly in this slide as first class citizens there are all the four elements identified by Forrester -- process, service, event and information, with addition of two elements - entity and context; entity is often considered to be hidden behind information which represents it, and context has not yet gained recognition as a first class citizen, although some analysts think it will become a first class citizen.

This illustration does not provide all connections among all elements, but it is an event-centric view of the universe, let's go briefly over the different connections of events to the rest of the universe:

1. Event can be related to another event - by various types of causality, by the fact that an event is included in a complex event, and by the fact that an event is a participant of the derivation function of a derived event (further discussion is here).

2. Event can be related to context - by the fact that it starts or ends context-instance, classified to context, and is created within a certain context.

3. Event can be related to service - by the fact that service can be either a producer or a consumer of an event, a service may emit events, and an event may be sent to a service, or even invoke a service.

4. Event can be related to process - by the fact that process state may be observed as event, while an event can affect (insert/modify/cancel) a process, or affect the orchestration of processes.

5. Event can relate to entity - by the fact that event references entities within a certain role; more semantic view of event processing can be done also according to relationships among entities.

6. Event has various relations to information -- it can be represented as a (kind of) information, a database operation by itself can be an event (like in active databases) and the result of derived events can be a database update action.

For Paul's question - how does CEP fit in? I agree that CEP touches all, but would not use Venn Diagrams to show the connections here.

• I view event, process, information, entity, service and context as conceptually different terms, thus there is no intersection between any two; There are relations between each pair.
• The main function of event processing is to process event, meaning take one or more events as input, do something (transform, enrich, aggregate, find pattern, derive...) and produce one or more events as output. Interestingly event processing has relations with all the six entities (I'll use EP instead of CEP, since I am using the CEP according to its glossary definition and not according to the "marketing" definition - see discussion here)

1. EP consumes and produces events.

2. EP executes within context that determines what will be executed.

3. EP can be wrapped as a service. Also EP interacts with services.

4. EPN is a kind of process - but cannot be managed by typical BPM tools due to the different scheme of routing, it can also interact with processes.

5. Entities are not explicitly part of EP, although some type of processing may also process dependencies among entities.

6. EP has various interactions with information - same as events.

Note that sometime conceptually there are activities on the borderline between event processing and other terms, for example - the action to determine what should be done with an event on the consumer side, can be done as part of EP system, or as part of the consumer software - depends on the level of coupling between them.

More - including discussion how does the term situation fits in this game - later.

On performanc metrics and the new coffee machine

Morning, in my office with the morning coffee and the Blog... In this building the coffee machines are gradually being replaced with new ones. The new machine produces somewhat better coffee, but noticeably slower. This tie back to one of the topics that I am working on recently - performance metrics for event processing networks. From the coffee machine I can learn that people are ready to switch one property (speed) with another property (quality), which, of course, indicates that performance metrics typically does not consist of a single property. Even the dimension themselves are tricky, in previous posting I have indicated that defining latency in event processing network may have multiple interpretations, besides this we can look at minimizing the average latency, or minimizing the maximal latency. This is not identical -- "real time Java" implementation which smooth the garbage collection functions are making the maximal latency much lower, but there is a price in average latency (try and observe)... The autonomic computing principle of self-optimization applied on event processing network given multiple criteria is one of the major challenges of the next generation of event processing implementations. This is an evolving thinking, so more thoughts on: WHAT are the optimization parts and HOW they can be optimized -- in later posts.

On Top Down and Bottom Up

My B.A. degree is in Philosophy (well, to be accurate most of the studies were related to logic), later I have learned business administration and Computer Science, but I think that Philosophy was the most important thing I have learned, as the other studies provided techniques, while Philosophy provide more basic competencies.

When I have decided to study Philosophy, my late father, who has been very practical person, invested an entire evening trying to convince me that I am going to waste my time, and why I cannot study engineering, medicine or law like everybody else, well, I have listened carefully, and went to study philosophy, a decision I have never regretted - actually some of my friends switched to study philosophy when they realized that I am enjoying my studies and they are not.

I was reminded about this episode with time wasting when I caught up in what happened in Blog-land during my vacation, and discovered the blog posting entitled: fallacies of self-fullfiling CEP use case studies. I am quite amused to read these type of Blogs, I hope that the author is also feel amused when writing them... well, after being amused for a few minutes, I thought -- well, today there is a conference call of the EPTS use case workgroup, maybe I should cancel the call, since all the participants including myself are going to waste their time in addition for the time already wasted on fallacies... Fortunately, I have studied Philosophy, and moreover investigated fallacy types a long time before Wikipedia was created, and failed the fallacy in the identification of fallacies (an absence of fallacy event..), so I decided not to cancel the call.... I have a feeling that the participants did not think that they have wasted their time, but who knows...

Enough humor, time to have some serious stuff on this posting -- so I think that top-down and bottom-up approaches is a good topic to discuss. While the original posting used the metaphor of space rockets

I'll stay on the earth and use the metaphor of a gadget - let's imagine that some inventor invents a gadget maybe like this:

This gadget has a lot of features, and it was designed in a totally top-down manner.

At some point customers start to acquire this gadget and are using it in different ways for different purposes. The inventors of this gadget have a lot of ideas of what else to do in the next version of the gadget, but besides the top-down innovation there is also a bottom-up process that is called in control theory: feedback. Typically, a large enough sample of customers is being interviewed to understand - what is this used for ?, how it is used ?, are various features needed? understood ? used the same way as imagined by the inventors or otherwise? are there some requirements for this gadget to connect to other gadgets? some requirements about the operational aspects? this information can be used both to explain new customers about the experience gained, and information that can tune up the priorities and ideas of the next generation.

Back from the fascinating gadget world to our not less fascinating EP world -- the use case workgroup study is intended to understand both about the ways in which EP technologies are used today, and about additional requirements that customers who already used EP technologies are looking at (customers that did not use a certain technology typically have difficulties to express requirements about such a technology, unless they have studied the area...). .

My assumption is that the end result of this study will be beneficial for the entire community - customers who would like the best practices, vendors that design their next generation, researchers who wish to analyze this market etc... However, assumptions are just philosophy and as such, my assumptions are as good as the counter-assumption that this is all a self-fulfilling fallacy. Since I have grown up and am wearing the scientist hat these days, I suggest to take the empirical approach, namely, be patient to see the end result and judge it.

And bottom line about bottom-up and top-down: Top-Down and Bottom-up work have in general complimentary roles, and are used in different phases in the life-cycle of products/technologies/areas. Important concepts such as: use patterns and best practices are, by definition, Bottom-Up...

On faithfull representation and other comments

Back home from the vacation in Turkey, the vacation took place in the Limak Limra hotel, about 1.5 hours drive from Antalya airport (see picture of one of the many swimming pools above). It was a great British philosopher who preached to workaholists people like myself about "in praise of idleness" . So - not taking the laptop with me, I have learned several things:

1. Unlike the Israeli beach which consists of soft sand, the beach in Turkey consists of small and large stones;

2. Turkish chefs know how to cook many types of foods quite well, but have a lot to learn still in preparing Sushi,

3. The reputation of Charter flights about long delays is actually true (however, this is also true today for many regular flights).

Since Richard Veryard has sent me an Email about his Blog postings entitled "Faithfull Representation" in which he referred to an illustration that I have made as a "simple situation model" and attributed this model to both Tim Bass and myself (goodness gracious me!). Tim, who constantly claims that he has much more general view than me, could not believe that his name and my name are mentioned in the same sentence as agreeing on something, and asserted (I am using "cut and paste" from Tim's Blog:) "Opher tends to view CEP as mostly an extension of active database technology where I see CEP as a technology that is much more closely aligned with the cognitive models".

Here are some comments:

1. The illustration that Richard is quoting does not mean to explain what a situation is, but to show the relations among several concepts, I am enclosing it again -

As can be seen I am writing there that composite events (which are taken from active database terminology) and complex events (which are not) may both represent situations, which does not say that this is the only way to represent situation (as saying that fish is an animal does not define what is an animal).

2. I have explained the basic idea of situation in this posting , simply said - a situation is a concept in the "real world" domain (not in the computer domain) that requires reaction. In some cases a single event determines a situation, in some cases, detecting a pattern determines a situation, and in other cases, patterns only approximate the notion of situation, and there is no 1-1 mapping between events and situation, note that in that posting I also have provided an example of non deterministic situations.

3. Regardless of the situation definition, Richard is absolutely right that all over the event processing life-cycle we may have instances in which the events are inaccurate or uncertain , and the reader is referred to this posting for some examples of uncertainty issues we are dealing with. This is an area that I am investigating in the last few years together withAvi Gal from the Technion and Segev Wasserkrug (our joint Ph.D. student who graduated recenlty with a Ph.D. dissertation was denoted as excellent by the exam committee). Hot from the oven - A paper about it is published in the recent (August 2008) issue of IEEE Transactions on Knowledge and Data Engineering, which is dedicated to "SPECIAL SECTION on Intelligence and Security Informatics". The actual paper can be downloaded from Avi Gal's website. Another paper related to the same study has been presented in DEBS 2008.

4. While I totally agree that in some cases the uncertainty is needed - and certainly some security applications are example, I also believe that the potential market for the more basic deterministic world is much higher, and we are far from picking up all the low hanging fruits of the deterministic event processing.

5. We still have challenges in defining the semantics of the different cases of handling uncertain events/patterns/situations. The fact that there are arithmetic of uncertainty help, but not everything that exists in AI research fits the real world requirements of scalability, performance etc..

6. About the comment of me viewing event processing as extension of active database technology -- I view event processing as a discipline by its own right (and this is a topic for another discussion which I'll defer), it has origins in several disciplines, one of them is active databases, but it has several more ancestors - sensor fusion, discrete event simulation, distributed computing/messaging/pub-sub and some more, and draws concepts from each of them. Anybody who reads my Blog can realize that there is a fundamental difference between active database that extends database engines and event processing that is not based on database technology, there are some other differences too.

7. My friendly advice to Tim is that before he makes assertion about how and what people think (and this does not refer necessarily to myself) he will re-read his own excellent posting :"red herring fallacies" .

More on event processing as a discipline - at a later post.