On the multiple types of patterns in event processing

The red arrow in the map shows where I am now - in Kaufbeuren, a town in the south-west corner on Bavaria. I have arrived last night by train, and it has been a challenge to find a place to eat in 9PM, but with a few minutes of search I have found it. I am on the way to the patterns conference, in which we have a working group to establish a partnership towards EU project proposal. Thus, I would like to write a few lines about patterns. In my DEBS 2008 tutorial I have used the following slide:

In this slide there are four possible meanings for patterns in event processing:

(1). Patterns in the sense of functions that event processing application may perform (e.g. enrich, route, transform, filter, detect pattern).

(2). Patterns that are detected on the history of events (note that "detect pattern" is just one of the patterns of the first type, but the support of this pattern is what makes it CEP).

(3). Patterns in the user's domain - the way things are presented to the user (which may not be translated 1-1 to an implementation pattern)

(4). Patterns in the software engineering sense - best practices of how to use some language / product / technology.

All of the four are interesting and also have some dependencies, my tutorial has concentrated mainly on patterns of type (2), I have written about type (1) a lot in the past -- types (3) and (4) are frontiers that need to be conquered. More - Later.

On formal models and agents

This is an aerial picture of Karlsruhe, in Germany. It is morning, I have the habbit to get up early, so have some time for Email and Blogging (this hotel does have internet connection in the room!). Here I am a guest of the FZI team that is trying to apply semantic models to event processing, which is one of my recent occupations. One of the property of researchers everywhere (and IBM is not different) that they are constantly looking for research funding - since one of the available sources in this region is EU projects, researchers from the relevant countries (and Israel is part of the EU, for the sake of the R&D programs), so there is a need to join forces. The FZI group is looking at applying transaction logic as a formal model for both validation and run-time control of event processing applications. I am in favor of having formal model behind event processing to define precisely the semantics. A model that will succeed should also be relatively simple. I was asked whether I see a possibility to have a single formal model that inclues all variations of event processsing - and my answer is that my assumption is that this is possible.

Dealing with formal models - I have been asked by several people about the choice I've made in my DEBS 20008 tutorial to define "event processing agent" as an atomic unit of processing.

First - I agree that the term agent is overloaded, like many other terms -- event processing agent is not (necessarily) an intellignet agent, it is more a software agents. The glossary defines agent as a software module that processes events. I have making a distinction between conceptual agent and run-tine agent - the run-time agent may consist of multiple conceptual agents, and the decision is up to self-optimization process (will write about optimization issues in one of the next blogs). It is easier to reason (and optimize) when the abstraction deals with atomic elements. Whether should be called - "atomic agent" - may be. I prefer shorter names, and since this does not contradict the original definition. more - later.

On events and Data

I am back at home for 2.5 days, in Monday morning I'll continue travelling the world - this time in Germany mostly for the DoReMoPat consortium meeting, which is related to the tutorial I've given in DEBS on event processing patterns. Today I would like to refer to one of the topics discussed in DEBS, as a result of the tutorial given by my friends Shailendra and Dieter from Oracle - the questions is what are the boundaries (if any) between database processing and event processing. I have already posted in the past a blog entry about the topic: is event processing a footnote for databases? and will not repeat what was written there, to illustrate the difference between event and data, I'll take the state machine example -- data is like states and events are like transition. Thus, a database reflects a snapshot that captures a single state in the universe. In event processing we process not the state, but the history of transitions, thus, the type of processing is slightly different (although as we noticed from the Oracle's presentation, there is now an attempt to take the pattern notion from EP and get it into SQL extension, IMHO - the result is somewhat complicated, since is not a natural hybrid), Anyway, one of the controversial point has been - whether sensor reading is an event or a data -- according to the state/transition test - a sensor reading is an event -- it transitions the database to a new state that includes the new transition. Some clever guy asked me the question - what happens if the sensor reading is the same as the previous one, nothing is change in the universe, so it is not a transition. Well - good question, but IMHO this is still a transition, first transitions may return the state machine to the same state (as seen in the state machine example above - the top state has a transition that maps back to itself). Why is it a transition ? -- first the timestamp is different, it is a new piece of information that exists (the combination of reading and timestamp) and has the potential also to change the state of the universe by mapping to another state. More - Later.

On DEBS 2008

Still in Rome, still in the hotel's bar since my room does not get enough wireless signal. The DEBS conference site is just next door to the church in which the famous Moses status is located.

I have also finished tonight my participation in DEBS (it goes one more day, but I will be heading for home tomorrow according to the plan). The DEBS conference (disregarding some organizational challenges) have gone well, while it has been a scientific conference, there has been a relatively big participation from industry - I have seen there people from vednors as well as customers, DEBS has been originally the home of the pub/sub and messaging community, however they have successfully extended it to include various other event processing "sub-communities" like the CEP guys, the streaming guys, people from modeling area etc..

I have contributed my part by providing a tutorial about EP patterns - which is a work in progress for building a meta-language which abstracts the semantics behind event processing functionality.

Since it is late tonight I'll defer discussions that I had in the conference to later posts, will just make a note that I owe everybody discussion on three points:

• Boundaries and territories among disciplines -- in the wake of discussion with Dieter Gawlick from Oracle.
• The term "event processing agents" - some discussions, and Blog comments from Paul Vincent
• The notion of pattern in event processing.

Overall - very good conference, an opportunity to meet a lot of friends, and (I am ashamed to admit) my first visit in Rome ever.

On the EPTS working group use cases

Hello from hot and humid Rome, Italy. I have just finished eating a Pizza dinner just across the street from the Coliseum, seen in this picture, lot of ancient stuff around. I am now writing this Blog in the hotel lobby - just 5 minutes walk from there, why am I writing from the hotel lobby? One of my criteria to chose hotels is that they advertise that they have Internet access in their rooms, and this hotel also claim to have Internet access - nominally they have , it is just that the signal in the rooms is too weak, and in order to work one has to come to the lobby - which is an information that the receptionist gladly shares with you after you check in... Anyway, they have here a comfortable desk to work and air-condition, which is more than we had today in the university, where we had been meeting for the EPTS working group. So - to start from the beginning, I have arrived to Rome last night for the DEBS conference. If you look at the DEBS site you can discover that the air-condition in the university has been broken, and they move the conference to anther site -- well, but just from Wednesday, and tomorrow we'll have to suffer another day in the non air-conditioned building. This has been the first time in history that I have closed my laptop, since it got so hot that I feared it will not survive. Tomorrow I'll give a tutorial in front of a sweating audience just after lunch, it will be fun. Today, we had a F2F meeting of the use cases working group - interesting mix of four people from different vendors (Dieter Gawlick from Oracle, Richard Tibbetts from Streambase, Brian Connell from WestGlobal and myself), one academic person (Pedro Bizzaro) and one customer (Alex Kozlenkov from BetFair), there are some other members that could not participate, the working group is larger that the six of us. Despite the heat it has been long and productive day -- what are we trying to achieve? We are trying to learn something from the variety of use cases out there, we have heard until now in the three EPTS meetings and Dagsthul seminar, something like 80 use case. However, we cannot learn much of them, since they have been presented in an unstructured way, we could not analyze them on the same scale - the purpose is to try and classify event processing applications to several classes - and observe what are the important requirements for each class - in terms of functional, non-functional requirements, ROI measurements etc..., this is done to enable existing and potential customers to better understand the types of event processing (complex event processing or other event processing types) - today we have worked on the template - what are the criteria to analyze each individual use case, and got to a draft there that will be finalized in the next few weeks. We shall present the template together with some use cases on this template in the next EPTS F2F meeting in September - so stay tuned. Tomorrow -- DEBS. While DEBS is an academic conference, I am glad to see here many of my industrial colleagues (both vendors and some customers), the establishment of a research discipline around event processing, and help growing the academic research in this area, has been one of the EPTS basic missions, and we are supporting DEBS as the research academic flagship. Will write my impressions from the conference later this week (if my laptop will survive the heat)...

On cost-effective EP when high performance is not required

Brian Connell has asked me to provide an example in which Complex Event Processing can be used when low latency is not required and it is still be better than other solutions.

I have already referred to this issue in the past. There are several concrete examples that come to mind - here is one (real system):

An enterprise has internal regulations to handle suppliers, these requirements relate to the interaction with suppliers, that may be represented as events, and involve actions that must be done, or should not be done, within certain amount of time, or until some event happens. The application is to monitor the compliance with these internal regulation in audit mode, meaning the results are alerts (which are also treated as events, since there are time-outs from alert sending) and not direct interference in the business processes.

The throughput is far below what is considered as "high throughput" and it is several thousands events per day. The latency is also not required to be extremely low -- if the alert will be issued within a minute or two, it is still very fast auditing. It also does not require any analytics of intelligent procedures, since the regulations are given and deterministic

What are the benefits for the customer to use EP software and not any other solution?

First - some of the regulations are fairly complex - writing them in hard-code can be time and resource consuming, putting everything on databases and using SQL was also considered - but some of the regulations are not easy to express in SQL.

Second - Regulations tend to change frequently, the users wish to control these changes, and getting them through the slow IT development cycle will delay the introduction of the change.

So in this case the customer's motivation are - agility and de-complexity; more later

On embedded intelligence within event processing application

In the previous post I have referred to the term "Intelligent Event Processing" - one question that I have asked - is this a new term ? how does it related to the other "X event processing" terms ? -- I am not sure if the term "intelligent event processing" will stick around, I would say that a better way to explain what it is may be - "embedded intelligence in event processing".

If we look at event processing architectures

- there are: producers who produce events, consumers who consume the processing results, and the EPN (Event processing Network) in the middle, which really does the processing. So where are intelligent techniques can help - here are some (real) examples:

1. In the producer - the producer has a video stream of all cars that pass below the camera, an intelligent process (using image processing techniques) isolates the license plate number of the car, and send it for further processing (security, traffic violation, billing etc..).

2. In the "meta-data" composition -- a "pattern detection" node typically looks at pre-defined patterns and attempts to detect them in run-time. In current applications the patterns are entered by the developers or users. In some cases the patterns are "moving target" like in - fraud detection -- if the patterns for fraud are discovered they are of little value, and thus in the other side of the law - people are constantly looking for new loopholes, thus, intelligent techniques, such as machine learning are used to refresh the patterns that are looked for in run-time. The run-time does not change - same pattern-detection mechanism, just different sources of where these patterns come from.

3. Intelligent nodes within the EPN -- in some cases the process of derivation of new events cannot be expressed as derivation expression and need some intelligent derivation process - e.g. a heuristic algorithm to determine the traffic light policies based on traffic events.


There are many more examples - like creating predicted events and more -- but this was more to give some flavor. Is it useful -- yes, it is useful for a variety of applications. Does every CEP application need embedded intelligence -- not really. More - later.