On EuroPLoP and Event Processing Patterns

This is Kloster Irsee in Germany, where the EuroPLoP conference is held. The conference itself is held in kind of an opposite way to other conferences, in a regular conference, the author of a paper presents, in this conference, the author of a paper hears other criticizing his paper, and does not talk. It also have some other properties -- some of the sessions are held in room which the chairs have been taken out, so people are sitting on the floor, or standing, there are also games embedded in the program --- my daughter would have said if she was here that there is a bunch of nerd children trying to behave in a cool fashion - well... anyway, we came here to combine work on creating a consortium for EU project and having a three hours "focused group" inside the conference, organized and moderated by Adrian Paschke. The audience was mixed - some were the EP gang, those who came to work on the EU project, and some where regular participants in the conference. The various interpretations of patterns related to EP have been discussed, and the need to advance in all of them. This is directly related to my tutorial in DEBS 2008, there is now work in progress on a pattern meta-language that we are doing in IBM, in a few weeks I'll tell you more about it - stay tuned.
More about my tutorial in DEBS 2008 - Paul Vincent from TIBCO has taken a picture of a slide that I've presented with different computer (due to some unclear technical problem) with probably different character set (Paul also blogged about this slide) - in footnote 1. Paul has been kind enough to send me the picture - so here it is, you may see some unknown characters...

More about patterns - soon.

On the multiple types of patterns in event processing

The red arrow in the map shows where I am now - in Kaufbeuren, a town in the south-west corner on Bavaria. I have arrived last night by train, and it has been a challenge to find a place to eat in 9PM, but with a few minutes of search I have found it. I am on the way to the patterns conference, in which we have a working group to establish a partnership towards EU project proposal. Thus, I would like to write a few lines about patterns. In my DEBS 2008 tutorial I have used the following slide:

In this slide there are four possible meanings for patterns in event processing:

(1). Patterns in the sense of functions that event processing application may perform (e.g. enrich, route, transform, filter, detect pattern).

(2). Patterns that are detected on the history of events (note that "detect pattern" is just one of the patterns of the first type, but the support of this pattern is what makes it CEP).

(3). Patterns in the user's domain - the way things are presented to the user (which may not be translated 1-1 to an implementation pattern)

(4). Patterns in the software engineering sense - best practices of how to use some language / product / technology.

All of the four are interesting and also have some dependencies, my tutorial has concentrated mainly on patterns of type (2), I have written about type (1) a lot in the past -- types (3) and (4) are frontiers that need to be conquered. More - Later.

On formal models and agents

This is an aerial picture of Karlsruhe, in Germany. It is morning, I have the habbit to get up early, so have some time for Email and Blogging (this hotel does have internet connection in the room!). Here I am a guest of the FZI team that is trying to apply semantic models to event processing, which is one of my recent occupations. One of the property of researchers everywhere (and IBM is not different) that they are constantly looking for research funding - since one of the available sources in this region is EU projects, researchers from the relevant countries (and Israel is part of the EU, for the sake of the R&D programs), so there is a need to join forces. The FZI group is looking at applying transaction logic as a formal model for both validation and run-time control of event processing applications. I am in favor of having formal model behind event processing to define precisely the semantics. A model that will succeed should also be relatively simple. I was asked whether I see a possibility to have a single formal model that inclues all variations of event processsing - and my answer is that my assumption is that this is possible.

Dealing with formal models - I have been asked by several people about the choice I've made in my DEBS 20008 tutorial to define "event processing agent" as an atomic unit of processing.

First - I agree that the term agent is overloaded, like many other terms -- event processing agent is not (necessarily) an intellignet agent, it is more a software agents. The glossary defines agent as a software module that processes events. I have making a distinction between conceptual agent and run-tine agent - the run-time agent may consist of multiple conceptual agents, and the decision is up to self-optimization process (will write about optimization issues in one of the next blogs). It is easier to reason (and optimize) when the abstraction deals with atomic elements. Whether should be called - "atomic agent" - may be. I prefer shorter names, and since this does not contradict the original definition. more - later.

On events and Data

I am back at home for 2.5 days, in Monday morning I'll continue travelling the world - this time in Germany mostly for the DoReMoPat consortium meeting, which is related to the tutorial I've given in DEBS on event processing patterns. Today I would like to refer to one of the topics discussed in DEBS, as a result of the tutorial given by my friends Shailendra and Dieter from Oracle - the questions is what are the boundaries (if any) between database processing and event processing. I have already posted in the past a blog entry about the topic: is event processing a footnote for databases? and will not repeat what was written there, to illustrate the difference between event and data, I'll take the state machine example -- data is like states and events are like transition. Thus, a database reflects a snapshot that captures a single state in the universe. In event processing we process not the state, but the history of transitions, thus, the type of processing is slightly different (although as we noticed from the Oracle's presentation, there is now an attempt to take the pattern notion from EP and get it into SQL extension, IMHO - the result is somewhat complicated, since is not a natural hybrid), Anyway, one of the controversial point has been - whether sensor reading is an event or a data -- according to the state/transition test - a sensor reading is an event -- it transitions the database to a new state that includes the new transition. Some clever guy asked me the question - what happens if the sensor reading is the same as the previous one, nothing is change in the universe, so it is not a transition. Well - good question, but IMHO this is still a transition, first transitions may return the state machine to the same state (as seen in the state machine example above - the top state has a transition that maps back to itself). Why is it a transition ? -- first the timestamp is different, it is a new piece of information that exists (the combination of reading and timestamp) and has the potential also to change the state of the universe by mapping to another state. More - Later.

On DEBS 2008

Still in Rome, still in the hotel's bar since my room does not get enough wireless signal. The DEBS conference site is just next door to the church in which the famous Moses status is located.

I have also finished tonight my participation in DEBS (it goes one more day, but I will be heading for home tomorrow according to the plan). The DEBS conference (disregarding some organizational challenges) have gone well, while it has been a scientific conference, there has been a relatively big participation from industry - I have seen there people from vednors as well as customers, DEBS has been originally the home of the pub/sub and messaging community, however they have successfully extended it to include various other event processing "sub-communities" like the CEP guys, the streaming guys, people from modeling area etc..

I have contributed my part by providing a tutorial about EP patterns - which is a work in progress for building a meta-language which abstracts the semantics behind event processing functionality.

Since it is late tonight I'll defer discussions that I had in the conference to later posts, will just make a note that I owe everybody discussion on three points:

• Boundaries and territories among disciplines -- in the wake of discussion with Dieter Gawlick from Oracle.
• The term "event processing agents" - some discussions, and Blog comments from Paul Vincent
• The notion of pattern in event processing.

Overall - very good conference, an opportunity to meet a lot of friends, and (I am ashamed to admit) my first visit in Rome ever.

On the EPTS working group use cases

Hello from hot and humid Rome, Italy. I have just finished eating a Pizza dinner just across the street from the Coliseum, seen in this picture, lot of ancient stuff around. I am now writing this Blog in the hotel lobby - just 5 minutes walk from there, why am I writing from the hotel lobby? One of my criteria to chose hotels is that they advertise that they have Internet access in their rooms, and this hotel also claim to have Internet access - nominally they have , it is just that the signal in the rooms is too weak, and in order to work one has to come to the lobby - which is an information that the receptionist gladly shares with you after you check in... Anyway, they have here a comfortable desk to work and air-condition, which is more than we had today in the university, where we had been meeting for the EPTS working group. So - to start from the beginning, I have arrived to Rome last night for the DEBS conference. If you look at the DEBS site you can discover that the air-condition in the university has been broken, and they move the conference to anther site -- well, but just from Wednesday, and tomorrow we'll have to suffer another day in the non air-conditioned building. This has been the first time in history that I have closed my laptop, since it got so hot that I feared it will not survive. Tomorrow I'll give a tutorial in front of a sweating audience just after lunch, it will be fun. Today, we had a F2F meeting of the use cases working group - interesting mix of four people from different vendors (Dieter Gawlick from Oracle, Richard Tibbetts from Streambase, Brian Connell from WestGlobal and myself), one academic person (Pedro Bizzaro) and one customer (Alex Kozlenkov from BetFair), there are some other members that could not participate, the working group is larger that the six of us. Despite the heat it has been long and productive day -- what are we trying to achieve? We are trying to learn something from the variety of use cases out there, we have heard until now in the three EPTS meetings and Dagsthul seminar, something like 80 use case. However, we cannot learn much of them, since they have been presented in an unstructured way, we could not analyze them on the same scale - the purpose is to try and classify event processing applications to several classes - and observe what are the important requirements for each class - in terms of functional, non-functional requirements, ROI measurements etc..., this is done to enable existing and potential customers to better understand the types of event processing (complex event processing or other event processing types) - today we have worked on the template - what are the criteria to analyze each individual use case, and got to a draft there that will be finalized in the next few weeks. We shall present the template together with some use cases on this template in the next EPTS F2F meeting in September - so stay tuned. Tomorrow -- DEBS. While DEBS is an academic conference, I am glad to see here many of my industrial colleagues (both vendors and some customers), the establishment of a research discipline around event processing, and help growing the academic research in this area, has been one of the EPTS basic missions, and we are supporting DEBS as the research academic flagship. Will write my impressions from the conference later this week (if my laptop will survive the heat)...

On cost-effective EP when high performance is not required

Brian Connell has asked me to provide an example in which Complex Event Processing can be used when low latency is not required and it is still be better than other solutions.

I have already referred to this issue in the past. There are several concrete examples that come to mind - here is one (real system):

An enterprise has internal regulations to handle suppliers, these requirements relate to the interaction with suppliers, that may be represented as events, and involve actions that must be done, or should not be done, within certain amount of time, or until some event happens. The application is to monitor the compliance with these internal regulation in audit mode, meaning the results are alerts (which are also treated as events, since there are time-outs from alert sending) and not direct interference in the business processes.

The throughput is far below what is considered as "high throughput" and it is several thousands events per day. The latency is also not required to be extremely low -- if the alert will be issued within a minute or two, it is still very fast auditing. It also does not require any analytics of intelligent procedures, since the regulations are given and deterministic

What are the benefits for the customer to use EP software and not any other solution?

First - some of the regulations are fairly complex - writing them in hard-code can be time and resource consuming, putting everything on databases and using SQL was also considered - but some of the regulations are not easy to express in SQL.

Second - Regulations tend to change frequently, the users wish to control these changes, and getting them through the slow IT development cycle will delay the introduction of the change.

So in this case the customer's motivation are - agility and de-complexity; more later