Dagstuhl seminar on event processing - the fourth day

Packing and about to leave the spartan room in the Dagstuhl Schloss towards the last session. Today at noon the Dagstuhl seminar will be over, and the easier part of the mission will be complete, now we'll have to finish the document and devise the follow-up action items and mechanism to track them. I'll summarize the seminar later-- yesterday there were deep dives on standards and on the issue of relations between event processing to other areas and disciplines. We also had evening session with Alex Buchmann as moderator, about the question -- whether "event processing" has become a research community, is the flagship conference of the community DEBS succeeding, and how we would like to evolve it? should we do ACM SIG and when? should we have a publication like Sigmod Record and when? should we have an academic journal and when? it seems that we are on track, still need to recruit more people whose participation can gain DEBS a status of top conference that don't send paper to DEBS since they are sending just to top conferences, so it is a chicken and egg issue, and we have a challenge to reach out to adjacent communities. We'll take it into account when designing DEBS 2011.

More -later.

Dagstuhl seminar on event processing - the third day

The third day in a Dagstuhl seminar traditionally has half day of trip outside the castle, this time we have traveled to a place called Metllach seen in the picture, we have sailed in a boat on the Saar river, and saw this island, and also went to a place from which we could view this island from the hill above, we also traveled to a winery and tasted six kinds of wine and heard long explanations (in German) about these wines.

In the morning we had one more breakout session, and a deep dive into topic 2: what are the functions of event processing (including non-functional function), though for some there was difference of opinions whether it is functional or non-functional (e.g. provenance).

There are discussions about the boundaries of event processing: are "actions" internal or external to event processing: they seem to be external, but for provenance and retraction, the event processing system should be aware of them. The team also identified a collection of topics that require further research, here is the list:

–

• Use of EP to predict (anticipate) problems
• Use of predictions (e.g. from simulations) in EP
• Complex actions
• Action processing as the converse of event processing
• Decomposition of complex actions with time constraints
• Goal directed reaction
• Adaptive planning
• Implicit validation
• Function placement and optimization
• Real-time machine generated specification
• Compensation and Retraction
• Privacy and Security
• Probabilistic events
• Provenance

I'll write more about future research topics. Today we finish the deep dives and starting to wrap-up, determine the structure and schedule of the final document, and move to discuss the most important stuff -- what we want to achieve, what are the follow-ups, and what will be the follow-up actions?

There is another Dagstuhl tradition - to take a group picture, always in the same place, on the stairs of the castle's old church:

Dagstuhl seminar on event processing - the second day

This is the Dagatuhl logo, we are about to start the third day -- a shorter day in terms of work, since we are doing excursion for half a day today. In the second day: two sessions of breakouts in groups, I am visiting each session a different group, and two "deep dive" session -- the first one on "why" event processing, in which they have tried to come up with cost/benefit model, and the second is the event processing grand challenge, in which they presented "smart society" as a main theme, with variety of "smart" application, but the main message is that while each application in the list may be a challenge, the grand challenge is the holistic view, namely, the interdependencies among all these systems.

In the evening, over some wine, beer and cheese, there was a more informal session (no slides) in which different vendor representatives told us some lessons:

Marco Sierio from ruleCore started in eight lessons he devised in the train on his way, one of them is that people have hard time to adjust to the "event driven" thinking, since they are programmed to think in "request/response", Marco will probably put all eight lessons on his Blog, but one of the interesting lessons is that reading research papers is a time well spent, although it is not an easy task to do.

Richard Tibbetts from Streambase also provided some insights from his experience, ending with a statement that he did not see a lot of demand for pattern detection with his customers. Maybe in trading applications patterns detection is not a natural thing to do ?

Badrish Chandramouli from Microsoft provided some insights about StreamInsights. The interesting feature is the temporal model, which allow events to be defined as point event, interval event, and interval events with fuzzy boundaries. They also allow speculative computation and out-of-order processing, at least to some extent.

Alex Kozlenkov from Betfair, is both an user, and a developer of Prova, a self-made event processing platform, that follows the EPN/EPA model that we describe in the EPIA book (well - some variation of it) and explained its features in his usual enthusiastic way.

Martin Hirzel from IBM System S, talked about some of his impressions in working on that team, Martin is a programming languages person, working on the SPL (Stream Processing Language) and provided some insights about building a language by multi-disciplinary team.

Udo Pletat also from IBM, but from the software services organization, provided some insights on applying RFID oriented applications with customers (this is based on the Websphere Sensors Events product of IBM which has embedded Amit), and provided some examples of why they needed to use event processing patterns to track people's movement in chemical plants with restricted zones.

The session ended after 11pm and was very interesting gathering. Today -- the deep dive about "what is event processing" - more later.

Dagstuhl seminar on event processing - the first day

It is evening here in Schloss Dagstuhl, the castle that you can see in the picture above. My room is on the ground floor of the part of the building in the left-hand side of the picture. This seminar organized by Rainer von Ammon, Mani Chandy and myself, is a five days event, and we just finished the first one, during the day I lost my voice for about an hour, and regained it by eating ice-cream. In this seminar we are working in five sessions, you can see this presentation includes the outline of what we are doing in the opening session and the one slide that every participant (well - almost, a few have not sent the slides) put about himself/herself.

Besides the opening session we have done the first meeting of each of the five groups, and had initial discussion about them. Tomorrow -- two breakout sessions and two deep dives. One of the things we started to discuss and we'll finish this discussion later this week is ---- looking in the future five years from now - how can we measure the success of this seminar to change the world --- more about it later.

SAP acquired Sybase: the event processing angle

It seems that Paul Vincent will have to update his product genealogy again. There are various analysts like James Kobielus or Philip Howard have expressed their opinion about the motivation and possible implications of this merge, and I leave this discussion to them; I'll write on the narrow perspective of the event processing angle (to justify the name of this Blog).

Taking the EP perspective, there is a series of big fish swallows smaller fish, until it gets to the ultimate big fish. Event processing is shifting from being dominant by start-ups to be dominant by bigger companies, which happened before in other areas, and is sign of getting this technology to the main stream of computing, this is consistent with some of the trends we identified as the current trends in the conclusion chapter of the EPIA book, event processing is going from narrow domains to wider domains and from being stand-alone technology to being part of bigger frameworks. As four big fish -- now the four big software companies - IBM, Microsoft, Oracle and SAP are all in the EP market, and all of them view EP as part of a bigger game; I guess that there will always be smaller stand-alone EP companies for niche markets, but most applications do not live in an isolation and have some relations to other areas - BPM, BRMS, BI and some more.

The fact that most players in this market are big and medium (TIBCO, Progress Software, Software AG) companies, can make the climate more comfortable to advance standards now -- which is one of the topics we plan to discuss in Dagstuhl next week.

When we established EPTS, SAP was not interested to join, since they did not have a product at that time, we'll see whether they will want to inherit Sybase's membership (which by itself was recently inherited from Aleri)... We'll stay tuned for more on this front...

Some footnotes to Brenda Michelson's post on TUCON

Las Vegas is a place where companies like to do customer conferences; I guess that the quantity and sizes of hotels is a major factor. Last week IBM has done the big customer conference: IMPACT 2010, after a few years of attending this conference, I have skipped this one, so I did not cover it in the Blog. This week TIBCO has done its customer conference - TUCON 2010, naturally I have not been there either, as I am not a TIBCO user, but looked at some impressions on the Blogland, and came across Brenda Michelson's Blog describing the keynote sessions. With the disclaimer that my information on this is second hand (but a good hand !), I'll make some footnotes:

1. I believe in the direction that events will be more and more in the center of enterprise (and other applications); we see it happening and we'll see much more.

2. Like Brenda, I am not sure that the phrase "enterprise 3.0" will catch, at the past there was an attempt to push "SOA 2.0" but it did not catch either, I think the world became tired from recycling the n.0 phrase.

3. "2 second advantage" is catchy, and may be true for some applications, but in some of them 2 second might not be enough to make a difference. In general there are two aspects: immediate reaction -- react fast (or faster than others); and proactive --- react in time to mitigate or eliminate future event; I am not sure it that if the Interpol will know about a crime 2 seconds earlier it will be enough time to eliminate the crime.

4. About NoSQL and relations to event processing - I'll write a separate post, it is an interesting insight that NoSQL is gaining traction.

5. Context as additional dimension -- I have written several times in the past about contexts, and am planned to deliver a tutorial about contexts in DEBS 2010; I believe that contexts are starting to be a major factor in computing.

It is good to see that the event-centric universe is getting more traction. More -later.

On inter-system event causalities

A lot has been spoken and written about the Wall Street Plunge, David Luckham raised the question about the role of event processing systems in prevention and/or postmortem analysis of understanding how such a situation happened. I am not sure exactly how much event processing system have been part of the reason of causing this plunge in the first place, but here we have an interesting challenge, the challenge is that we need to understand causality in a real event cloud, where the events are part of multiple systems. The nature of the challenge is twofold:

First: Each of the automated trading systems logic is, of course, not transparent, since they are used as competitive edge by the traders, thus it is difficult to know how various systems would function, and there will be a need to learn their logic (which can be constantly changed) based on their actual behavior;

Second: The logic of the various systems is expressed in various ways, and there is no standard causality model that can be used across systems.

This can be one of the challenges that we can put into the challenges of the next generation of event processing systems, that we plan to discuss in Dagstuhl.