DEBS 2010 program has been published

The program of DEBS 2010 is now published on the conference's website. Note that IBM has become a research power, based on its representation in DEBS.

In total IBMers have authored or co-authored 10 papers out of the 21 papers in the research and industry tracks, and 2 out of the 6 tutorials.

Some details:

Tutorials:

Ella Rabinovich and myself will deliver a tutorial prepared by several members of our team, on: context Aware Computing and its Utilization in Event-Based System

This is the third year in a raw that I am giving (alone or with others) a tutorial in DEBS.

Guy Sharon, another member of our team, participated in the preparation of a tutorial on Measuring Business Value of CEP-Applications.

Papers:

I am a co-author of the paper: Analyzing the Behavior of Event Processing Applications.

Another paper by members of our team is: Industry Experience with the IBM Active Middleware Technology (AMiT) Complex Event Processing Engine (I have not participated in writing this one), both of these papers are planned to be presented by Ella Rabinovich.

Three additional papers are co-authored by other persons in IBM Haifa Research Lab:

Magnet: Practical Subscription Clustering for Internet-Scale Publish/Subscribe

Quilt: A Patchwork of Multicast Regions

On Trade-offs in Event Delivery Systems

Four papers are authored by IBM Watson Research center's guys:

Evaluation of Streaming Aggregation on Parallel Hardware Architectures

Placement of Replicated Tasks for Distributed Stream Processing Systems

Experiences with Codifying Event Processing Function Patterns

Workload Characterization for Operator-Based Distributed Stream Processing Applications

And another paper co-authored by Udo Pletat from IBM Boeblingen

Distributed Heterogeneous Event Processing

This is an indication for the importance that IBM sees for investing in research and thought leadership in the areas of event processing, stream analytics, and pub/sub. More about DEBS -- in July; meanwhile another business trip to Europe is waiting for me in Sunday.

The event processing grand challenge - take one

In the Dagstuhl seminar on event processing that ended 10 days ago we launched the "event processing grand challenge activity". You may be familiar with the DARPA grand challenge of driving driver-less car through the desert, you can see the two winner cars in the pictures above. We are not DARPA, and don't have money to distribute, but we would like to take the opposite direction, first define the grand challenge, and then convince funding authorities that they want to support it.

Why do we need this grand challenge? The research community has incubated the "event processing" area as we know today --- some research projects in the 1990-ies, such as: David Luckham's Rapide in Stanford, Mani Chandy's Infospheres in Caltech, John Bates' Apama in Cambridge, and our own Amit project in IBM Haifa Research Lab, followed by the various stream processing projects, like Jennifer Widom's stream project in Stanford, and later the Aurora project, to name a few (there are many more, of course).

The state of the practice is now in the hands of the software vendors, and event processing is becoming part of the main-stream of enterprise computing. However, software vendors are by nature advancing technology in an incremental fashion. On the other hand, there is a strong feeling that "event processing" has barely scratched the surface of its potential to impact society, this goes beyond current applications, and even beyond enterprise computing as we know it.

It is now the role of the research community to jump-start and incubate the step-function required to be achieved in order to get this kind of impact. We would like to have a call to the research community in event processing to focus on such a grand challenge, and as an incentive (and enabler) to get funding agencies worldwide to adopt it.

A substantial brain power have been invested in Dagstuhl, and in a follow-up to work on it, there have also been some people outside the community that deal with socio-technical systems and bio-informatics.

As a metaphor we can view many type of IT systems, social systems and biological system as a "live ecology", similar to a single organism with a lot of brains, eyes, ears, hands and feet.
Event processing serves as the nervous system, and events are what flow between the different players. This subsumes the "Internet of things" vision, where many sensors are connected, and also robots of various kinds as actuators, serving as hands. Such an infrastructure will enable changing the life as we know them.

We are working now on various scenarios that will be enabled by such an infrastructure; you can hear some thoughts about it in my talk next week in the OMG Event Processing community of practice that holds its first event processing symposium. Stay tuned for much more on that topic.

On event processing standards

One of the Dagstuhl follow-up will be to have action items in advancing standards in the event processing world. One of the topics discussed in Dagstuhl was standards, The team working on it was moderated by Paul Vincent, who already blogged about it. While I'll write more about it when the final report will be ready, here are some initial thoughts:

1. It seems that in the era where the vendor community is now dominated by bigger companies, the atmosphere for standards become more friendly.
2. For other communities - standards have been critical success factor, e.g. web services.
3. Somebody mentioned the immortal Stonbraker's phrase about SQL being "intergalactic data speak", we need the "intergalactic event speak" - and it is not an extension to SQL.
4. There are different standardization issues -- event representation, meta-modelling, event processing language; as well as extensions to many existing standards possible.
5. The language standardization will be trickiest - due to the variety of languages styles exist, here I think that we'll better start with a language at the PIM (platform independent model) level. In the EPIA book we provided one that can serve as a starting point.

I believe that standards is one of the important ways to go forward, and will write more about it when we'll have the Dagstuhl final report.

Dagstuhl seminar on event processing - the conclusion

The Dagstuhl seminar on event processing has ended in Friday, and I have arrived home Saturday early morning. This has been my fourth visit in Schloss Dagstuhl, and the magic that this place projects keeps flowing. Organizing such a seminar is a hard work, and this time we have taken ambitious task, the seminar goals were:

a) Brainstorm and devise an actionable plan for evolving event processing to be a critical technology in a grand challenge that will have major impact on society, as a wide-community effort;

b) Brainstorm and devise and actionable plan for creating a community-agreed document about the value, boundaries, functions, and synergies with other areas and communities.

c) Brainstorm and devise and actionable plan for the evolution of specific event processing standards, and employ/extend existing standards.

There are a lot of follow-ups, conclusion of the Dagstuhl seminar can be found on the seminar's webpage.

More about the document and grand challenge - later.

Dagstuhl seminar on event processing - the fourth day

Packing and about to leave the spartan room in the Dagstuhl Schloss towards the last session. Today at noon the Dagstuhl seminar will be over, and the easier part of the mission will be complete, now we'll have to finish the document and devise the follow-up action items and mechanism to track them. I'll summarize the seminar later-- yesterday there were deep dives on standards and on the issue of relations between event processing to other areas and disciplines. We also had evening session with Alex Buchmann as moderator, about the question -- whether "event processing" has become a research community, is the flagship conference of the community DEBS succeeding, and how we would like to evolve it? should we do ACM SIG and when? should we have a publication like Sigmod Record and when? should we have an academic journal and when? it seems that we are on track, still need to recruit more people whose participation can gain DEBS a status of top conference that don't send paper to DEBS since they are sending just to top conferences, so it is a chicken and egg issue, and we have a challenge to reach out to adjacent communities. We'll take it into account when designing DEBS 2011.

More -later.

Dagstuhl seminar on event processing - the third day

The third day in a Dagstuhl seminar traditionally has half day of trip outside the castle, this time we have traveled to a place called Metllach seen in the picture, we have sailed in a boat on the Saar river, and saw this island, and also went to a place from which we could view this island from the hill above, we also traveled to a winery and tasted six kinds of wine and heard long explanations (in German) about these wines.

In the morning we had one more breakout session, and a deep dive into topic 2: what are the functions of event processing (including non-functional function), though for some there was difference of opinions whether it is functional or non-functional (e.g. provenance).

There are discussions about the boundaries of event processing: are "actions" internal or external to event processing: they seem to be external, but for provenance and retraction, the event processing system should be aware of them. The team also identified a collection of topics that require further research, here is the list:

–

• Use of EP to predict (anticipate) problems
• Use of predictions (e.g. from simulations) in EP
• Complex actions
• Action processing as the converse of event processing
• Decomposition of complex actions with time constraints
• Goal directed reaction
• Adaptive planning
• Implicit validation
• Function placement and optimization
• Real-time machine generated specification
• Compensation and Retraction
• Privacy and Security
• Probabilistic events
• Provenance

I'll write more about future research topics. Today we finish the deep dives and starting to wrap-up, determine the structure and schedule of the final document, and move to discuss the most important stuff -- what we want to achieve, what are the follow-ups, and what will be the follow-up actions?

There is another Dagstuhl tradition - to take a group picture, always in the same place, on the stairs of the castle's old church:

Dagstuhl seminar on event processing - the second day

This is the Dagatuhl logo, we are about to start the third day -- a shorter day in terms of work, since we are doing excursion for half a day today. In the second day: two sessions of breakouts in groups, I am visiting each session a different group, and two "deep dive" session -- the first one on "why" event processing, in which they have tried to come up with cost/benefit model, and the second is the event processing grand challenge, in which they presented "smart society" as a main theme, with variety of "smart" application, but the main message is that while each application in the list may be a challenge, the grand challenge is the holistic view, namely, the interdependencies among all these systems.

In the evening, over some wine, beer and cheese, there was a more informal session (no slides) in which different vendor representatives told us some lessons:

Marco Sierio from ruleCore started in eight lessons he devised in the train on his way, one of them is that people have hard time to adjust to the "event driven" thinking, since they are programmed to think in "request/response", Marco will probably put all eight lessons on his Blog, but one of the interesting lessons is that reading research papers is a time well spent, although it is not an easy task to do.

Richard Tibbetts from Streambase also provided some insights from his experience, ending with a statement that he did not see a lot of demand for pattern detection with his customers. Maybe in trading applications patterns detection is not a natural thing to do ?

Badrish Chandramouli from Microsoft provided some insights about StreamInsights. The interesting feature is the temporal model, which allow events to be defined as point event, interval event, and interval events with fuzzy boundaries. They also allow speculative computation and out-of-order processing, at least to some extent.

Alex Kozlenkov from Betfair, is both an user, and a developer of Prova, a self-made event processing platform, that follows the EPN/EPA model that we describe in the EPIA book (well - some variation of it) and explained its features in his usual enthusiastic way.

Martin Hirzel from IBM System S, talked about some of his impressions in working on that team, Martin is a programming languages person, working on the SPL (Stream Processing Language) and provided some insights about building a language by multi-disciplinary team.

Udo Pletat also from IBM, but from the software services organization, provided some insights on applying RFID oriented applications with customers (this is based on the Websphere Sensors Events product of IBM which has embedded Amit), and provided some examples of why they needed to use event processing patterns to track people's movement in chemical plants with restricted zones.

The session ended after 11pm and was very interesting gathering. Today -- the deep dive about "what is event processing" - more later.