The event processing grand challenge - live ecology

Still travelling in Europe, I have spent one day in Luxembourg, serving as a reviewer in the Pronto project that deals with "event recognition for resource management", the highlight was showing us a nice demo of a van that was covered with sensors that track noise, density and location of the car, and can provide diagnostics about the comfort of passengers (too dense, too noisy) and the quality of driving (unsafe driving, sharp turns). I also discovered that one can miss train connections and not only flight connections, and arrived to the hotel in Luxembourg very late at night. From Luxembourg I continued in another 3 hours train ride to Brussels, from which I am writing now.

In Brussels I am attending an EU conference about the FET Flagship program. The European Commission is trying to diversify its programs in order to achieve more impact, and invest more in high risk - high impact long term research; there are several big consortia already established, and I will try to see whether the "event processing grand challenge" ideas, can fit into one or more of these, I am planned to deliver a short presentation today on "live ecology", taking the theme of making Asimov, my favorite writer, come to live with some of his amazing ideas, starting from Gaia ("The Foundation Edge"), a galaxy behaving like a single organism, and moving through the fantastic voyage, the blend between human and robot, and predicting the society's course (Psychohistory), and then hopefully return home later tonight.

Events of the day - Darmstadt and OMG

I am now in Darmstadt, Germany, starting a long day of visiting the Technical University of Darmstadt, and giving a seminar talk at the university. In the evening I'll have to reach my next destination, Luxembourg, and this seems to be done in combination of 3 trains from here, will still need to figure it out.

Also, today you can hopefully hear me in the OMG event processing symposium along with some respectable members of the event processing community. This symposium has been reshaped as a virtual symposium, so people can tune in using cyberspace.

On business user's computing

Friday is our shopping day (our working week is Sunday to Thursday, thus Friday and Saturday are weekend days here); today in a coffee shop in our neighborhood shopping center I have noticed a sign: "We don't accept checks or credit cards", they meant that they neither accept checks nor credit cards, obviously, but if we'll feed this sentence to a computerized program that parses logical propositions, I am not sure what the result would have been. There is a famous experiment in which they asked people to formalize a query about people who live in New-York or in Philadelphia, and many people formalized this query as people who live in New-York and Philadelphia, which reduces the answer to a person who happens to have two addresses in both cities. I am mentioning it, since recently I had some discussions about the ability of business user to write their own: {rules, patterns, queries} - chose your favorite terminology. There are users which I call "semi-technical users" who do not know how to program in any programming language, but can be taught to formalize their requirements in an accurate way. However, there are users who are not able to do it, and giving them natural language interfaces will lead to unexpected results.

So - should we give up on that? not necessarily, there are several ways to advance here, one of them is to have parameterized rules/queries/patterns, and let the user instantiate it by entering parameters. This, of course, limits the scope of programming to the predefined patterns that have been created by the IT developer; another way is to use some kind of dialog with the user to interview the user about the precise meaning, this area needs more work, but it is vital in extending the scope of those who can program event processing (and also some tangent areas like business rules) applications. More about it - later

DEBS 2010 program has been published

The program of DEBS 2010 is now published on the conference's website. Note that IBM has become a research power, based on its representation in DEBS.

In total IBMers have authored or co-authored 10 papers out of the 21 papers in the research and industry tracks, and 2 out of the 6 tutorials.

Some details:

Tutorials:

Ella Rabinovich and myself will deliver a tutorial prepared by several members of our team, on: context Aware Computing and its Utilization in Event-Based System

This is the third year in a raw that I am giving (alone or with others) a tutorial in DEBS.

Guy Sharon, another member of our team, participated in the preparation of a tutorial on Measuring Business Value of CEP-Applications.

Papers:

I am a co-author of the paper: Analyzing the Behavior of Event Processing Applications.

Another paper by members of our team is: Industry Experience with the IBM Active Middleware Technology (AMiT) Complex Event Processing Engine (I have not participated in writing this one), both of these papers are planned to be presented by Ella Rabinovich.

Three additional papers are co-authored by other persons in IBM Haifa Research Lab:

Magnet: Practical Subscription Clustering for Internet-Scale Publish/Subscribe

Quilt: A Patchwork of Multicast Regions

On Trade-offs in Event Delivery Systems

Four papers are authored by IBM Watson Research center's guys:

Evaluation of Streaming Aggregation on Parallel Hardware Architectures

Placement of Replicated Tasks for Distributed Stream Processing Systems

Experiences with Codifying Event Processing Function Patterns

Workload Characterization for Operator-Based Distributed Stream Processing Applications

And another paper co-authored by Udo Pletat from IBM Boeblingen

Distributed Heterogeneous Event Processing

This is an indication for the importance that IBM sees for investing in research and thought leadership in the areas of event processing, stream analytics, and pub/sub. More about DEBS -- in July; meanwhile another business trip to Europe is waiting for me in Sunday.

The event processing grand challenge - take one

In the Dagstuhl seminar on event processing that ended 10 days ago we launched the "event processing grand challenge activity". You may be familiar with the DARPA grand challenge of driving driver-less car through the desert, you can see the two winner cars in the pictures above. We are not DARPA, and don't have money to distribute, but we would like to take the opposite direction, first define the grand challenge, and then convince funding authorities that they want to support it.

Why do we need this grand challenge? The research community has incubated the "event processing" area as we know today --- some research projects in the 1990-ies, such as: David Luckham's Rapide in Stanford, Mani Chandy's Infospheres in Caltech, John Bates' Apama in Cambridge, and our own Amit project in IBM Haifa Research Lab, followed by the various stream processing projects, like Jennifer Widom's stream project in Stanford, and later the Aurora project, to name a few (there are many more, of course).

The state of the practice is now in the hands of the software vendors, and event processing is becoming part of the main-stream of enterprise computing. However, software vendors are by nature advancing technology in an incremental fashion. On the other hand, there is a strong feeling that "event processing" has barely scratched the surface of its potential to impact society, this goes beyond current applications, and even beyond enterprise computing as we know it.

It is now the role of the research community to jump-start and incubate the step-function required to be achieved in order to get this kind of impact. We would like to have a call to the research community in event processing to focus on such a grand challenge, and as an incentive (and enabler) to get funding agencies worldwide to adopt it.

A substantial brain power have been invested in Dagstuhl, and in a follow-up to work on it, there have also been some people outside the community that deal with socio-technical systems and bio-informatics.

As a metaphor we can view many type of IT systems, social systems and biological system as a "live ecology", similar to a single organism with a lot of brains, eyes, ears, hands and feet.
Event processing serves as the nervous system, and events are what flow between the different players. This subsumes the "Internet of things" vision, where many sensors are connected, and also robots of various kinds as actuators, serving as hands. Such an infrastructure will enable changing the life as we know them.

We are working now on various scenarios that will be enabled by such an infrastructure; you can hear some thoughts about it in my talk next week in the OMG Event Processing community of practice that holds its first event processing symposium. Stay tuned for much more on that topic.

On event processing standards

One of the Dagstuhl follow-up will be to have action items in advancing standards in the event processing world. One of the topics discussed in Dagstuhl was standards, The team working on it was moderated by Paul Vincent, who already blogged about it. While I'll write more about it when the final report will be ready, here are some initial thoughts:

1. It seems that in the era where the vendor community is now dominated by bigger companies, the atmosphere for standards become more friendly.
2. For other communities - standards have been critical success factor, e.g. web services.
3. Somebody mentioned the immortal Stonbraker's phrase about SQL being "intergalactic data speak", we need the "intergalactic event speak" - and it is not an extension to SQL.
4. There are different standardization issues -- event representation, meta-modelling, event processing language; as well as extensions to many existing standards possible.
5. The language standardization will be trickiest - due to the variety of languages styles exist, here I think that we'll better start with a language at the PIM (platform independent model) level. In the EPIA book we provided one that can serve as a starting point.

I believe that standards is one of the important ways to go forward, and will write more about it when we'll have the Dagstuhl final report.

Dagstuhl seminar on event processing - the conclusion

The Dagstuhl seminar on event processing has ended in Friday, and I have arrived home Saturday early morning. This has been my fourth visit in Schloss Dagstuhl, and the magic that this place projects keeps flowing. Organizing such a seminar is a hard work, and this time we have taken ambitious task, the seminar goals were:

a) Brainstorm and devise an actionable plan for evolving event processing to be a critical technology in a grand challenge that will have major impact on society, as a wide-community effort;

b) Brainstorm and devise and actionable plan for creating a community-agreed document about the value, boundaries, functions, and synergies with other areas and communities.

c) Brainstorm and devise and actionable plan for the evolution of specific event processing standards, and employ/extend existing standards.

There are a lot of follow-ups, conclusion of the Dagstuhl seminar can be found on the seminar's webpage.

More about the document and grand challenge - later.