Announcing the EPTS awards

EPTS has announced on a competition to grant awards of two types:

The innovative application award, in cooperation with OMG Event Processing Community of Practice, will be awarded to an application that have used event processing in an innovative way.

The innovative principles award, will be awarded to a research contribution that had a significant impact on the state-of-the-art in event processing.

The competition is open to everybody. Deadline for nomination submissions is September 30, and the award will be given in November and will publicized in the media.

Full details about the award program can be found on the EPTS external website.

Next week in DEBS we'll hand out a flyer with the announcement and start communicating the award program

On VLDB 2010 -- events and streams related papers

After a few years of missing VLDB, I plan this year to participate in the VLDB conference in Singapore (which will be an opportunity to visit Singapore, I have never been there). I have a tutorial accepted entitled: Event Processing - past, present, future. VLDB is one of the major research conferences of the database community (my original home community).

The list of accepted papers is now on the website -- looking at it there are some papers whose title include either events or streams:

Complex Event Detection at Wire Speed with FPGAs

High-Performance Dynamic Pattern Matching over Disordered Streams

Achieving High Output Quality under Limited Resources through Structure-based Spilling in XML Streams

SECRET: A Model for Analysis of the Execution Semantics of Stream Processing Systems

Recognizing Patterns in Streams with Imprecise Timestamps

On Dense Pattern Mining in Graph Streams

Database-support for Continuous Prediction Queries over Streaming Data

Conditioning and Aggregating Uncertain Data Streams: Going Beyond Expectations

From a Stream of Relational Queries to Distributed Stream Processing

iFlow: An Approach for Fast and Reliable Internet-Scale Stream Processing Utilizing Detouring and Replication

And some demos:

Active Complex Event Processing: Applications in RealTime Health Care

Efficient Event Processing through Reconfigurable Hardware for Algorithmic Trading

Geospatial Stream Query Processing using Microsoft SQL Server StreamInsight

As you can notice - in the database community the term "streams" is more common than the term "events", and I'll go back to the discussion of streams vs. events soon.

IBM will have a substantial presence in VLDB with 9 research papers, 5 industrial papers, 3 demos and 1 tutorial.

More - later.

The event processing grand challenge - take one

In the Dagstuhl seminar on event processing that ended 10 days ago we launched the "event processing grand challenge activity". You may be familiar with the DARPA grand challenge of driving driver-less car through the desert, you can see the two winner cars in the pictures above. We are not DARPA, and don't have money to distribute, but we would like to take the opposite direction, first define the grand challenge, and then convince funding authorities that they want to support it.

Why do we need this grand challenge? The research community has incubated the "event processing" area as we know today --- some research projects in the 1990-ies, such as: David Luckham's Rapide in Stanford, Mani Chandy's Infospheres in Caltech, John Bates' Apama in Cambridge, and our own Amit project in IBM Haifa Research Lab, followed by the various stream processing projects, like Jennifer Widom's stream project in Stanford, and later the Aurora project, to name a few (there are many more, of course).

The state of the practice is now in the hands of the software vendors, and event processing is becoming part of the main-stream of enterprise computing. However, software vendors are by nature advancing technology in an incremental fashion. On the other hand, there is a strong feeling that "event processing" has barely scratched the surface of its potential to impact society, this goes beyond current applications, and even beyond enterprise computing as we know it.

It is now the role of the research community to jump-start and incubate the step-function required to be achieved in order to get this kind of impact. We would like to have a call to the research community in event processing to focus on such a grand challenge, and as an incentive (and enabler) to get funding agencies worldwide to adopt it.

A substantial brain power have been invested in Dagstuhl, and in a follow-up to work on it, there have also been some people outside the community that deal with socio-technical systems and bio-informatics.

As a metaphor we can view many type of IT systems, social systems and biological system as a "live ecology", similar to a single organism with a lot of brains, eyes, ears, hands and feet.
Event processing serves as the nervous system, and events are what flow between the different players. This subsumes the "Internet of things" vision, where many sensors are connected, and also robots of various kinds as actuators, serving as hands. Such an infrastructure will enable changing the life as we know them.

We are working now on various scenarios that will be enabled by such an infrastructure; you can hear some thoughts about it in my talk next week in the OMG Event Processing community of practice that holds its first event processing symposium. Stay tuned for much more on that topic.

Dagstuhl seminar on event processing - the fourth day

Packing and about to leave the spartan room in the Dagstuhl Schloss towards the last session. Today at noon the Dagstuhl seminar will be over, and the easier part of the mission will be complete, now we'll have to finish the document and devise the follow-up action items and mechanism to track them. I'll summarize the seminar later-- yesterday there were deep dives on standards and on the issue of relations between event processing to other areas and disciplines. We also had evening session with Alex Buchmann as moderator, about the question -- whether "event processing" has become a research community, is the flagship conference of the community DEBS succeeding, and how we would like to evolve it? should we do ACM SIG and when? should we have a publication like Sigmod Record and when? should we have an academic journal and when? it seems that we are on track, still need to recruit more people whose participation can gain DEBS a status of top conference that don't send paper to DEBS since they are sending just to top conferences, so it is a chicken and egg issue, and we have a challenge to reach out to adjacent communities. We'll take it into account when designing DEBS 2011.

More -later.

On Adaptive services

Chameleon is an adaptive animal, it can adapt it color to the environment. In the next few weeks my main task will be to complete a proposal for EU project that deals with adaptive services. I have (mistakenly?) agreed to coordinate a consortium that creates a proposal. We had a meeting in Trento following the EPTS event processing symposium. EU project has a benefit of getting funding for research that can explore more advanced topics then can be funded by commercial corporates, and also provide an opportunity to collaborate with some very good people both in industry and academia. The down side is that there should be much investment in the proposal, since it became extremely competitive.

The proposal is about the idea of adaptive services -- which means in plain words that the behavior of the system should be adapted to (unexpected?) events. This adaptation may require human interaction (e.g. modifying medical treatment protocols), or self-adaptation (e.g. change of emergency handling protocol on the fly). The research challenges here are in area of event processing -- having more advanced features that are beyond the state-of-the-art; adaptation, modeling and methodology of how to build such systems. More about this topic - later.

Live from DEBS 2009 II - our paper presentation

Stratification is one of the terms that computer scientists borrowed from Geology. In 2007, Ayelet Biger, my former M.Sc. student has done her thesis about Complex Event Processing Scalability by Partition which looked at semantic partition of an EPN graph to strata, where in each stratum all agents are independent and can run safely in parallel. Today we have reported about 2008 research project that took the stratification idea and developed a system to assign agents to machines in a distributed environment. Geetika Lakshmanan delivered the talk about this project today. I have posted it on Slidshare. Enjoy !

This project is an interesting example for a life-cycle of a project:

• It started as an academic thesis;
• It has flown to a research project within IBM Haifa Research Lab ;
• After showing promising results in the lab, it evolved to a more "down to earth" project that deals with assignment of agents to machines and threads within the IBM product - WBE-XS (Websphere Business Events - Extreme Scale) which enable event processing on grid environment. This project has to take into account products and their implementation, deal with product instrumentation, and other stuff that pure research projects do not deal with.

While starting an idea in the academia and then going to a start-up has its magic, the work in IBM Research enables to get stuff from academic projects until impact through products, and using my hat as an adjunct professor in the Technion, this is possible. However -- as noted before, getting things pushed in a big company are not necessarily easy.

On the next generation of event processing

I was asked by several people to post the presentation I have given recently in several places - the presentation including some views about where should we go in the next generation of event processing, what are the challenges in the various areas (the above illustration is a slide classifying the challenge areas) and a survey of some activities we are doing either in IBM Haifa Research Lab or with graduate students at the Technion. The presentation is now on slideshare;
enjoy.

What the event processing discipline can learn from other disciplines ?

This is a picture of the Bahai Gardens, one of the famous sites in my home-city Haifa, the Bahai religion is an interesting one, more modern than most religions (I myself am agnostic and do not practice any religion, so this is not meant to be an endorsement), the Bahai people see Haifa as one of their holy sites and invest a lot in the city, this week they have a major celebration. I have returned to Haifa after my short trip abroad, in which I have given four times the same talk about "event processing - the next generation" (I'll post in on the web soon). One of the discussion points have been what can the event processing discipline (complex or not) learn from other disciplines that succeeded. Coming from a database background, it is always interesting to me to make a comparison there. When relational databases started to become products in the early 1980-ies, I have been a database practitioner with experience in several DBMS products, and in the beginning I looked at the relational model without much respect, it seemed to me to be over-simplification that gives up semantics and creates a lot of anomalies. However, the simplicity has been the main benefit. The relational model has won, and also created a big research community around it that concentrated forces around a single model and developed query optimizations, more semantic abstractions on top of it and some other stuff. The fact that there has been a substantial brain power dedicated towards a single direction was a contribution to success. The fact that was not a critical mass of work around object-based databases contributed to the fact that its success has been modest. What can we learn from that in the "event processing" discipline ? We need to strive to find the formal model that will be the basis for concentrating the community around. The model is not extension to the relational model, since this extension will loose the main benefit of the relational model -- simplicity. There are several relational algebras around, but all of them do not meet the simplicity criterion, on the contrary -- they are quite complex. So it is still a major challenge for the community. More on that and possible directions -- in subsequent postings.

Visiting Aston University and MEAP for "event processing in action"

This are pictures from Aston University in Birmingham which I am visiting today within my short trip, it turns out that from the university network I cannot get into the IBM VPN, so my hosts brought one of the university IT guys who confirmed that they are blocking access to private VPNs, as a matter of security policy, and the only person who knows how to configure a bypass is away today, and even if he has not been away, he doubts if he would have willing to do it, I could in principle go to town an look for an Internet Cafe, but I am too lazy, and can live a couple of days without Emails. Besides that the visit (which has not been over yet) has been interesting and we may have grounds for collaboration . I have given my talk about "event processing - the next generation" third time this week (fourth time tomorrow in another place...), I'll post it on the web after my trip. One of the question I was asked was whether (complex?) event processing techniques could have predicted the economic crisis, but I'll leave discussion about predictions to another Blog posting. They also may teach an event processing course next year and are looking for a textbook to base the course upon.

Speaking of book -- the publisher of the "Event Processing in Action" book has gone another step and included the EPIA book (currently the first two chapter drafts) in the MEAP (Manning Early Access Program), the referenced site explains how readers can become part of the authoring process by receiving draft of new chapters, and making comments/questions to the authors using the forum to ask questions and communicate with the authors. As mentioned before the introduction chapter has been posted as a green paper, and is free download to all. Although open type of writing is somewhat more difficult and time-consuming to the authors then just write the book without interruptions, I believe that this process can improve the quality of the book beyond the formal review process. So this is a call for the community to take advantage of this program and help in creating this book. Hans Gilde has already made the first set of comments for Chapter one.

I am not sure whether it is related to blogging about the book being written, but yesterday (Tuesday the 24th of March, 2009) has been a record high in terms of amount of visitors for this Blog in a single day, today has not ended it, and is also looks strong, so I wonder why.

Tomorrow -- visiting the IBM Hursley Lab in UK, before returning home.

Visiting Cambridge

Today I am in Birmingham, England. Yesterday I have visited Cambridge University for a few hours and gave a presentation about "event processing - the next generation", second time out of the four times I am going to give this presentation in various places this week. Cambridge has a relatively large research group called Opera, directed by Jean Bacon and Ken Moody, which deals with event-based middleware (among other things). Jean is also the "grandmother" of Apama, since John Bates has been her Ph.D. student. Apama still maintains a site in Cambridge, and some of the Apama guys came to hear my talk. While there were some periods that I did other things and had looser connections with the research community, I have returned last year to a scientist position, and am trying to keep closer contact with the research community, by visits, collaboration, and helping in the organization of DEBS as the flagship research conference in the event-based computing area. Later today I'll be visiting Aston University here in Birmingham, where my good friend, colleague, and mentor, Robert Berry, has taken recently a position of Dean of the School of Engineering and Applied Sciences which I'll meet later today.

BTW -- for some reason I cannot get to the IBM VPN from here, probably something in security setting of the local Internet server, so anybody that contacted me by Email may have to wait until later this week.

On Continuous Monitoring and Continous Actions

In Israel, a new law has gone into effect in December 1st, to fight Spam - it forbids sending commercial material over Email, SMS and recorded phone calls, unless the person signed up explicitly (e.g. through a website); recorded phone calls are especially bad, picking up the phone and talking to recorded message is something I am allergic to... Unfortunately, this law is in effect only in Israel, and I keep getting Spam in Turkish and Russian -- I have already blocked more that dozen spamming sources, but others keep pooping.

Anyway -- this week, my Master student, Elad Margalit, had his final exam on the thesis, his thesis dealt in event-driven control of traffic lights, and he has shown (by simulation) that when the traffic lights frequencies of green and red in individual junctions are optimized based on the actual traffic it can shorten substantially the waiting times (there may be various goal functions here). This is done by continuous monitoring of the traffic (entrance and exit from a road segment by camera). IBM Research has done a lot of work in the area of "continuous optimization" that deals with solving optimization problem continuously, there are two variations: Solve the optimization problem all the time (in our case - for each cycle of the traffic light), or monitor all the time and solve the optimization problem only when there is a significant change from the base assumptions (in our case -- when the distribution of traffic in the various direction is significantly different, otherwise leave the traffic lights policies as is), this is based on the assumption that the cost of monitoring is much lower than the cost of actual changing the system behavior; if no such difference then we can use the continuous actions.

It is interesting to note that Coral8 added in its logo the trademarked expression "Continuous Intelligence". Interesting phrase, not sure if there is indeed intelligence, but I guess that intelligence is in the eyes of beholder.

On the Genesis and Exodus in Event Processing

One of the greatest scientists I had the honor to meet in person (in a conference in France, 1991) is Lofti Zadeh the inventor of "fuzzy sets" which is one of the major ways to formulate inexact thinking. When I was an undergraduate student, there was an urban legend that Zadeh came with the fuzzy notion when his wife went out of town and left him a cooking recipe, trying to formalize the recipe he came out with the notion of fuzzy. Later in life I've met another great scientist and wonderful person, the late Manfred Kochen, who told me overa lunch in Ann-Arbor, that he has been a graduate student together with Zadeh in Columbia University; so I told him the urban legend and asked him if it is true, he was quite amused to hear it, and said that the problem that actually started the thinking about fuzzy theory was - formalizing the process of parking car between two parking cars, assuming the Fred Kochen told me the truth, was the genesis of fuzzy logic. It was interesting to observe that Tim Bass, in a couple of his latest Blog postings, have returned to the genesis of "complex event processing" citing topics that emerge from the papers that David Luckham's group in Stanford published in the late 1990-ies - the list contained:

• Network Level Monitoring and Management;
• Cyber Security: Network Intrusion Detection;
• Enterprise Monitoring and Management,
• Modelling and Simulation of Collaborative Business Processes;
• Business Policy Monitoring;
• Analysis and Debugging of Distributed Systems.

These applications are all still very much alive and kicking in the event processing space.

It is interesting to note that the genesis of data stream management in one of the earliest papers of the "stream" project, has been, surprise, surprise -- "network traffic management". It also should be noted that David Luckham and Jennifer Widom reside in the same building.

As the area of event processing have many ancestors - they have some more genesis books, for example, the term "active database" was first coined by Morgenstern in his VLDB paper from 1983 , and the genesis of Morgenstern has been - consistency and integrity applications. We still see compliance and governance (our current names) as major applications. Other ancestors are in the area of system management whose genesis has been the "root cause analysis" application - i.e. diagnostics of problems out of symptoms. We in the AMiT project in IBM Haifa Research Lab started also with looking at system management applications, and what is now called "business services management" - impact analysis of events in the IT on business processes. I think that at least some of the pub/sub companies started with distribution of new versions of software to subscribers, and of course some of the current event processing vendors started with applications like algorithmic trading in capital markets.

If we have used the biblical term genesis, we also may remember that the successor of "genesis" is "exodus", and in our term -- moving on and not staying just where we happened to start. While some of the software industry is based on niche players, where the niche may be quite big (one of the biggest IT companies in Israel has concentrated for many years mostly in the area of Telco billing, probably big enough to enable niche companies of several thousands employees), however, for more basic software like event processing tools, there is a big benefit in the ability to generalize beyond the genesis, and indeed we see now that some vendors are going after other markets that may seem beyond their "comfort zone" and need to make some adjustments (this phenomenon may be one of the drivers for standardization in this area, but I'll discuss this issue in another time), thus, we are watching growing list of applications and business problems that event processing can be part of its solution, both in the infrastructure area (which should grow to internet scale infrastructure) and the enterprise application area. To conclude this posting with citing another great speaker, Professor Stu Madnick from MIT, whom I remember giving an amusing talk about theoretical computer science saying something like: A bunch of people went to a close room taking with them some problems from the outside world, and since then they are still in the same close room, still working on the same problems, and sometimes inventing new problems . Well - we shall still solve the original problems, but also look around to find new ones, we are just in the early days of the event processing area, and probably did not discover much of its power to impact the business world. More - Later.

On the EPTS 4th event processing symposium

Still in Connecticut, whose flag you can see above. The three days of the 4th event processing symposium are over - what a relief, took a lot of effort to organize it - but it seems that people were happy with the content; I'll let other blogs review the meeting, some already done it - Brenda Michelson had several postings (this one, for example), Paul Vincent also wrote some of his impressions. Needless to say that I am exhausted from these days, but I still need to summarize the (relatively big!) amount of action items and send the EPTS members, may be tomorrow morning. Anyway - some short impressions:

• It seems that people feel more mature now to discuss issues like: benchmarks and languages, that they did not really want to discuss before, we have action items on both areas.



• Brenda Michelson has given an interesting presentation in the business oriented panel, showing that EP is actually under-hyped relatively to some other hypes, the panel moderator, Alan Lundberg, asked the audience if anybody think that EP (or CEP) is over-hyped, and no one raised hand. It should be noted that audience consisted not only of vendors but from analysts, consultants, academic people and customers.



• The use cases workgroup demonstrated some of the varaince we have under event processing (not sure that everything is CEP according to the glossary definition, but EPTS is about EP, not just CEP). The six has different use cases were: Alex Kozlenkov from Betfair talked about fraud detection in the betting business, although the audience did not agree that all his examples represented fraud; Arkady Godin from MITRE talked about large dissemination information (sophisticated pub/sub in the area of weather), Brian Connell from WestGlobal talked about performance monitoring of mobile operators (a monitoring system), Dieter Gawlick from Oracle talked on "first responders" in case of emergency - again a dissemination system, Guy Sharon from IBM talked about facility safety - a kind of monitoring system, and Richard Tibbetts from Streambase talked about alternative trading venues (operational system).



• Susan Urban presented a list of research challenges, and there was report about various research projects, including two presentations from IBM Research which has very active research in this area; Fred Douglis talked about "system S", while my colleague from IBM Haifa Research Lab, Guy Sharon, talked about the history of AMiT, and about the current research projects we are conducting now, the one which attracted attention is the EPDL (Event Processing Description Language) project, and it seems that we may accelerate its exposure to the larger community.



• Chris Ferris, the IBM CTO of Industry standards, explained the value of standards in general started with the chinese king who unified China, following by a standard panel - it seems that the community is now more mature to start talking about standards than in the previous years - I wonder what is the reason.



• Today in the business meeting - quite a lot of activities and working groups have been proposed. We'll see how much of it will materialize, people are more enthusiastic to commit on investing time when they are out of the office, until we return to the grey reality... but it seems that the EPTS activity is accelerating.

Some Trivia:

• We had 70 participants (the upper limit we set) - 41 vendors, and the rest - academic people, customers, analysts and consultants.

We had participants from 11 countries - see chart below (since the conference has taken place in the USA, we miss much of the European academic community).

More - Later.

On research and practice in event processing

Triggered by a question of Hans Glide to a previous posting, today's topic is the relationships between research and practice in event processing. I'll not go to ancient history of the event processing area ansectors such as: simulation, active databases etc.., but start from mide to late 1990-ies, when the idea to generate a generic languages, tools and engines for event processing has emerged. This area has emerged in the research community. David Luckham and his team in Stanford, has done the Rapide project, Mani Chandy and his team in Cal Tech has done the Infospheres project, John Bates has been a faculty member in Cambridge University and Apama was a continuation of his academic work, my own contribution has been in establishing the AMIT project in IBM Haifa Research Lab, which is also part of the research community (kind of..). In the "stream processing" front there have been various academic projects - The Stream project in Stanford, The Aurora project in Brown/MIT/Brandeis, this are just samples, and there were more - however, the interesting observation is that the research projects have been there before the commercial implementation, furthermore, many of the commercial implementation were descendents of academic projects, examples are: Isphers was descendent of Infospheres, Apama was descendent of John Bates' work, Streambase was a descendent of Aurora, Coral8 was a descendent of the Stnaford stream project, and probably there are more. However, when commercial products are introduced, the world is changing, and there is a danger of disconnect between the research community and the commercial world, since products have life of their own, and are being developed to various directions, while people in the research community continue in many cases with the inertia to work on topics that may not be consistent with the problems that the vendors and customers see. While wild research is essential to breakthroughs, the reality provides a lot of research topics that have not been anticipated in the lab, and there is a need to do synchronization in order to obtain relevant research.

The Dagstuhl seminar in May 2007, where people from academia and industry met for five days and discussed this issue has been one step, my friend Rainer von Ammon organizes periodic meetings on these issues, and a European project may spin off these meetings We shall discuss this topic in the EPTS symposium, we have more than 20 members that are part of the research community, many of them will also participate in the meeting.

Bottom line: the life cycle is --

1. Ideas start in the research community.

2, At some point the commercial world catches-up.

3. Parallel directions - research continues, commercial products evolve to their own way.

4. Synchronization, exchange of knowledge, ideas flow in both directions -- need guidance.

More - later.

On event Driven BPEL

WS-BPEL has become a leading standard in the area of business process management, and since event processing is typically part of a bigger picture, it often interacts with business process management systems. One of the success factors is to make the environment ready for event processing, i.e. being able to get the right events, and being able to perform the right actions, while the event processing network has detected a situation (in one of the next postings I'll discuss how situation is related to the EPN world).

My student, Alex Kofman, has been doing recently a M.Sc. thesis looking from the BPEL perspective, he has determined what need to be done in order to extend BPEL to be "event driven", and proposed detailed modifications to the standard. This is intended to enable easy interaction between event processing system and BPM systems that plays as producer and consumer from the event processing viewpoint, while from the BPM viewpoint, input events can serve in decisions about orchestration, while output events can provide decisions external to the BPM based on the BPM state.

Here are some of the main points raised in this thesis:

1. Ability of BPM subscribe/unsubscribe to events.
2. Ability of BPM to publish events.
3. Ability to invalidate one or more processes as a result of an event (partially exists).
4. Ability to start a new process as a result of an event (partially exists)
5. Ability to invoke a task upon occurrence of an event
6. Adapt process execution upon occurrence of an event
7. Support the definition of an event processing pattern requested by the BPM engine, as a "callback" to an event processing system.
This in general will raise the level of abstraction and make the integration easier...

More about this project, later -- the thesis is now being written, Alex gave presentation about it last week in the department seminar, so this is a good time to start communicating these ideas.

On EPTS again - calling for customers to join EPTS

After the first excitement of the launch (first press release is out, press briefing is out, some more press articles are coming soon, some of the members have referred in blogs or news, second press release is on its way) this is the time to extend the community. We have very good coverage of vendors, good coverage of academic people, some coverage of analysts. As I have written in the last blog, the target now is to add more customers. EPTS will issue a call for customers - in various places and opportunities (e.g. the Gartner EPS). What is the main motivation of customers to join:

• Impact: A lot of activities are forthcoming that will help shaping technologies, their positioning, studies about ROI, terminology, and interoperability issues. EPTS members will be able to impact all these activities - either directly (participating in working groups) or indirectly (reviewing and commenting). Some of the customers community have a lot to contribute - they give talks about EP, they blog about it, and they express opinions on forums. This will be an organized way to translate the opinions and knowledge to impact.
• Learning: EPTS will provide various community activities that will able its members to learn both on the technology and the business impact side of event processing, as well as sharing and learning best practices.
• Networking: The various conferences, the work-group conference calls, and other activities will enable active networking, besides networking facilities already exists (user groups, forums and social networks - that will all still exist).
• Help influence the establishment of "event processing" as an academic direction, impact research directions, and university level courses.

To remind you - there is no cost associated with joining EPTS, directions about joining can be found here: http://www.ep-ts.com/content/view/59/93/

Stay tuned for call for participation and contributions in the EPTS annual meeting in September 17-19, coming soon.

On Event Processing Research Challenges

This picture has been taken a year ago on the stairs to the old church in Schloss Dagstuhl, in that meeting we had people both from academia and from industry discussing the state of the art and future of event processing. I have returned to the conclusion of the Dagstuhl Seminar done by my colleague Peter Niblett from IBM recently, after my return to the IBM Research Division from spending several years in the product organization, I was asked about challenges to the research community, as seen by the product development community (or the industry in general), since we had a session about it in Dagstuhl, in which people from the industry expressed their opinions about the same questions, I just had to take it as a basis of a presentation in this area.

In Dagstuhl four major areas has been put as challenge to the Research community (adding my own interpretation and comments)

1. Event Processing Algebra and Meta-Language: Like the database area in the pre-relational era, the first (and probably second) generations of event processing are "engineering based", various vendors are building implementations based on the use cases they see and their innovative ideas, bringing to the table, not only a variety of languages, but also a variety of (typically - implicit) conceptual models behind this implementation. One of the indication for a maturity of an area is the existence of an agreed upon conceptual model. The relational model has done it for databases, the browser concept has done it for the internet, now we need the same for event processing. David Luckham's concept of EPN/EPA is a possible starting point, but more work is needed on this - the challenge is still there; after constructing the "relational model" for event processing we also need the "SQL" (not necessarily SQL extension - but this is a possibility) for it - in term of meta-language that describes the model and can be mapped to various implementation.

2. Software Engineering issues: This is a challenge from a different perspective. Event processing and Event Driven Architectures impose different thinking about computing. We are programmed to think in a certain way in all programming language, and the event based programming is somewhat different paradigm. Even the basic principle of decoupled asynchronous processing is something that is not easy to digest f0r people. We need software engineering tools, methodologies and best practices.

3. Implementation optimization: If we have done the analog of SQL, another analog come to mind - database tuning and query optimization. There are many optimization issues especially when the event processing applications are distributed and may have various QOS requirements - parallel processing, acceleration of hardware, inter-operability, support in illites in general - all are challenges. The goal function in this optimization is not unique, while in some systems it is maximal throughput, in other it may be scalability in number of rules/queries/patterns (there are applications in which the throughput is measured in millions, and others in which the number of rules, the number of producers, or the number of consumers is measured in millions...), the goal function can be a combination of multiple criteria.

4. Variety of "small issues" - examples: uncertain events, out-of-order events, retention and vacuuming of events etc....

It will be interested to look at recent research project to see how much progress has been done on these, and if the list should be modified after a year that has passed. The major research event of the EP community this year is DEBS 2008 , the program has not been published yet, and it will be interesting to compare the accepted papers with this list (I'll do it when surveying the conference itself in July). We also intend to do in the EPTS event processing symposium in September a session with academic people about it - stay tuned for further notice. I am also thinking about a follow-up Dagstuhl Seminar, may be in 2009 or 2010 - to re-discuss event processing in perspective. More - later.

Moving to a new job - but actually staying in the same office

This is part of the Haifa Bay as seen from the IBM building (not from my office - I am in the back side!). On April 1st I am moving to a new job - without physically moving office (which means that I am still in IBM).

What role I am completing ? in the last three years I have been in a technical strategy role for the IBM Websphere product organization.

I have arrived to the product organization when people there did not really know much about "event processing" (well, IBM has done some forms of event processing in the past, but the people inside IBM did not think of that this way), I have worked on technical strategy, and business strategy, chairing and participating in workgroups, worked on business evaluation with the business side of IBM, working on comprehansive requirement survey in multiple industries, participating in many reviews, preparing answers for skeptic people etc.. I am leaving this position in a state that IBM sees event processing as part of its main stream middleware (as evident in the acquisition of Aptsoft, and some other activities), and as the next big thing.

Of course, the relationships between my presence and IBM's decision to jump on board the event processing ship may be just a coincidence...

Anyway - it seems that the next challange for me will be in shaping the next generation, while I still be somewhat involved in the work on the current generation.

One of my findings is that the "event revolution" has not occured yet, and I would like to go back to the drawing board, and do some work to address some fundemental challanges. In order to concentrate on this goal, I am re-joining to the IBM Research division and will focus on these research challanges (hopefully without many interruptions, each organization collects its own taxes on one's time, and IBM Research is no different ). What has not changed ? I am continuing to represent IBM in the EPTS and will honor all external commitments.

Stay tuned for call for membership in EPTS - coming this week.