EPTS virtual symposium - March 24, 2011

Mark your calanders -- EPTS will hold its next (sixth) event processing symposium as a virtual symposium.
The idea of the virtual symposium is that a large audience will be able to participate through webcast, the symposium will also be recorded, so you'll be able to view the symposium or any part of it,   
The symposium will be co-located with the OMG Technical Meeting in Arlington, VA, USA, some of the speakers will be there in persons and the others will participate from remote.


The agenda was published on the EPTS site.    


The symposium will consist of three general themes:

1. Follow-up to the Dagstuhl seminar.    The concluding document ("event processing manifesto") in its final version will be published around the end of February, and the different teams will present their conclusions.
2. Discussion of event processing related standards:  this has been one of the manifesto chapters and there will also be presentation of Jim Odell about ideas within OMG around standards.
3. Report on several EPTS working groups activities

The EPTS business meeting and awards will be held in conjunction with DEBS 2011 in July.   

My 2010

In 2010 I have posted less entries in this blog, relative to 2008 and 2009, 128 postings, roughly once every three days. Maybe I've been busier, and maybe I've been lazier - probably the combination of these two. 

The happiest day of the year was surprisingly the day in August when I've returned from a family vacation in western Canada and found two things that got by mail:   

12 copies of the book EPIA that fresh out of press (I have already gave 10 of them as a present, left 2 for myself - one at home and one in the office); the EPIA book was a major task for 18 months, so I was relieved to see it in print, at some points it fell like never-ending saga. 

 

The second item that arrived it the same day is a plaque designating the fact that I've received an IBM corporate award, which is the highest award that IBM gives,  the order of the events was somewhat funny: first I got the plaque in the mail, two weeks later I got a letter signed by IBM's CEO, which notifying me that I am receiving the award, two weeks later than the list of awards was publicly published, and then after six more weeks there was the award granting ceremony, giving me again the plaque I got in August.

Another notable event was the Dagstuhl seminar on event processing in May 2010.   We are now in the final phase of editing the end result of this team work.  Dagstuhl is a wonderful place, and we had a very good team there spending  5 days in dealing with the present and future of event processing.

I had some trips abroad - both business and pleasure.   Unlike most years I spent only four days in the USA in October, for the OMG financial market conference,  spent two and half weeks in a family vacation in Western Canada,   here is a picture with my daughter Daphna somewhere in the Canadian Rockies.

Another trip was to VLDB in Singapore, with a couple of days vacation in Hong-Kong on the way, and several days vacation in Singapore.

Here is a picture from Hong-Kong's wax museum,  where I am photographed with an old friend.

Here are two pictures from Singapore, one from a zoo that resides in a rain-forest in the northern part of the island, and the second in a spa where you can get your feet cleaned by fish

Another conference I have participated in was DEBS 2010 in Cambridge, UK.

 This year I have also started the work on a new project that deals with proactive computing, and will write more about in in 2011.    

Overall --  interesting year,  the leaves a lot of unfinished challenges for the future.  

DEBS 2010 -- first day of the main conference

The first day of the main conference yesterday ended in an organ recital in the King's College Chapel, seen in the picture, magic place - people keep talking about being in Hogwarts. The conference itself started with the first keynote by Mike Franklin who shared his experience both from his work in Berkeley and in his start-up Truviso. Mike's main motivation is to extend database technology to react in a continuous way in addition to the traditional batch way of analytics, and view batch as a special case of the continuous. The talk was a kind of "lite sell pitch", but had some good points, like the observation that a start-up can take a known technology and try to use it for new types of applications (which is what Turviso does), or devise a new technology and tries to attack existing applications better, but it is very difficult to do both at the same time (new technology and new applications). I think that the event processing area indeed tries to do both.

There were some other talks, mostly by graduate students. One of the interesting observations (not a new observation to me), that there is a lot of energy in this community to re-invent wheels, in different variations. I think that one of the good results in the database community of the relational model was that a large part of the research community took it as a basis and constructed the research on top of it, and people did not try to re-invent databases from scratch for every thesis. In event processing we are still not there, and IMHO the area will have more substantial results, if the research efforts will be more focused on advancing the state-of-the-art instead of re-inventing most and advancing a little bit (in the best case). Today we'll have follow-up meeting to the Dagstuhl seminar, and the research grand challenge we are discussing is aimed to that matter.

In the late afternoon there were a fast abstract session, where I gave a short (8 minutes, 4 slides) talk on the interactions between business rules and event processing (I'll write about this subject some other time), and then a session of demos and posters, with a couple of follow-ups for me.

More -later. The second day of the main conference is about to start.

On event processing standards

One of the Dagstuhl follow-up will be to have action items in advancing standards in the event processing world. One of the topics discussed in Dagstuhl was standards, The team working on it was moderated by Paul Vincent, who already blogged about it. While I'll write more about it when the final report will be ready, here are some initial thoughts:

1. It seems that in the era where the vendor community is now dominated by bigger companies, the atmosphere for standards become more friendly.
2. For other communities - standards have been critical success factor, e.g. web services.
3. Somebody mentioned the immortal Stonbraker's phrase about SQL being "intergalactic data speak", we need the "intergalactic event speak" - and it is not an extension to SQL.
4. There are different standardization issues -- event representation, meta-modelling, event processing language; as well as extensions to many existing standards possible.
5. The language standardization will be trickiest - due to the variety of languages styles exist, here I think that we'll better start with a language at the PIM (platform independent model) level. In the EPIA book we provided one that can serve as a starting point.

I believe that standards is one of the important ways to go forward, and will write more about it when we'll have the Dagstuhl final report.

Dagstuhl seminar on event processing - the conclusion

The Dagstuhl seminar on event processing has ended in Friday, and I have arrived home Saturday early morning. This has been my fourth visit in Schloss Dagstuhl, and the magic that this place projects keeps flowing. Organizing such a seminar is a hard work, and this time we have taken ambitious task, the seminar goals were:

a) Brainstorm and devise an actionable plan for evolving event processing to be a critical technology in a grand challenge that will have major impact on society, as a wide-community effort;

b) Brainstorm and devise and actionable plan for creating a community-agreed document about the value, boundaries, functions, and synergies with other areas and communities.

c) Brainstorm and devise and actionable plan for the evolution of specific event processing standards, and employ/extend existing standards.

There are a lot of follow-ups, conclusion of the Dagstuhl seminar can be found on the seminar's webpage.

More about the document and grand challenge - later.

Dagstuhl seminar on event processing - the fourth day

Packing and about to leave the spartan room in the Dagstuhl Schloss towards the last session. Today at noon the Dagstuhl seminar will be over, and the easier part of the mission will be complete, now we'll have to finish the document and devise the follow-up action items and mechanism to track them. I'll summarize the seminar later-- yesterday there were deep dives on standards and on the issue of relations between event processing to other areas and disciplines. We also had evening session with Alex Buchmann as moderator, about the question -- whether "event processing" has become a research community, is the flagship conference of the community DEBS succeeding, and how we would like to evolve it? should we do ACM SIG and when? should we have a publication like Sigmod Record and when? should we have an academic journal and when? it seems that we are on track, still need to recruit more people whose participation can gain DEBS a status of top conference that don't send paper to DEBS since they are sending just to top conferences, so it is a chicken and egg issue, and we have a challenge to reach out to adjacent communities. We'll take it into account when designing DEBS 2011.

More -later.

Dagstuhl seminar on event processing - the third day

The third day in a Dagstuhl seminar traditionally has half day of trip outside the castle, this time we have traveled to a place called Metllach seen in the picture, we have sailed in a boat on the Saar river, and saw this island, and also went to a place from which we could view this island from the hill above, we also traveled to a winery and tasted six kinds of wine and heard long explanations (in German) about these wines.

In the morning we had one more breakout session, and a deep dive into topic 2: what are the functions of event processing (including non-functional function), though for some there was difference of opinions whether it is functional or non-functional (e.g. provenance).

There are discussions about the boundaries of event processing: are "actions" internal or external to event processing: they seem to be external, but for provenance and retraction, the event processing system should be aware of them. The team also identified a collection of topics that require further research, here is the list:

–

• Use of EP to predict (anticipate) problems
• Use of predictions (e.g. from simulations) in EP
• Complex actions
• Action processing as the converse of event processing
• Decomposition of complex actions with time constraints
• Goal directed reaction
• Adaptive planning
• Implicit validation
• Function placement and optimization
• Real-time machine generated specification
• Compensation and Retraction
• Privacy and Security
• Probabilistic events
• Provenance

I'll write more about future research topics. Today we finish the deep dives and starting to wrap-up, determine the structure and schedule of the final document, and move to discuss the most important stuff -- what we want to achieve, what are the follow-ups, and what will be the follow-up actions?

There is another Dagstuhl tradition - to take a group picture, always in the same place, on the stairs of the castle's old church:

Dagstuhl seminar on event processing - the second day

This is the Dagatuhl logo, we are about to start the third day -- a shorter day in terms of work, since we are doing excursion for half a day today. In the second day: two sessions of breakouts in groups, I am visiting each session a different group, and two "deep dive" session -- the first one on "why" event processing, in which they have tried to come up with cost/benefit model, and the second is the event processing grand challenge, in which they presented "smart society" as a main theme, with variety of "smart" application, but the main message is that while each application in the list may be a challenge, the grand challenge is the holistic view, namely, the interdependencies among all these systems.

In the evening, over some wine, beer and cheese, there was a more informal session (no slides) in which different vendor representatives told us some lessons:

Marco Sierio from ruleCore started in eight lessons he devised in the train on his way, one of them is that people have hard time to adjust to the "event driven" thinking, since they are programmed to think in "request/response", Marco will probably put all eight lessons on his Blog, but one of the interesting lessons is that reading research papers is a time well spent, although it is not an easy task to do.

Richard Tibbetts from Streambase also provided some insights from his experience, ending with a statement that he did not see a lot of demand for pattern detection with his customers. Maybe in trading applications patterns detection is not a natural thing to do ?

Badrish Chandramouli from Microsoft provided some insights about StreamInsights. The interesting feature is the temporal model, which allow events to be defined as point event, interval event, and interval events with fuzzy boundaries. They also allow speculative computation and out-of-order processing, at least to some extent.

Alex Kozlenkov from Betfair, is both an user, and a developer of Prova, a self-made event processing platform, that follows the EPN/EPA model that we describe in the EPIA book (well - some variation of it) and explained its features in his usual enthusiastic way.

Martin Hirzel from IBM System S, talked about some of his impressions in working on that team, Martin is a programming languages person, working on the SPL (Stream Processing Language) and provided some insights about building a language by multi-disciplinary team.

Udo Pletat also from IBM, but from the software services organization, provided some insights on applying RFID oriented applications with customers (this is based on the Websphere Sensors Events product of IBM which has embedded Amit), and provided some examples of why they needed to use event processing patterns to track people's movement in chemical plants with restricted zones.

The session ended after 11pm and was very interesting gathering. Today -- the deep dive about "what is event processing" - more later.

Dagstuhl seminar on event processing - the first day

It is evening here in Schloss Dagstuhl, the castle that you can see in the picture above. My room is on the ground floor of the part of the building in the left-hand side of the picture. This seminar organized by Rainer von Ammon, Mani Chandy and myself, is a five days event, and we just finished the first one, during the day I lost my voice for about an hour, and regained it by eating ice-cream. In this seminar we are working in five sessions, you can see this presentation includes the outline of what we are doing in the opening session and the one slide that every participant (well - almost, a few have not sent the slides) put about himself/herself.

Besides the opening session we have done the first meeting of each of the five groups, and had initial discussion about them. Tomorrow -- two breakout sessions and two deep dives. One of the things we started to discuss and we'll finish this discussion later this week is ---- looking in the future five years from now - how can we measure the success of this seminar to change the world --- more about it later.

On inter-system event causalities

A lot has been spoken and written about the Wall Street Plunge, David Luckham raised the question about the role of event processing systems in prevention and/or postmortem analysis of understanding how such a situation happened. I am not sure exactly how much event processing system have been part of the reason of causing this plunge in the first place, but here we have an interesting challenge, the challenge is that we need to understand causality in a real event cloud, where the events are part of multiple systems. The nature of the challenge is twofold:

First: Each of the automated trading systems logic is, of course, not transparent, since they are used as competitive edge by the traders, thus it is difficult to know how various systems would function, and there will be a need to learn their logic (which can be constantly changed) based on their actual behavior;

Second: The logic of the various systems is expressed in various ways, and there is no standard causality model that can be used across systems.

This can be one of the challenges that we can put into the challenges of the next generation of event processing systems, that we plan to discuss in Dagstuhl.

On consolidation and pure play in the EP market

Marc Adler returned to the Blogland this week to claim, among other things that the fact that the list of pure-play event processing platform vendors is being reduced, is sad. The fact that there have been several acquisitions of "pure play" vendors recently is true, with the acquisition of Aleri by Sybase earlier this year, and the recent acquisition of RTM by Software AG. If you are interested in the genealogy, Paul Vincent is keep documenting it. I have traced some of the IBM acquisitions and realize that merges are sometimes tough from organizational culture point of view and sometimes there is a need to change direction in a not easy fashion, as shown in the picture above, so it might be sad for some individual people, but not necessarily bad from the industry point of view.

However, the fact that the pure play vendors are being acquired has another side to the coin, which means that the big or medium software vendors are buying them. This is a sign that event processing is getting to be part of the main stream of the enterprise computing, and is part of growing up of this area. This has also happened before in other areas, and was predicted several years ago by analysts to happen in the event processing area.

In fact, now most of the major players in enterprise software area : IBM, Microsoft, Oracle, TIBCO, Progress Software, Sybase and Software AG - have now an event processing platform as part of their enterprise computing offering.

This is an indication that all these vendors realize that event processing is required part of their enterprise applications, and that event processing is not really stand-alone but it is increasingly getting consumed as part of larger play, and getting to further areas and industries in addition to the early adopters of capital market trading applications.

There may still be a role for pure play event processing products, especially in various niches that are not being properly handled by the current products. Some of them may even develop event-driven enterprise computing platform, and join the medium companies, it happened before, but it is not easy.

While the bigger companies advance the projects at their own paste, the smaller companies as well as the research community sometimes have roles of catalysts to advance the area. We'll discuss the future of the event processing area in depth in the Dagstuhl seminar planned to start in 2 weeks. More about it - later.

On computer science research publication

I have read one of the recent issues of the Communication of the ACM - as ACM member I am getting the hard copy of this magazine and from time to time I am browsing through it. One of the articles that caught my eye is an editorial article by the editor in chief, Moshe Vardi. It summarized some opinions about publications in computer science, it turns out that most disciplines make journals as the primary vehicle of publications, and view conferences more as a social gathering where most submitted papers are being accepted, while in the computer science area, conferences are a very dominant publication vehicle, and as such, the "good" conferences are very competitive, in fact, people are writing in their CV, what is the acceptance rate of each conference they published in. Vardi's article shows support in opinions that computer science has to mature, and behave like all other disciplines. This observation is true, in my previous incarnation, as faculty member in an inter-disciplinary school at the Technion, I quickly discovered that other disciplines don't really understand the term "refereed conferences", and that conference publication "does not count". One of the wrong (in my view) academic metrics is to count papers, but this is not the topic I am discussing now. From the industry research perspective, the current state of conference-centric, is very comfortable. Since (unlike our academic colleagues) writing papers is a secondary occupation, then it is comfortable to write relatively short papers based on the conferences's page restriction, it is comfortable to tell management that you have a deadline for publication then to dedicate time for undetermined deadline that always gets lower priority to other things that have deadline, and it is more comfortable for management to let the reviewers do the filtering of who should travel to conferences. Despite these facts, I think that getting computer science to act like a mature discipline is the right way to go forward. The journals and conferences will need to be adjusted. Academic journals will also are also going through a phase of coping with the Internet era, and various business models for journals are emerging. The event processing perspective of it is that we should consider to found an event processing journal, when we feel that we have enough quality content for it. The numbers of submissions to DEBS are indication that critical mass might exist, I guess that this will be discussed in some of the coming meetings (like the Dagstuhl seminar),

OMG event processing symposium and other event processing conferences

I was recently invited to give a talk in the OMG event processing symposium that will take place in Washington DC (actually, Arlington VA) in May 24-25, 2010. I will give a talk about -- event processing seven years from now --- talking about where the event processing area goes in the future, this will be a week following the Dagstuhl seminar on event processing, and some of the conclusions may be reported there. The OMG new event processing consortium that will be kicking off in that event is another indication to the importance that the IT community places on the event processing, and the visibility it gains. This is a customers' oriented conference, whose aim is to educate the general community about event processing.

We also announced that EPTS will be doing its annual conference in November, co-located with the Gartner Enterprise Architecture Foundations Seminar, in Los Angeles. In this conference EPTS (possibly in collaboration with OMG) will grant two kind of awards: event processing innovative foundation awards (for research work) and event processing innovative application award. Stay tuned for call for nominations soon.

Towards an event processing manifesto - take one

The word manifesto brings to many people an association with the Communist Manifesto by Marx and Engels, which is an example how manifesto can be used, reused and abused. The term manifesto has been adopted by the computing community, and we have seen variety of works entitled: manifesto, some of them are: object oriented database manifesto, the manifesto for agile software development, and the business rules manifesto - just to name a few examples.

In May 2010 I am co-chairing (together with Mani Chandy and Rainer von Ammon) a Dagstuhl seminar, whose task is to compose "event processing manifesto", which we strive to make it as influential effort. A Dagstuhl seminar is a five days retreat in an isolated place in which a selected group of people are convened to make deep discussions about a certain area in computer science, many of these seminars are geared towards the academic community, but it this particular one we strive to keep the balance and have half of the participants from industry (vendors, independent consultants, analysts, and some innovative users), the number of attendees in each seminar in limited, and there is a big competition on this resource (and long waiting list after approved, we have put the application to do it in 2008). Here is a picture of Schloss Dagstuhl (located in Saarland, Germany):

I guess that I'll write a lot about the Dagstuhl seminar, the event processing manifesto, and related stuff in the next few months, so this is the modest starting point.

The intention of the manifesto is simple (but not easy) -- define what event processing is, what is the scope of event processing, are there mandatory features in event processing, what are the various options and variations, what are the synergies with other disciplines, and have as an appendix a call for action for the community: what are the research challenges and what are the pragmatic challenges.

I'll start today with some of my own thoughts on the scope issue.

I view "event processing" as a name of a computing discipline like "data management", "image processing" and "information retrieval" - relatively wide disciplines that cover wide collection of applications. This is also true for event processing. Alas, we still have some misconceptions even on the scope, I have used before the metaphor of blind people touching various parts of an elephant and concluding that each of them has identified what an elephant is, so this time I'll use another animal - ant whose sight is limited to its immediate environment.

I am using ant, since one of the great Hebrew poets has written a poem saying (in free translation): "my world is narrow as the world of an ant". Some examples of it:

1. people who view event processing as aggregation of time-series events, this is definitely an event processing application, but not the only one;
2. other people are looking at detection of anomalies within event collections, where the anomalies types are not predefined as what event processing is all about, this is indeed another event processing application, but not the only one;
3. yet other people are looking at real-time matching of predefined patterns with streaming events, which is somewhat different from the two previous examples.

One might claim that there is no event processing discipline at all, but there are very distinct areas that process events in different ways for different purposes, probably using different technique; while this may be a possible conclusion from the Dagstuhl meeting, my experience shows that there are many applications that require blend of event processing functionalities, and that the space is not discrete, but has some continuity, thus event processing in the wide sense should deal with all of them (and some more) and the synergies between them. So this is a point for discussion.

While the amount of people that will be involved in the actual composition of the manifesto is restricted, we'll strive to disseminate the draft to a wide discussion, both within the EPTS community and other avenues. As a comment, the Dagstuhl seminar is not directly related to EPTS, many of the attendees are not EPTS members, and some are members of adjacent communities and not the event processing community, however I think that EPTS should have a role in follow-ups.

On event processing grand challenges

This is the logo of the "Grand Challenges SIG" of the UK Electronic design knowledge transfer network. In the 4th Event Processing Symposium last year, Arkady Godin from MITRE, proposed in the business meeting to launch a work group that will deal with grand challenges in event processing. The EPTS steering committee decided that such a workgroup may be premature, and it is better to complete some work in the other workgroups -- languages, architecture, use cases, interoprability, in order to have a better understanding of the grand challenges, however, this issue has neither forgotten nor forsaken. We determined that the best forum to discuss the grand challenges is the (second) Dagstuhl seminar on event processing, which I'll co-organized with Rainer von Ammon and Mani Chandy, we also asked to invite all people that we saw as the right set of people to participate in such an event (if you have not been invited and think you would like to contribute, please let me know, there is a waiting list for available slots if any, no promises though).

In the coming 5th event processing symposium in Trento, we shall hold the first brainstorming about grand challenges. Pending technical feasibility (still requires confirmation from local organizers) we plan to do this as a public session that all EPTS members will be able to use through audio conference call, EPTS members will get full details with call-in numbers when it will be finalized.

Grand challenges can be in multiple areas and also refer to multiple addressees. There are grand challenges that will require a community effort, and these challenges should be picked up by EPTS, some challenges are to advance the state-of-the-art, and this will be addressed to the research community, with some creative incentives. Some will be referred to the product vendors, and may be some to other adjacent communities.

Idea for such grand challenges are solicited from various sources:

• EPTS workgroups that already are active, each workgroup will be asked to contribute a challenge in its area
• The reach-out sessions for other communities in the symposium (the BPM community and the the IT management community) might also lead to some challenges
• The research community is always a source for such challenges
• Last but not least -- all EPTS members (and other interested people that wish to contribute, and thus are hopefully future EPTS members), specifically customers who has long-term vision about their systems, like our MITRE colleagues who started this discussion, and our analysts colleagues who have cross-vendor and cross-customer perspective.
  
  Anybody who wishes to contribute ideas to this session, please let me know by the end of this week.

I'll wrap-up this posting, by pointing out a Blog entry I have seen just now, written by Niels, who runs a consultancy company called "SQL Develop" as its name testifies deals with DB related issues. The upcoming Microsoft product "StreamInsight" has raised the interest in the database community, as Microsoft seems to take an SQL oriented approach to its product, and Niels is blogging about "CEP resources" he covers Blogs in this area, on my Blog he writes: THE blog about CEP, if you were to read only one blog about CEP, this is it! First, thanks to Niels for the endorsement, I don't view the Blogging area as a competition, as people who read Blog tend to read multiple Blogs, and it is also kind of a network of Blogs that sometimes react to each other. To be fair, also most Blogs in this area are marketing oriented Blogs, that they have somewhat different motivation from my Blog, and one cannot compare. Anyway, a good way to start the day.

On criteria for evaluation of event processing products

Yesterday, I spent much of the evening in Nofim elementary school, my youngest daughter Daphna, is graduating from elementary school, and they have done their grand celebration, with a lot of speeches, and performance of the children in signing, dancing and playing. It was nice, but somewhat long (the speeches part), and they have done it outside in the hottest day of the year (so far).





Another experience is that I needed to replace the door lock cylinder in my former apartment, since she is going to rent it now. I know that the Americans tend to do everything (including oil change in cars) by themselves, when I joined IBM, I was surprised to discover that I am now a "technical person", actually my current IBM title is - IBM Senior Technical Staff Member; I have never thought about software development as a technical activity, in my mind, a technical person is a person that knows how to hold a screwdriver properly, a skill I never possessed, so I looked at the Internet and called a person whose profession is to do that, we set a meeting at 6PM, and I went out in the middle of a meeting to go there, at 6:10PM I called him, he said: the person is on his way, will be there within 10-15 minutes, I waited until 6:30, and my patient started to run out, I called again -- the answer: he is almost there, give him couple of minutes. I called again in 6:40, got the same answer. My last call was in 6:48, saying -- I am cancelling this order, and will try to find another one that when he says 6:00 he really means it. Today I called another guy from the Internet, and again set a meeting with him at 6:00PM, this time I called him before leaving home, and he said -- I'll be there in 30 minutes, and indeed, he was there 25 minutes later, and changed the lock. I told him that I am happy to see that there are some professional people who arrive on time, and he said that he as also a customer hates service people who are late... I always thought on punctuality as a value, unfortunately, I am (almost) the only one in my family who has a sense of time.
Anyway, I got today some spreadsheet that an analyst made to evaluate various event processing products, the good thing is that people are trying to devise such criteria, however, looking at the criteria there, my observation is that the people devised this list came with a long laundry list of criteria, many of them seem to me irrelevant for many of the applications I know, while others that I would imagine that will be there, are left out, or treated in a shallow way. This gets me back to the difficulty of devising performance benchmarks in this area.

My observation is that the event processing area is not monolithic; people are using event processing for different reasons, and have different functional and non-functional requirements and priorities. Trying to think in a monolithic way, may yield a result that is not valid for anybody. I think that there should be deeper research in this area and provide criteria for classes of applications; this classification is not easy, as the diversity of applications that each individual organization or vendor has, is limited. This is one of the things that require a community effort, and I hope that the EPTS use case working group will be able to provide it, so I am looking forward to hear their tutorial in DEBS 2009. This also relates to our goal of establishing event processing manifesto, which will be the main theme of the event processing Dagstuhl seminar in May 2010. Stay tuned for more.

On the first step in the way to "event processing manitfesto"

It was a very busy week and alas I had to neglect the blogging hobby, now it is Friday night, I am watching a TV program with old Hebrew songs (my favorite), and decided it is a good time to blog a bit, however, our relatively new cat, who looks somewhat like this (this is not his picture, but of a similar cat I've found on the web) decided that I am a good place to rest on, and did not want to move, another creature who is trying to manage me... He is really a kitten that my daughters found and adopted, and as I have written before, giving names in our family is not an easy task, so he has several names and is known by "the cat". I call him Gilgamesh the terrible.

In 2007 we had the first Dagstuhl seminar on event processing, and we the same set of organizers (Mani Chandy, Rainer von Ammon and myself) decided to apply again for a second Dagsthul seminar in 2010, and the seminar has passed the committee, with some clarifications that we need to provide about scope. I'll let you know if and when it will be finally approved.

The intention of this Dagstuhl seminar (that lasts for 4.5 days) is to have an opportunity for a selected group of people to have a meeting in an isolated place to have in-depth discussions. The proposed goal of this Dagstuhl Seminar is to work on "event processing manifesto". There has been several manifestos of different area in the past, for example: OODB manifesto, Hopefully, by the time of the Dagstuhl seminar we'll have advanced work done by the various EPTS working groups that are being launched this month, and we'll be able to utilize their results in order to better define what "event processing" is -- note that I don't use "complex event processing", and I explained the reasons before.





One of the questions asked is what is the scope of "event processing", since working with events is quite wide area - starting from interrupt handling in operating systems, moving through graphical programming and more -- much of this is related to programming with events in conventional programming, and there are even books dealing with this area. However, our scope is more modest: generic tools for processing events in IT systems. This scope talks on what is needed to build a generic tool, and not ad-hoc programming hard-coded for every single application, and IT systems and not operating system, embedded systems etc..

The illustration above is a first step in thinking about -- what event processing system should include -- parts of it should be mandatory and some optional, however from functionality point of view there are:

• Routing and filtering -- the most basic form of event processing.
• Mediation -- transformation, enrichment, aggregation, split -- the next level of sophistication.
• Pattern Matching --- (I called it in the past "pattern detection") which may involve multiple events from multiple types.

On the bottom of the illustration there are two other entities:

Event processing platforms which are enablers for scalability, distribution and other good qualities. Event processing platforms may have their own functions or host others (or both)...

Pattern discovery that falls under the category "Intelligent Event Processing". It can be done off-line (typically this is the case) or on-line - and then the pattern matching may be unified with the discovery.

In different types of applications we may need different subsets, for example: fraud detection requires pattern discovery, security type detections (e.g. denial-of-service attack or intrusion) may use on-line pattern detection. On the other hand, other applications don't require pattern discovery at all, for example: compliance with regulations, where the regulations are given and cannot be discovered, or BAM systems in which the Key Performance Indicatros are determined according to the corportate strategy and cannot be discovered. Furthermore, there are applications in which pattern matching is not required at all, and all processing is of type filtering, routing, enrichment and aggregation.

And I'll finish with a footnote to David Luckham's recent article. David is trying to answer "critisizm on the Blogsphere" about CEP as a marketing hype, and lack of value from the current set of products. First, I never thought that there is over-hype, on the contrary, relative to the potential of event processing there is under-hype. I am re-posting this illustration taken from Brenda Michelson panel presentation in the last EPTS annual symposium.


The hype is relatively low, and in contrast, the analysts report are all indicating that the EP market has grown by 50% or so in 2008, and IDC even claims that for a second year in a raw that is the fastest growing middleware type. About the Blogsphere crtisizm, as I already written before, much of it stems from diferent interpretations of the term "complex event processing", for example, some of the postings of Tim Bass lead me the conlusion that he believes in the equation : complex event processing = on-line pattern discovery. Again, eliminating the quantification "complex", there is a large set of applications (probably most of the applications I know) of event procssing, do not require stochastic reasoning at all.

I'll post a continuation Blogs about application types, and functions they require.. It is very late - going to sleep.

On event processing as a paradigm shift

The readers are probably familiar with this picture where it shifts between seeing two faces facing each other (in black) and a white vase. I came across a (relatively) new blogger in this area, Pern Walker, blogging for Oracle's "event driven architecture". The title of the posting is:

Event servers, a disruptive technology. It describes the components of the (former) BEA framework, nothing new here, but the interesting part is the conclusion - event processing COTS is a disruptive technology, it displaces custom code in event processing, since it is more cost-effective.

This reminds me of a discussion we had in May 2007 in the Dagstuhl seminar on event processing, it was a night discussion with wine, and was lead by Roy Schulte, the question that Roy has posed to the participants : "Will Event Processing (EDA) become a paradigm shift in the next few years or not?”.

Today, I don't intend to answer this question, instead I'll post part of the discussion in Dagstuhl that included observations about "paradigm shifts" (thanks to my colleague, Peter Niblett, who documented the entire Dagstuhl seminar). I'll return to this topic again, with my (and maybe other) opinions about the answer, after the EPTS event processing symposium

Observations (from the Dagstuhl discussion):

• Paradigm shifts can’t happen if there are too many barriers; have the entry barriers for "event processing" already been removed? ;
• Paradigm shifts are more likely to happen when adopters decide they need a whole new avenue of applications; they are less likely to happen as a way of re-engineering existing systems. For example the German population will reach 1:2 old: young ratio by 2020 so this requires a paradigm shift of healthcare models. Can we identify new avenues of relevant applications?
• Paradigm shifts usually happen as a result of some external change, not just because of innate strengths of the technology itself. Can we identify such external changes?
• Standardization is not necessary for a paradigm shift, but good, appropriate standards (de facto or otherwise) certainly help

Another question is to where in essence is the "paradigm shift" - is it the decoupled "event-driven" paradigm ? is it the "complex event processing", i.e. ability to find patterns on multiple events? is it the entire processing framework as the Oracle's Blog claim?

More - Later

On Event Processing Research Challenges

This picture has been taken a year ago on the stairs to the old church in Schloss Dagstuhl, in that meeting we had people both from academia and from industry discussing the state of the art and future of event processing. I have returned to the conclusion of the Dagstuhl Seminar done by my colleague Peter Niblett from IBM recently, after my return to the IBM Research Division from spending several years in the product organization, I was asked about challenges to the research community, as seen by the product development community (or the industry in general), since we had a session about it in Dagstuhl, in which people from the industry expressed their opinions about the same questions, I just had to take it as a basis of a presentation in this area.

In Dagstuhl four major areas has been put as challenge to the Research community (adding my own interpretation and comments)

1. Event Processing Algebra and Meta-Language: Like the database area in the pre-relational era, the first (and probably second) generations of event processing are "engineering based", various vendors are building implementations based on the use cases they see and their innovative ideas, bringing to the table, not only a variety of languages, but also a variety of (typically - implicit) conceptual models behind this implementation. One of the indication for a maturity of an area is the existence of an agreed upon conceptual model. The relational model has done it for databases, the browser concept has done it for the internet, now we need the same for event processing. David Luckham's concept of EPN/EPA is a possible starting point, but more work is needed on this - the challenge is still there; after constructing the "relational model" for event processing we also need the "SQL" (not necessarily SQL extension - but this is a possibility) for it - in term of meta-language that describes the model and can be mapped to various implementation.

2. Software Engineering issues: This is a challenge from a different perspective. Event processing and Event Driven Architectures impose different thinking about computing. We are programmed to think in a certain way in all programming language, and the event based programming is somewhat different paradigm. Even the basic principle of decoupled asynchronous processing is something that is not easy to digest f0r people. We need software engineering tools, methodologies and best practices.

3. Implementation optimization: If we have done the analog of SQL, another analog come to mind - database tuning and query optimization. There are many optimization issues especially when the event processing applications are distributed and may have various QOS requirements - parallel processing, acceleration of hardware, inter-operability, support in illites in general - all are challenges. The goal function in this optimization is not unique, while in some systems it is maximal throughput, in other it may be scalability in number of rules/queries/patterns (there are applications in which the throughput is measured in millions, and others in which the number of rules, the number of producers, or the number of consumers is measured in millions...), the goal function can be a combination of multiple criteria.

4. Variety of "small issues" - examples: uncertain events, out-of-order events, retention and vacuuming of events etc....

It will be interested to look at recent research project to see how much progress has been done on these, and if the list should be modified after a year that has passed. The major research event of the EP community this year is DEBS 2008 , the program has not been published yet, and it will be interesting to compare the accepted papers with this list (I'll do it when surveying the conference itself in July). We also intend to do in the EPTS event processing symposium in September a session with academic people about it - stay tuned for further notice. I am also thinking about a follow-up Dagstuhl Seminar, may be in 2009 or 2010 - to re-discuss event processing in perspective. More - later.