Almost midnight, after a long day that started in 8AM and did not end until now.... some days are like that, tomorrow might be a slightly shorter work day. Before retiring to sleep, I'll write something on this Blog as a kind of doing something else...
I have written before about situations, however, I am not sure that this term is still well understood (the same applies for context, but I'll leave context for another posting). In the EPIA book, when revising the book, we decided that the term situation is a very basic one in event processing, and put it in the introduction chapter. Situation can be thought of as event that requires reaction, but this is somewhat simplistic definition, since indeed situation is an event, but it is something that is considered as event in the user's domain, and not necessarily an event in the physical domain. Sometimes we wish to react to a raw event, sometimes we can say that a derived event represent the circumstances which we want to react, and sometimes a derived event is just approximation for the situation.
Let's look at a couple of example.
Example 1: The situation we would like to react is that a driver crosses toll both without paying -- In Israel, as high-tech country, in the only toll highway, we have camera that takes pictures of the license plate and send the bill to the driver (unless one is subscriber and then it has some identification device in the car), thus we don't really have toll booths, but in the USA it is quite pervasive. Anyway -- this situation can occur if a driver moved through "EZPASS" lane without having a device, or somehow sneaks without paying. Assuming that there are cameras that capture it, then the derived event, which is a disjunction pattern of these two events, is a complete match for detecting the situation.
However, life is not that simple, and we move to example 2. This example, that I used before in different opportunities, is that in a call center we are looking for frustrated customers in order to send a friendly customer relations officer to call them and ease their frustration.
Sometimes the person who handles request can detect the frustration, by the tone of the voice, if it is a phone call, or style of writing, if Email, but sometimes not, so what we can do is based on past experience, devise criteria for who is frustrated customer, and this may be -- a customer that sent three requests on the same topic during a single day. This criterion may be an approximation for the "frustrated customer" situation, but not an exact match, thus we can get false positives or false negatives. Handling such approximation cases is part of inexact event processing, and this seems to be one of the issues that will be part of the next generation of event processing languages and models. More - Later.
In the last few days I spent part of my time in the NGITS 2009 conference, that was hosted by the IBM Haifa Research Lab, today the keynote speaker was Alon Halevy, whose picture you can see above. Alon gave an excellent talk about "searching the structured Web". As a database person that joined Google 4 years ago he discovered that many of the things he knew from databases were not valid in Google. and from data management point of view, he went several steps forward, and his challenge was how to process structured data that is hidden in the "deep Web" behind web forms and interfaces. This reminded me one of my colleagues in the Israel Air force, who spent a year of his life more or less (almost around the clock) to write a proprietary database system based on low level I/O operations, for some logistics process (I think it was automatic warehouse), he wrote his version for conncurecny control, query capability, recovery, and many other stuff. He thought at that time that the DBMS products available in the market (this was 1978 I think) are not good enough, and using them will be a step backwards relative to what he had in mind. I have written in the past about single application vs. more general one in the context of why network and system management guys have not developed more general event processing products.
I was asked several times what is really the main issue behind all the work I am doing (along with many others) about "event processing" as a discipline, what is the new thing -- people have processed events forever. People also processed data long before DBMS has been introduced, but the way that my old colleague worked was not scalable. Not many people had the skills to do it, and it was not that cost-effective way.
Likewise, there were and still are many event processing applications of all types, colors and sizes, that are developed in an ad-hoc fashion, as there are still applications that process data that do not use DBMS products, because some aspect does not make it feasible or cost-effective.
However, it is clear that the DBMS area has made a tremendous contribution to the IT and business in the last few decades.
My own goal around event processing is to make event processing pervasive as part of enterprise computing, this will be achieved by generic software. Many of the database issues have been developed due the need for genericity -- take query optimization as an example, if one writes query by hand, it is not needed, since every developer can write optimized ad-hoc queries, the requirement to do query optimization came from the fact that a generic query language is being used for many purposes.
Personally, my interest is not in building "complex systems" (as the readers of my Blog know, I tend not to use the "CEP" acronym, due to the ambiguity of the term "complex") or one of kind systems. I think that the generic event processing systems will enrich their functionality over time, my interest is to make it pervasive. The first generation of products went some steps in this direction, and the next generations will do more. I have presented several times in various places my view about the next generation of event processing which refer to the challenges in the way to do it properly.
This topic will be discussed later this year in the 5th EPTS event processing symposium and the event processing Dagstuhl seminar planned for May 2010.
I was asked by several people to post the presentation I have given recently in several places - the presentation including some views about where should we go in the next generation of event processing, what are the challenges in the various areas (the above illustration is a slide classifying the challenge areas) and a survey of some activities we are doing either in IBM Haifa Research Lab or with graduate students at the Technion. The presentation is now on slideshare;
enjoy.
This is a picture of the Bahai Gardens, one of the famous sites in my home-city Haifa, the Bahai religion is an interesting one, more modern than most religions (I myself am agnostic and do not practice any religion, so this is not meant to be an endorsement), the Bahai people see Haifa as one of their holy sites and invest a lot in the city, this week they have a major celebration. I have returned to Haifa after my short trip abroad, in which I have given four times the same talk about "event processing - the next generation" (I'll post in on the web soon). One of the discussion points have been what can the event processing discipline (complex or not) learn from other disciplines that succeeded. Coming from a database background, it is always interesting to me to make a comparison there. When relational databases started to become products in the early 1980-ies, I have been a database practitioner with experience in several DBMS products, and in the beginning I looked at the relational model without much respect, it seemed to me to be over-simplification that gives up semantics and creates a lot of anomalies. However, the simplicity has been the main benefit. The relational model has won, and also created a big research community around it that concentrated forces around a single model and developed query optimizations, more semantic abstractions on top of it and some other stuff. The fact that there has been a substantial brain power dedicated towards a single direction was a contribution to success. The fact that was not a critical mass of work around object-based databases contributed to the fact that its success has been modest. What can we learn from that in the "event processing" discipline ? We need to strive to find the formal model that will be the basis for concentrating the community around. The model is not extension to the relational model, since this extension will loose the main benefit of the relational model -- simplicity. There are several relational algebras around, but all of them do not meet the simplicity criterion, on the contrary -- they are quite complex. So it is still a major challenge for the community. More on that and possible directions -- in subsequent postings.
Today, Friday, is part of our weekend, so it is a good time to do shopping and other arrangements.
My wife and myself went to our local friendly bank to open some new account for some purpose. The lady that handles our account said that they have a new software to open an account that is extremely difficult to operate, with a lot of screens that one has to understand what is asked, and suggested she'll do it off-line and call us when ready, so we'll come to sign the papers. Once, opening an account was simple and lasted a few minutes, just signing some forms; the more sophisticated a software becomes, the more difficult it to operate, and sometimes it becomes obstacle to the business. Often, developers don't really care about the human engineering aspects. Hans Gilde wrote recently about the fact that CEP software is not smart. I agree, in several occasions I have given talks to an audience of high-school students which gives a rough introduction to AI, under the title: can a computer think ? while there some works in AI that strive to do it, today's software does cannot really think, and is not really smart. One can use the software to do things that look smart, but the wisdom is not in the software itself, it is in the way it is used. In the bank case, the software does not even look smart...
This week I had three visitors from Germany, Rainer von Ammon and two of his CITT colleagues, and we made some progress towards defining the EDBPM project that we plan to submit as EU project. They have asked me to pose in my office under my " wall of plaques" (half of them are in Hebrew, so they could not really read them...). So this is my most current picture..
One short clarification -- after my posting entitled : "event processing platforms - yes, but..."
I received some private communication claiming that there is a confusion between the terms "platforms" and "engines". The claim is that there are vendors who refer to their engines as platforms, moreover, some people refer to any run-time software as an engine. So I thought it worth clarifying how do I see the distinction:
- Event Processing Platform is a software that enables the creation of event processing network, handle the routing of events among agents, management, and other common infrastructure issues.
- Event Processing Engine is a software that enables the creation of the actual function - in the EPN term implementing agents.
This is similar to the difference between an application server and a single component.
What is the connection ---
- On one extreme, there are closed platforms, i.e. platform that can run only one type of engines, in this case the distinction becomes more fuzzy.
- On the other extreme -- there are open platforms, in this case these concepts are totally separated, a platform that can run multiple engines. The main issue about it is that there may be a collection of different languages that come with the different engines, and this may make the development of an application more difficult.
The first generation of event processing has started with engines that are stand-alone, the emergence of platforms, and making them open, are the signs of the second generation. I'll say more about the challenges of constructing the next generations -- more later.
David Luckham posted on the complexevents site the question: Is there a commercial need for quantum leap in CEP products. David has also a continuation article that discusses event hierarchy abstractions as a quantum leap possibility. Before answering David's question let's me make some observations. Early in my career, 33 years ago, I have been a programmer in the Israeli Air-Force (the Air-Force was in opinion that letting me flying aircraft's will be too dangerous for the public safety...) and we have been early adopters of IMS, which has been relatively new, had severe performance issues, and needed a lot of manual tuning, and did not really work as advertised. IMS was actually a second generation database, it has a predecessor called DL/I and still used the DL/I language. IMS was a huge improvement over file systems that we have used before in level of abstractions, concurrency control, and many other utilities, yet it had many issues that have been resolved over time. The relational databases are actually the third generation of database systems and it also had a rough childhood, until query optimizations have been matured; The relational database has been a disruptive technology, and also had its own childhood problems, until query optimization have been better understood.
Back to event processing --- I assume that event processing products of 2019 will be totally different from those of 2009, the questions are:
- Is there going to be a "disruptive technology" as the relational database has been in the database area, OR "just" gradual evolution will occur.
- What will drive the progress to next generations ?
What will trigger the next generations ?
- Customer requirements that require substantial change
- Competitive pressure
- Disruptive technology
- (more reasons) ?
I'll leave these questions for now as a food for thought and will discuss them in subsequent postings.
