On two years of the event processing thinking blog

These three pictures are taken from the posting : On unicorn, professor and infant, which is, by far, the most read posting on this Blog. Today is August 28, exactly two years since I started my first posting - this is posting number 314 (PI * 100)

• Event processing technology is still a "child" (maybe no more infant), in the the life-cycle of technologies. It is moving to the youth stage, but have not achieved maturity. It still has challenges in standardization, interoperability with the rest of the universe, engineering and optimization issues, and consumability.
  
• Event processing is not a unicorn, it is not mythological but real, and it has more than one horn, it is not strongly associated with a single application, as some people tended to believe in the past. Each technology has its early adopters, but we see now that the utilization of event processing is pervasive for each industry, no less than technologies like databases, web services and workflows.
  
• The blind people touching an elephant and describing it in a different way is still an excellent metaphor. People have come from multiple perspectives and see the world in a different ways. Our challenge is to get a good understanding of the entire elephant, and overcome this syndrome. This is the mission of the Dagstuhl seminar in 2010.
  

While the unicorn and infant posting was by far the most popular - other popular postings were some postings from 2007 entitled: On event stream processing in which I have written first time that the ESP term has virtually died, and Agnon, the dog, playing and downplaying,
which responded to the claim that I am downplaying the role of rules. Among the 2009 postings the most popular was the one about the event processing language analysis workgroup,


Some more current statistics.

• The number of distinct readers is around 35,000 over these two years, however, many of them are just one-time browsers, the loyal readers, which consist of those who entered the site at least 101 times, their number is around 2,000, if we'll add more than 1,000 more who entered the site between 51-100 times, and another 1,000 who visited between 26-50 times, I would say that there are around 2,000 loyal readers, and around 2,000 more occasional readers - not big numbers in the Internet scale, but fair enough for a Blog focused on a narrow area with a developing community.
  
• The countries with most readers are: USA (still more than 50% of the readers), UK, Germany, Canada, France, India, Australia, Holland, Sweden and Spain. Israel and Japan have many entries, but less readers, which say concentration of loyal readers.
• The biggest referring source, no surprise is Google, other big reference sources are: the TIBCO blog, David Luckham's CEP site, Marco's rulecore Blog, and ebizQ.net.
  

The Blog has been cited by the CEP Knol and has become a major reason for the fact that I was approached by a publisher to write a book.

The other Blogs of the community are either company's Blog - like the TIBCO, Aleri, Streambase and Apama Blogs which express the company's view; or those of independent people or professional bloggers. While I work for IBM, I have never made it as an IBM marketing Blog, but a personal one. I have never been asked by anybody in IBM to write or not to write about anything, and although as IBM employee I do have some restrictions, I have taken upon myself more restrictions as the EPTS chair, among them the restriction of not to write evaluations about products; I am leaving this task to others; the only time that I was asked to revise a posting was fairly recently, when I cited the Gartner Enterprise Architecture Hype Cycle report, where I was asked by Gartner to make clear distinction between what I am citing from Gartner, and what are my own opinions, so I have revised it and restructured the posting in a way the distinction between them will be clear.

Sometimes new topics emerge, and sometime topics are repeating discussions, but never a dull moment in Blogland. Will return with a professional posting s00n.

On downloadables

Among all topics that I mentioned in my previous "catching up" posting, I chose to talk about downloadables. The Forrester report mentioned "free downloadable" as a criterion, and one of the authors defended this criterion in a short article. Paul Vincent has written about his impressions from the summer school he participated in that some of the academic people like to use open source, so they can play with the implementation code.

There are two issues here: whether downloadable is a criterion in product evaluation, and whether for academic reasons one should use open source or commercial product downloadable ?

As for the first question: the Forrester report stated that event processing is moving from being niche technology to being part of mainstream computing. An enterprise can chose in using a specific product, either because it is being sold through some solution / application, and in this case the line of business owning the application, may select this application and get the product behind it as a by-product, this is exactly niche technology kind of behavior. Getting it as part of mainstream computing, mealing that this product is being used in enterprise architecture, and being endorsed by the IT side of the enterprise. I have spent 8 years in my life in an IT shop of a big enterprise, and from this angle, the world seems somewhat different, decisions (if not constrained by force majors) are being determined after the IT developers have studied the products from all angles. Big enterprises don't typically have problems to get any software for free evaluaton; however, where free downloadable may help is for people who want to explore products by their own initiative, whether they are developers in organizations that have not decided (yet) that they want to do formal evaluation, or by students, who are the future generation of developers. Investment in students is considered a good investment and thus many vendors provide free software for academic purposes. As I have written in the past, the publisher of the EPIA book we are writing is very keen on providing downloadable software for the reader. Since we told him that there are multiple approaches and we don't want to restrict the reader to a single approach, we have approached several vendors, who agreed to: a). provide free downloadable; b). provide a solution of (at least a subset) of the "Fast Flower Delivery" example that accompanies this book. Currently we have several vendors that already agreed to work with us on this: Aleri (representing the Stream programming style), Rulecore (representing the ECA rules programming style), Apama (representing the script programming style). We also got agreement from Esper (which is already an open source) to provide the solution of the case study in ESPER. I hope that other vendors will follow. I am sure that the readers will be benefited by this exercise (we'll put all solutions in an appendix to the book, and point at the soft copy of the solution, and downloadable from the book's website).

Now, to the question of downloadable vs. open source in teaching. I am giving students two types of programming assignments (depends on the course) -- one is to write some application using a product, the other is to write a subset of an event processing engine.

For the first type of assignment, both product downloadable and open source can be used. In the database area, I guess that the majority of the basic database courses are using My SQL, however, in the event processing area we don't have a standard language yet, and typically products are more advanced than open source.

For the second part -- writing a subset of an event processing engine -- one can take an open source as a basis and extend it, which may be one approach (common in operating systems course), my personal preference is to let students build such system from scratch, the reason is that again, due to lack of standard, the use of open source will restrict the student to use the approach taken by that open source, alas, the approach in which I am teaching event processing is somewhat different. I also think that students have to think about how to implement concepts rather than to ruminant on other food. More on teaching considerations -- later as I am prepared to teach an event processing course twice in the two semesters for two different populations, in the Information Systems Engineering program at the Technion, where I'll ask the students to use products, and in the Computer Science program, where I'll ask the students to write a mini-product.

A comment: I am moderating comments after some spam attack in the past, recently I am getting everyday a comment to this Blog in a charachter set which may be Chinese or Japanese.
I wanted to respond: לא הבנתי מה רצית ממני, but as this Blog is being handled in English, I will not post comments in other languages, I don't even know whether they are spam or real comments.
I also don't post comments from people who don't identify themselves.

Last but not least, a reminder: The EPTS 5th event processing symposium is getting closer, if you wish to participate, and have not registered yet, please do it on the conference site, the reason we need this registration ASAP is that there are quite a few of requests to register from non EPTS members, and since we limit the number of participants, we want to make sure that we are within these limits. BTW - the program on the site is not up-to-date. The updated program will hopefully be published next week.

ON catching up

This picture is another picture from Finland, this time from the national park Helvetinjärvi in the Tampere area (more or less), but time to move on and return to work, which I am doing rather slowly (apologies to all Emails I have not answered yet, will catch up by the weekend).
A lot of catching up in work, but now I took a small break, drink bitter coffee, and had a quick scan on some Blogs, some deserve detailed responses, but in a catching up mode I'll only provide highlights:

• Marco has written about patterns, there are two comments here - one a terminology comment, he seems to be confused by the distinction between complex event, composite event, situation, event correlation (a term I don't really use) and pattern. These terms came from different sources, and they are not exactly synonyms, I have tried in the past to look at their definitions and outline the relationships among them, the interesting question is do we want to stay with all legacy terms or settle in a fewer terms. This is a question for the community, and for the glossary work group.
• Marco also started to look at list of patterns. I'll refer to it in one of the next postings, I am now working on the chapter in the EPIA book about patterns (chapter 9). We also are looking at it within the language analysis work group.
• Paul Vincent had a few interesting postings, some of them summarize iimpressions from the summer school in Norway in which there seems to have been discussion among different approaches.
• Paul Vincent also had another posting. getting back to the CEP vs. ESP issue. I've already made a short comment to this postings, that I think that this label discussion is somewhat anachronistic, since the term ESP is virtually gone away. Besides the labels, there are indeed different approaches to approach event processing applications, and I think that we'll see more of hybrid approaches. This also deserves a longer discussion.
  
• Paul Vincent also wondered about downloadable as criterion, following the Forrester report, that has put it as one of its 114 criteria to compare EP products. John Rymer from Forrester has responded in ComputerWorld UK. I would like to have further discussion on the Forrester criteria in general (within the catching up I also read the full spreadsheet of comparison, and have some interesting insights), also I need to follow up on the downloadable issue. Not sure if it is a criterion about product quality, but IMHO, free downloadables are important on getting event processing from niche technology to the mainstream, which was the major theme of the Forrester report, this issue of niche vs. mainstream, also deserves a discussion by its own right.
• Hand Gilde has written about the Gartner Hype cycle report. He was not happy from the way that Gartner articulated the value of event processing, and sent the challenge to EPTS to make better articulation of the value of event processing to the general community. I agree that EPTS needs to catch the ball here, and will propose to imitate such activity. As always, I am throwing back the ball to anybody that throws it towards me, I think that Hans is an excellent candidate to be one of the key persons in such an activity, and I call him to quit his bystander position and jump into the cold water.
  
• Mani Chandy who uses his recent Blogs to post articles, has written about "Smarter communities". This also deserves a longer discussion, I agree that we are heading this way in variety of areas that IBM labels as the "smarter world".
  
• Some other references to the Forrester report which seems to catch the attention and kept (almost) everybody relatively happy. I still need to discuss some of the observations from this report. While looking at the detailed spreadsheet, I have did a quick summary of the number of customers reported by the different products. The number looks impressive, I wonder how many of these are currently running systems in production. But I'll let the use case work group chase them to obtain use cases, and ask everybody to help on this -- another topic to discuss.

Well - enough for now, back to the long list of to-dos...

Some insights from the vacation in Finland

This is a picture of the cottage in Finland, where we spent the last week. The cottage is totally isolated, in a small Forrest, near a lake. In the previous week we spent in another cottage, also in Finland. This morning, I have returned home (and tomorrow return to work), and here are some fresh insights from Finland:

• There are almost 200,000 lakes in Finland; comparing to Israel, in which we have two (2) lakes, that we call each of them "sea", the dead sea is one of them.
• The local drivers are surprisingly obeying speed limitations much more than any other country I have seen so far.
• Sauna is an important part of the picture, we visited a place that has been prison until 16 years ago, and found out that even prisoners had right to use the sauna once a week; I tried using the sauna once, but I never liked sweating.
• The Finnish language is not similar to other Nordic languages, it sounds to me like Hungarian spoken in Italian accent, but this may be my own imagination.
• The summer in Finland ends roughly in August 10, we discovered that some of the attractions are already closed, since the summer (i.e. the summer vacation for schools) has gone. We succeed to visit the stone age village in Saarijarvi, thanks to the lady from the Saarijarvi tourist information that coordinated to open this site especially for us.
• I have used GPS in order to navigate, it brought us to the most deserted places, however, we found out that one needs to use GPS in Finland (and elsewhere) with a grain of salt, and when it directs you to a route that does not seem to you the intuitive way, prefer your own intuition.
• The most common "fast food" in Finland is Pizza.
  
• Overall -- great place for peace and quiet and nature walks, good for disconnect from civilization.

And one comment on the flights back -- we came back from Finland to Israel in a combination of a Finnair and ELAL (the Finnish and Israeli airlines). Both Finland and Israel see themselves as high tech super-powers, however, the software in their airline is not impressing. The seat assignment software of Finnair, has problem to understand that people who travel together want to sit next to each other, and around me several people switched places to achieve sitting together. It seem that ELAL has not heard about modern development in distributed computing, fault tolerance and some other trends. We checked in relatively early and it was OK, however, at some point, the centralized server in Israel crashed, and they could not bring it up, and it seems that the remote station was completely helpless, they ended up making manual seat assignment, and the flight was delayed for almost an hour. Back to professional Blogging - later this week.

On the criteria to evaluate event processing products

This is a map of Finland, the location of our family vacation for this year. The vacation is planned to start in Saturday, and I'll be disconnected from the cyberspace for 15 days. Working late at night to advance in the EPIA book we are writing.

It seems that this is the time of the year of analysts report, the community blogland was full of references to the Forrester wave report - Complex event processing platform, Q3 2009, dated August 4, 2009.

I will not comment about the grades that they gave the different products, the reason for that is that I decided that in my role as chair of EPTS, I see my role to work on the coop side of the coopetition, and leave the competition side of the coopetition to others. I think that the main competition is not between the different vendors (though they have point competitions), but against the barriers that the event processing area has to fulfill its potential and become a pervasive main-stream technology.

The Forrester reports starts the executive summary by saying: "Forrester evaluated nine complex event processing (CEP) platforms using 114 criteria".

Without getting into the long list of criteria (not part of the report itself, but I managed to look at it), I have some doubts about the ability to get a meaningful information to customers by weighing 114 criteria. There are two reasons, one is practical and one is methodological.

On the methodology side, the compensatory model of decision making advocates weighing of many criteria, however, experience shows that the actual decision making model is lexicographic, meaning ordering the criteria according to importance, and making the decisions according to the most important criteria. People may use a compensatory model of weighing a lot of criteria, if their organization require them to work this way, but this is done only as justification to decision that has already been made by the lexicographic model.

Let's move from decision making theory to the event processing universe. The event processing universe is diversified from both functional and non-functional requirements point of view. I really don't believe in a "one size fits all" in anything related to this area, and this goes also for set of evaluation criteria. Getting criteria that are good in variety of cases and weighing them together may not get a good solution to any particular case. The more practical approach is to set a collection of relatively small sets of important criteria, and also segment the space of application, and assign a set of criteria to each segment. Anybody that will manage to do it, will help customers more to make the best decision for a particular case. I hope that EPTS, through its use cases workgroup will be able to provide this segmentation, and this will be the starting point for analysts to come with the right criteria for each segment.

On the Gartner 2009 application architecture hype cycle

Here is a revised version of my Blog entry that relates to the Gartner Application architecture hype cycle report (Gartner Report ID number G00168300 from July 16,2009) , the revision was done at the request of Gartner who asked that I'll make exact citations in their report, and make clear distinction between what is quoted from the Gartner report, and my own remarks.

Here are a collection of citations from the report that are of interest from the Event Processing perspective:

1. "Event-driven architecture (EDA) is an architectural style in which a component (or several components) in a software system executes in response to receiving one or more event notifications". In the report EDA is positioned under the hype cycle phase "Climbing the slope of enlightenment" which according to Gartner's terminology is defined as " Focused experimentation and solid hard work by an increasingly diverse range of organizations lead to a true understanding of the technology's applicability, risks and benefits. Commercial off-the-shelf methodologies and tools ease the development process"
2. CEP is positioned under the hype cycle phase of "Technology Trigger" which according to Gartner's terminology is defined as "A breakthrough, public demonstration, product launch or other event generates significant press and industry interest", and is the phase that precedes the "peak of inflated expectations" phase.
3. For CEP: "market penetration is 1% to 5% of target audience"
4. CEP use is expected to grow at approximately 25% per year from 2009 to 2014, but the use of COTS CEP products is expected to grow more than 40% per year in this time frame
5. For CEP COTS products: " Most of these products are immature and incomplete"
6. "Most business analysts do not know how to identify business situations that could be addressed through CEP, and that is limiting the rate at which CEP use can expand. Most software engineers are not familiar with CEP development"
7. "The Event Processing Technical Society (EPTS) was launched in June 2008, and it is expected to facilitate the adoption of CEP".
   

Here are my own comments:

• Note that EDA and CEP are positioned in different phases of the hype cycle.
• The fact that the market penetration is low indicates that there is still a substantial growth potential, if we can overcome the adoption challenges
• The adoption challenges consist of product maturity and market awareness. We are now still in the first generation of products in this area and maturity is typically achieved in later generation. Awareness and understanding of value and positioning are indeed a challenge.
• EPTS indeed has been formed to facilitate the adoption of the event processing area. Both challenges mentioned here – advancing the state of the art to accelerate the next generations, and educate the general community about the value and positioning of event processing within the enterprise computing.
  

Concentrating on the important things with event processing

I would like to welcome to the Blogland - Mani Chandy, seen in this picture, one of the pioneers of the event processing area, the person who came to the first event processing symposium in 2006 to tell us what are the obstacles in the way to success in this area, and my partner in organizing the event processing Dagstuhl seminar in 2007, and the second one in 2010.
Mani has recently started a Blog named: Smart systems that sense and respond, so be sure to bookmark it. Mani writes his Blog as mini-papers, including references.

Inspired by the last posting on Mani's Blog I would like to write about a role of event processing that is sometimes overlooked -- not generating more information, but generating less information, or more precisely focus our attention on the right information.




Mani says in this posting that a human attention is the world's scarcest resource, thus focusing the attention on the right stuff has a very strong impact on what's happening. This is true in many senses, I think that in one of the many management courses I took over the years the instructor said something like --- you can do around 20% of what you plan to do, however the difference between success and failure is whether you can identify the right 20%. This is true in many areas of life, a smart student knows what is important to study in depth before the final exam, a smart physician knows which of the symptoms is important, a smart reporter knows how to identify the news item that will take him to fame, and there are many other examples.

Event processing is the basis for smart systems that can detect when it is important to attract the patient attention, in this case the role of the software is not to create more events, but to highlight existing ones, though the highlighted ones may be derived events and not raw events...

So -- focus your attention to other smart articles in the smart systems Blog. More - later.