On two years of the event processing thinking blog

These three pictures are taken from the posting : On unicorn, professor and infant, which is, by far, the most read posting on this Blog. Today is August 28, exactly two years since I started my first posting - this is posting number 314 (PI * 100)

• Event processing technology is still a "child" (maybe no more infant), in the the life-cycle of technologies. It is moving to the youth stage, but have not achieved maturity. It still has challenges in standardization, interoperability with the rest of the universe, engineering and optimization issues, and consumability.
  
• Event processing is not a unicorn, it is not mythological but real, and it has more than one horn, it is not strongly associated with a single application, as some people tended to believe in the past. Each technology has its early adopters, but we see now that the utilization of event processing is pervasive for each industry, no less than technologies like databases, web services and workflows.
  
• The blind people touching an elephant and describing it in a different way is still an excellent metaphor. People have come from multiple perspectives and see the world in a different ways. Our challenge is to get a good understanding of the entire elephant, and overcome this syndrome. This is the mission of the Dagstuhl seminar in 2010.
  

While the unicorn and infant posting was by far the most popular - other popular postings were some postings from 2007 entitled: On event stream processing in which I have written first time that the ESP term has virtually died, and Agnon, the dog, playing and downplaying,
which responded to the claim that I am downplaying the role of rules. Among the 2009 postings the most popular was the one about the event processing language analysis workgroup,


Some more current statistics.

• The number of distinct readers is around 35,000 over these two years, however, many of them are just one-time browsers, the loyal readers, which consist of those who entered the site at least 101 times, their number is around 2,000, if we'll add more than 1,000 more who entered the site between 51-100 times, and another 1,000 who visited between 26-50 times, I would say that there are around 2,000 loyal readers, and around 2,000 more occasional readers - not big numbers in the Internet scale, but fair enough for a Blog focused on a narrow area with a developing community.
  
• The countries with most readers are: USA (still more than 50% of the readers), UK, Germany, Canada, France, India, Australia, Holland, Sweden and Spain. Israel and Japan have many entries, but less readers, which say concentration of loyal readers.
• The biggest referring source, no surprise is Google, other big reference sources are: the TIBCO blog, David Luckham's CEP site, Marco's rulecore Blog, and ebizQ.net.
  

The Blog has been cited by the CEP Knol and has become a major reason for the fact that I was approached by a publisher to write a book.

The other Blogs of the community are either company's Blog - like the TIBCO, Aleri, Streambase and Apama Blogs which express the company's view; or those of independent people or professional bloggers. While I work for IBM, I have never made it as an IBM marketing Blog, but a personal one. I have never been asked by anybody in IBM to write or not to write about anything, and although as IBM employee I do have some restrictions, I have taken upon myself more restrictions as the EPTS chair, among them the restriction of not to write evaluations about products; I am leaving this task to others; the only time that I was asked to revise a posting was fairly recently, when I cited the Gartner Enterprise Architecture Hype Cycle report, where I was asked by Gartner to make clear distinction between what I am citing from Gartner, and what are my own opinions, so I have revised it and restructured the posting in a way the distinction between them will be clear.

Sometimes new topics emerge, and sometime topics are repeating discussions, but never a dull moment in Blogland. Will return with a professional posting s00n.

On downloadables

Among all topics that I mentioned in my previous "catching up" posting, I chose to talk about downloadables. The Forrester report mentioned "free downloadable" as a criterion, and one of the authors defended this criterion in a short article. Paul Vincent has written about his impressions from the summer school he participated in that some of the academic people like to use open source, so they can play with the implementation code.

There are two issues here: whether downloadable is a criterion in product evaluation, and whether for academic reasons one should use open source or commercial product downloadable ?

As for the first question: the Forrester report stated that event processing is moving from being niche technology to being part of mainstream computing. An enterprise can chose in using a specific product, either because it is being sold through some solution / application, and in this case the line of business owning the application, may select this application and get the product behind it as a by-product, this is exactly niche technology kind of behavior. Getting it as part of mainstream computing, mealing that this product is being used in enterprise architecture, and being endorsed by the IT side of the enterprise. I have spent 8 years in my life in an IT shop of a big enterprise, and from this angle, the world seems somewhat different, decisions (if not constrained by force majors) are being determined after the IT developers have studied the products from all angles. Big enterprises don't typically have problems to get any software for free evaluaton; however, where free downloadable may help is for people who want to explore products by their own initiative, whether they are developers in organizations that have not decided (yet) that they want to do formal evaluation, or by students, who are the future generation of developers. Investment in students is considered a good investment and thus many vendors provide free software for academic purposes. As I have written in the past, the publisher of the EPIA book we are writing is very keen on providing downloadable software for the reader. Since we told him that there are multiple approaches and we don't want to restrict the reader to a single approach, we have approached several vendors, who agreed to: a). provide free downloadable; b). provide a solution of (at least a subset) of the "Fast Flower Delivery" example that accompanies this book. Currently we have several vendors that already agreed to work with us on this: Aleri (representing the Stream programming style), Rulecore (representing the ECA rules programming style), Apama (representing the script programming style). We also got agreement from Esper (which is already an open source) to provide the solution of the case study in ESPER. I hope that other vendors will follow. I am sure that the readers will be benefited by this exercise (we'll put all solutions in an appendix to the book, and point at the soft copy of the solution, and downloadable from the book's website).

Now, to the question of downloadable vs. open source in teaching. I am giving students two types of programming assignments (depends on the course) -- one is to write some application using a product, the other is to write a subset of an event processing engine.

For the first type of assignment, both product downloadable and open source can be used. In the database area, I guess that the majority of the basic database courses are using My SQL, however, in the event processing area we don't have a standard language yet, and typically products are more advanced than open source.

For the second part -- writing a subset of an event processing engine -- one can take an open source as a basis and extend it, which may be one approach (common in operating systems course), my personal preference is to let students build such system from scratch, the reason is that again, due to lack of standard, the use of open source will restrict the student to use the approach taken by that open source, alas, the approach in which I am teaching event processing is somewhat different. I also think that students have to think about how to implement concepts rather than to ruminant on other food. More on teaching considerations -- later as I am prepared to teach an event processing course twice in the two semesters for two different populations, in the Information Systems Engineering program at the Technion, where I'll ask the students to use products, and in the Computer Science program, where I'll ask the students to write a mini-product.

A comment: I am moderating comments after some spam attack in the past, recently I am getting everyday a comment to this Blog in a charachter set which may be Chinese or Japanese.
I wanted to respond: לא הבנתי מה רצית ממני, but as this Blog is being handled in English, I will not post comments in other languages, I don't even know whether they are spam or real comments.
I also don't post comments from people who don't identify themselves.

Last but not least, a reminder: The EPTS 5th event processing symposium is getting closer, if you wish to participate, and have not registered yet, please do it on the conference site, the reason we need this registration ASAP is that there are quite a few of requests to register from non EPTS members, and since we limit the number of participants, we want to make sure that we are within these limits. BTW - the program on the site is not up-to-date. The updated program will hopefully be published next week.

ON catching up

This picture is another picture from Finland, this time from the national park Helvetinjärvi in the Tampere area (more or less), but time to move on and return to work, which I am doing rather slowly (apologies to all Emails I have not answered yet, will catch up by the weekend).
A lot of catching up in work, but now I took a small break, drink bitter coffee, and had a quick scan on some Blogs, some deserve detailed responses, but in a catching up mode I'll only provide highlights:

• Marco has written about patterns, there are two comments here - one a terminology comment, he seems to be confused by the distinction between complex event, composite event, situation, event correlation (a term I don't really use) and pattern. These terms came from different sources, and they are not exactly synonyms, I have tried in the past to look at their definitions and outline the relationships among them, the interesting question is do we want to stay with all legacy terms or settle in a fewer terms. This is a question for the community, and for the glossary work group.
• Marco also started to look at list of patterns. I'll refer to it in one of the next postings, I am now working on the chapter in the EPIA book about patterns (chapter 9). We also are looking at it within the language analysis work group.
• Paul Vincent had a few interesting postings, some of them summarize iimpressions from the summer school in Norway in which there seems to have been discussion among different approaches.
• Paul Vincent also had another posting. getting back to the CEP vs. ESP issue. I've already made a short comment to this postings, that I think that this label discussion is somewhat anachronistic, since the term ESP is virtually gone away. Besides the labels, there are indeed different approaches to approach event processing applications, and I think that we'll see more of hybrid approaches. This also deserves a longer discussion.
  
• Paul Vincent also wondered about downloadable as criterion, following the Forrester report, that has put it as one of its 114 criteria to compare EP products. John Rymer from Forrester has responded in ComputerWorld UK. I would like to have further discussion on the Forrester criteria in general (within the catching up I also read the full spreadsheet of comparison, and have some interesting insights), also I need to follow up on the downloadable issue. Not sure if it is a criterion about product quality, but IMHO, free downloadables are important on getting event processing from niche technology to the mainstream, which was the major theme of the Forrester report, this issue of niche vs. mainstream, also deserves a discussion by its own right.
• Hand Gilde has written about the Gartner Hype cycle report. He was not happy from the way that Gartner articulated the value of event processing, and sent the challenge to EPTS to make better articulation of the value of event processing to the general community. I agree that EPTS needs to catch the ball here, and will propose to imitate such activity. As always, I am throwing back the ball to anybody that throws it towards me, I think that Hans is an excellent candidate to be one of the key persons in such an activity, and I call him to quit his bystander position and jump into the cold water.
  
• Mani Chandy who uses his recent Blogs to post articles, has written about "Smarter communities". This also deserves a longer discussion, I agree that we are heading this way in variety of areas that IBM labels as the "smarter world".
  
• Some other references to the Forrester report which seems to catch the attention and kept (almost) everybody relatively happy. I still need to discuss some of the observations from this report. While looking at the detailed spreadsheet, I have did a quick summary of the number of customers reported by the different products. The number looks impressive, I wonder how many of these are currently running systems in production. But I'll let the use case work group chase them to obtain use cases, and ask everybody to help on this -- another topic to discuss.

Well - enough for now, back to the long list of to-dos...

Some insights from the vacation in Finland

This is a picture of the cottage in Finland, where we spent the last week. The cottage is totally isolated, in a small Forrest, near a lake. In the previous week we spent in another cottage, also in Finland. This morning, I have returned home (and tomorrow return to work), and here are some fresh insights from Finland:

• There are almost 200,000 lakes in Finland; comparing to Israel, in which we have two (2) lakes, that we call each of them "sea", the dead sea is one of them.
• The local drivers are surprisingly obeying speed limitations much more than any other country I have seen so far.
• Sauna is an important part of the picture, we visited a place that has been prison until 16 years ago, and found out that even prisoners had right to use the sauna once a week; I tried using the sauna once, but I never liked sweating.
• The Finnish language is not similar to other Nordic languages, it sounds to me like Hungarian spoken in Italian accent, but this may be my own imagination.
• The summer in Finland ends roughly in August 10, we discovered that some of the attractions are already closed, since the summer (i.e. the summer vacation for schools) has gone. We succeed to visit the stone age village in Saarijarvi, thanks to the lady from the Saarijarvi tourist information that coordinated to open this site especially for us.
• I have used GPS in order to navigate, it brought us to the most deserted places, however, we found out that one needs to use GPS in Finland (and elsewhere) with a grain of salt, and when it directs you to a route that does not seem to you the intuitive way, prefer your own intuition.
• The most common "fast food" in Finland is Pizza.
  
• Overall -- great place for peace and quiet and nature walks, good for disconnect from civilization.

And one comment on the flights back -- we came back from Finland to Israel in a combination of a Finnair and ELAL (the Finnish and Israeli airlines). Both Finland and Israel see themselves as high tech super-powers, however, the software in their airline is not impressing. The seat assignment software of Finnair, has problem to understand that people who travel together want to sit next to each other, and around me several people switched places to achieve sitting together. It seem that ELAL has not heard about modern development in distributed computing, fault tolerance and some other trends. We checked in relatively early and it was OK, however, at some point, the centralized server in Israel crashed, and they could not bring it up, and it seems that the remote station was completely helpless, they ended up making manual seat assignment, and the flight was delayed for almost an hour. Back to professional Blogging - later this week.