On small vendors, big vendors - individual view and industry-wide view

If you want to have a pet animal, you might prefer the little kitten over the big dog; if yo want to have somebody to guard your house, a big dog might be more effective, there also people who prefer big dogs as pets, a matter of personal taste.

Marc Adler responded to my previous post about consolidation and pure play in the EP market. Since Marc is in blog reduction mode, I will not be insulted if he'll not react to this posting, but I would like to provide further perspective.

There is nothing in Marc's response that I don't agree with, in fact when I have been 30 years younger (having 30 kg less from today, and this is after I've reduced 21 kg in the recent year).

I have worked in the IT shop of the Israeli Air-Force, I have been working with a new product that was called DB/1 and later renamed as Sapiens, at that time positioned as application generator for data-driven programming (there has been evolution in positioning over the years as well), it was a product in its beginning, I have done the first project with this product on the premises of the product developers, went with them to lunches, and went to their family weddings and funerals, thus, I got excellent service for them, could ask to add features, when there was any bug or problem I know exactly who is the responsible developer and could call him, or even invite him to solve it in our site. This has worked well since I had good personal contacts with them, but moreover, at that time the number of customers they had could be counted on the fingers of a single hand, and they could provide much attention to every customer. As said, for me, it was the ideal product to work with, and I fought some of my colleagues and superiors who thought that this is a too big risk for the Air-Force to depend on them in critical application, which was of course true. I guess that Marc had somewhat similar experience with Coral8 in his previous work; I also feel sympathy to the claim that big corporates have an inclination to come to customers with more people than the customer expects to see, and has less intimate atmosphere with customers, this was always been true.

Fast forward 30 years, I have somewhat different perspective on the universe; it was somehow surprising for me to find myself being hired by a big corporate, from an employee's point of view there are pros and cons to be employed by a big corporate, there is a nice posting on this topic.

People whose small companies are acquired by big corporates sometime dislike the culture of big corporates and move on, sometime they adjust, I know stories of both types, it is not black or white.

My perspective now is more on the macro level and looking at the question of: Is the current wave of consolidation good or bad in general for the event processing area, its assimilation into main-stream computing, and the ability of play a significant role in current and future enterprise computing?

My own view is that the market moves to the right direction. As I believe that the larger market is not in stand-alone event processing applications, but as a pervasive technology embedded in enterprise computing in general, there is some benefit to companies like - IBM, Oracle, TIBCO, Sybase, Software AG, and now we also see that Progress Software is making event processing as part of a more general platform. Some applications may not need it, but many others do, and getting event processing in the mainstream enterprise software infrastructure, can be done through owners of such existing infrastructure.

One other potential benefit that I see is that with a market that is more dominated by bigger vendors there is stronger probability to get to standards. Standards is one of the signs of maturity for an area (e.g databases, web services), and will be vital to get event processing into the mainstream. We started to discuss standards in the pre-EPTS meetings in 2006, and at that time the dominant startup companies were very much against it, since they both did not see the value for themselves, and also feared that standards will distract their limited resources. Bigger companies have more standard oriented culture, and experience in other areas of how to do it right. I think that the current developments in the market provides a good opportunity to raise the standards issue again, and this will be one of the topics planned to be discussed in the upcoming Dagstuhl seminar on event processing. I'll write more about standards follows the conclusion of the Dagstuhl seminar.

Back to the original theme --- there are times in life in which one prefers small cats, and other times that one prefers big dogs. Small cats may be more cute and pleasant, but in this phase in my life I am going to hunt, and big dog will probably be more effective for that task.

On consolidation and pure play in the EP market

Marc Adler returned to the Blogland this week to claim, among other things that the fact that the list of pure-play event processing platform vendors is being reduced, is sad. The fact that there have been several acquisitions of "pure play" vendors recently is true, with the acquisition of Aleri by Sybase earlier this year, and the recent acquisition of RTM by Software AG. If you are interested in the genealogy, Paul Vincent is keep documenting it. I have traced some of the IBM acquisitions and realize that merges are sometimes tough from organizational culture point of view and sometimes there is a need to change direction in a not easy fashion, as shown in the picture above, so it might be sad for some individual people, but not necessarily bad from the industry point of view.

However, the fact that the pure play vendors are being acquired has another side to the coin, which means that the big or medium software vendors are buying them. This is a sign that event processing is getting to be part of the main stream of the enterprise computing, and is part of growing up of this area. This has also happened before in other areas, and was predicted several years ago by analysts to happen in the event processing area.

In fact, now most of the major players in enterprise software area : IBM, Microsoft, Oracle, TIBCO, Progress Software, Sybase and Software AG - have now an event processing platform as part of their enterprise computing offering.

This is an indication that all these vendors realize that event processing is required part of their enterprise applications, and that event processing is not really stand-alone but it is increasingly getting consumed as part of larger play, and getting to further areas and industries in addition to the early adopters of capital market trading applications.

There may still be a role for pure play event processing products, especially in various niches that are not being properly handled by the current products. Some of them may even develop event-driven enterprise computing platform, and join the medium companies, it happened before, but it is not easy.

While the bigger companies advance the projects at their own paste, the smaller companies as well as the research community sometimes have roles of catalysts to advance the area. We'll discuss the future of the event processing area in depth in the Dagstuhl seminar planned to start in 2 weeks. More about it - later.

On DEBS 2010 tutorials day

My role in the organization of DEBS 2010 was to handle the tutorials day. Starting in DEBS 2008 in Rome, the tutorials day became (for me) the most interesting day of the conference. Today the tutorial webpage is up and running (this is a first version, some missing details will be added soon). Some of the tutorials have been submitted in push mode by their authors, and some have been solicited in pull mode by me.

Six tutorials are planned: three tutorials in parallel in the morning, and three tutorials in parallel in the afternoon.

There will be two tutorials provided by EPTS work-groups, this time:

• A tutorial about event processing architectures given by Adrian Paschke and Paul Vincent who lead this group
• A tutorial by the new work group that deals with the value of event processing to customers, this tutorial will be given by a collection of people. The workgroup leaders are Rainer von Ammon, Guy Sharon and Nenad Stojanovic.

The additional four tutorials - two of them can be thought as extensions to the event processing languages tutorial that Adrian Paschke, Jon Riecke and myself have presented in DEBS 2009 (and was viewed or downloaded by 3551 persons, last time I checked):

• A tutorial about SQL-based event processing, given by Bernhard Seeger.
• A tutorial about Logic based representation and reasoning of event recognition, given by Alexander Artikis et al.

The additional two planned tutorials are:

• A tutorial about event processing in wireless sensor networks that was delayed from last year since the presenter Antonio Lureiro had to cancel last minute
• A tutorial that was prepared with the help of our team here in IBM Haifa Research Lab that will deal with context aware computing and its utilization in event-based systems. The tutorial will be given by Ella Rabinovich (who will be very active in the conference as she will present two papers in additional to the tutorial) and myself. Additional of our team members that helped in preparing the material are: Yonit Magid, Inna Skarbovsky and Nir Zolotorevsky. I'll write more about this specific tutorial closer to the DEBS conference.

The DEBS research and industry tracks program chairs have also sent notifications about rejection and acceptance this week. I think that there are 21 papers accepted (16 in the research track and 5 in the industry track), the research track acceptance rate was 25%.

Our lab is very active in this conference (various people on various aspects), among the 21 papers there will be 5 papers that with all or some authors from IBM HRL, and it will also have co-authors on 2 of the 6 tutorials mentioned above. This is an indication to the focus that the event-based systems receive in the lab.

Preparing for the Dagstuhl seminar on event processing

The countdown for the Dagstuhl seminar on event processing is now 17 days, and we are preparing for that event. Schloss Dagstuhl, seen in the picture, is a castle in Germany, which hold a 5 days events that deal with notable topic in computer science, for each event there are 4-40-45 invitees, most of them are the leading figures in the area, and some are young promising researchers in that area. The Dagstuhl seminar that I'll be co-chairing with Mani Chandy and Rainer von Ammon, is planned for May 16-21. Hopefully there will not be ash clouds in Europe at that time. As I have written before, the idea is to focus on the event processing manifesto. We have now determined that the five chapters of this document (or five appendices, since the manifesto itself will probably be a one page document) will consist of five chapters that deal with the following five topics:

Topic 1: Event processing scope, classification and business value.

Topic 2: Event processing functions: present and future, common and additional functions.

Topic 3: Event processing and the rest of the IT world: relationships of event processing with other areas (databases, rules, BPM, analytics, cloud computing, social computing…).

Topic 4: Event processing standards: What standards should be done and when, what should be the starting point and roadmap in each standard.

Topic 5: Event processing grand challenge: What we would like to achieve if we can get a considerable worldwide research investment? What are the research goals? What are the means to achieve? What will be the end result of this research (e.g. what scenarios will be able to be dealt with?)

Each topic will be analyzed by a group of participants, and discussed with the entire set of participants. The end result will be a clearly articulated document that deals with the scope, analysis, and future of event processing, and seed a substantial federated research projects.

Besides the professional side, it will also be a good opportunity to meet old friends that I don't see on regular basis. Stay tuned to more on the Dagstuhl seminar (and I am sure that some of the event processing bloggers like Marco Sierio or Paul Vincent will provide you other perspectives about this seminar.

Revisiting Web 2.0 and event processing

To those who like interesting numbers -- I've noticed this week that the number of my contacts in LinkedIn reached 777, it will probably not stay long, as it keeps changing. I have written before about the use of event processing in social networks, an area that I think has big potential that has not been fully explored yet. I have challenged the students in my event processing class at the Technion to select a project that is somehow related to the use of event processing in Web 2.0, and social networks in particular.

I have written before that unlike the phrase, event processing is not always about threats and opportunities. In fact, typically in Web 2.0 the use of event processing is in information dissemination - getting the right information to the right person at the right granularity in the right time. This is a case of situation awareness while the situation here is the fact that some website content has been changed, or that some activity related to social network, Blog, Wiki, and other collaborative applications happened. This can have any of the event processing functions --- filtering seems to be quite dominant, but also transformation - aggregation, translations, splitting all have a role, and of course pattern matching may also have a significant role -- example: send me update after three changes, patterns related to number of readers in a Blog (Google Analytics provides some), patterns related to activities in social networks over time and space, patterns related to tags in twitter and more... I think we'll hear much more about this topic.

Letter to the editor in the Communications of ACM

Paul Vincent attracted our attention to an article in the Communication of the ACM written by Julian Hyde from SQLstream, and wondered about some of the assertions that were expressed in this article. Reading the original article, I indeed found some inaccurate assertions that somehow missed in the review process of the CACM magazine. After communicating with the editor-in-chief of CACM, he suggested that I'll write a short response (and the author will have a right to respond to the response). It takes some time, but today I got the electronic copy of CACM and found that my response, and the response of Mr. Hyde to my response have been published. You can read for yourself (under the title - "event processing anywhere"). In essence I have mentioned three inaccuracies in the original article:

1. On whether event processing is a blanket term for stream query systems: actually event processing is much broader term then stream query

2. There is a religious war between SQL and non-SQL vendor. I have not noticed the religious war, many vendors have both SQL and non-SQL interfaces.

3. Event processing is restricted to one application type in the financial services area, while other application areas are neglected; this is indeed a common misconception, and I have written about it in the past.

Mr. Hyde took his right of response by saying that the event processing vendors fail to satisfy the expectations of the BI community, this is a new claim, which does not seem to respond to any of the issues above.

Paul Vincent in his Blog's post raises another interesting assertion from the CACM article that streaming query engine is a new technology -- well, new is a relative term.

Bottom line: while people are used to view marketing material with a grain of salt, it is typically expected from a respectable magazine which employs review processes to be more accurate, since people tend to take it as a reliable source.

On the production phase of the EPIA book

I have co-edited some "collection of articles" books that are common in the research community, there is certainly some logistics associated with it, but relative to producing a "mass market" book it is a children's game comparing with the production of the EPIA book that I have been writing together with Peter Niblett. The publisher of our EPIA - Manning, has various ways to ensure quality. Manning is a publisher focusing on computing related book, thus it views book development as a software development project (and indeed many of its books are code-driven). While the book was initiated by Manning and not by us, they have sent the outline we sent to bunch of reviewers, all of them people with deep knowledge in event processing. Some of the advices we got were very useful. Since we wrote in the outline that we are going to base the book around a single example, the advice of one of the reviewers was that we'll use an example that everybody can understand, the reviewer added that if we would chose an example from the financial services domain (as many of the EP papers are doing), we might create a communication obstacle with some of the readers, who will not be familiar with the terms. We followed this advise and created the "Fast Flower Delivery" (FFD) that already received multiple implementations in many languages. For each 1/3 of the book, we had a milestone in which they sent the book to a bunch of reviewers (in one of them there was a response from 14 reviewers). The reviewers were mix - some people who have deep knowledge in event processing, and some who don't know what it is, both types of reviewers sent various types of comments, some of them resulted in restructuring of the book (originally we planned 15 chapters and 3 appendices, we ended up with 12 chapters and 2 appendices, but with around 100 pages more than we originally planned)- This was just the development process, and it last for 15 months. A few weeks ago we have started the production phase -- it has a workflow (supported by electronic content management system): Another technical review by a "technical proofreader" - here we looked at PhD student in this area, and thus addressed faculty members active in this area; the person who did this job is Samujjwal Bhandari, a PhD student in Texas tech university, he has read all chapters, made comments and caught various cases of inconsistencies among the different chapters, then it moves to the copy editor who makes editorial modifications, and returned to us with many comments and questions that we had to answer, after doing this round, each chapter is going to the proofreader, who is doing another editorial pass, and then comes back to us with questions and comments, so we are doing another pass on all chapters. After that it goes to the typesetting. In the background there is a graphical work to redo our amateurish figures in a professional way, and then hopefully the book will be ready. Currently we passed the phases of the copy-editing, and now working with the proofreader. There are a lot of editorial rules, and style issues that have to be dealt with (Peter is much better than me in noticing the small details), and in fact -- amount of work is much higher than I anticipated, it consumes much of my free time for over a year now, as we both are doing it in addition to our daily work -- but I hope the result will justify the investment. More about the book - later.