Came back today from a short vacation in the Grand Vista Boutique hotel in Kfar Yuval, at (almost) the most northern (and east) part of Israel. Looking at some related blogs, I found Joe Mckendrick's blog mirroring Colin Clark saying that the "CEP vendors" failed to produce the multi billion dollars revenues they promised to their investors this is due to two facts: 1). end users cannot understand it; 2). it does not provide the IT people a tool to build a complete application.
I am sure that Colin, who has been a key person in one of the first generation startups in this area is right in his perspective. However, coming from the big blue, I have a slightly different perspective.
I'll start from the last point, since this is a key to the rest: the claim is that the "CEP vendors" did not offer a tool to build a complete application. This is of course true; there are applications (indeed in capital markets but also in other industries) that have applications that are mostly pure event processing applications, however, this is not the major part of the market; the major part of the applications market is such that require event processing capabilities within an enterprise computing platform, that also includes BPM, BRMS, ESB and other stuff. This is the approach taken by the bigger companies that got into the event processing game. It is true that a start-up cannot do it alone, and start-ups that remain "pure event processing" companies are confined to the niches of stand-alone EP applications (in which the potential was also not reached, so there is certainly a place for products in this niche). As for the multi billion dollars, I don't think that there is any analyst who predicted multi billion dollars for that market in 2010, it takes time to develop a market, and as a stand alone EP products, may take longer time to get there. When we did the business evaluation of the EP market around 3 years ago, we got to the conclusion that for the market of "EP capabilities within an enterprise computing platform", one can look at accounting value or on economic value and they would yield different results. Let me give a simple example in order to show what I am talking about: let's say that a big enterprise is doing a software deal with the value of $15M for its enterprise computing infrastructure. This may include many software components licenses. Let say that the deal is a bundled deal, but if we'll try to isolate the value of the EP capabilities license, we'll get to $500K. However, in the RFP the event processing capabilities are required part, and a vendor that does not have the required capabilities is disqualified for this bid, then this vendor, losing the bid did not loose a $500K deal, but lost $15M deal. This is the distinction between accounting value and economic value. We see more and more RFP that either put event processing capabilities as mandatory, or give some weight to it as explicit criterion, this is done, even if the intended use of EP is planned in 2-3 years in the enterprise IT investment plan. I think that looking at economic value (e.g. deals that have EP components inside) we might already passed the 1B dollars mark, but this arithmetic, of course, makes sense to vendors who own enterprise computing platforms and not to start-ups that has to earn the actual accounting value of stand-alone EP applications (or sell it as embedded solution). About the "not appealing to end users" - I'll write a separate posting about it; some of the analysis from the user's point of view was done as part of chapter 1 of the "event processing manifesto" (a follow-up to the Dagstuhl seminar on event processing) which is planned to be published in January 2011, and presented in March 2011 within the virtual EPTS symposium. More about this topic - later.
With several more members that joined EPTS (= event processing technical society) recently, the number of members is getting closer to 100 (I think we are in 95 now). There are several recent activities associated with EPTS:
- The Dagstuhl seminar on event processing held in May 201 which is affiliated with EPTS and the follow-up activities have resulted in a complete draft of the "event processing manifesto" now up for review among the team. I'll write about it in a separate posting.
- EPTS is going to hold its next symposium as a virtual symposium, meaning both speakers and audience will be able to participate over the cyberspace and the talks will be recorded. The plan is now to hold it in March 2011 in conjunction with the OMG technical meeting, I'll write more about the agenda, when it will be finalized (probably next week).
- The EPTS use cases working group, lead by Pedro Bizarro and Matthew Cooper has posted a use case survey, which should take 15 minutes to fill, please contribute by responding to this survey.
Within the OMG technical meeting in March 2011 in Washington DC there will be a workshop on real-time systems, the call for presentation and tutorials is linked here, with the deadline of November 26. We plan also to do a virtual EPTS symposium during this event, to enable a broad audience to listen live or to a recording, more details - later.
I realized that I have not written anything in this blog, in the last week, well - there are weeks in which one is in a passive mode, anyway, I am at home, ready to go to the office (Sunday is a working day in Israel, our weekend is Friday-Saturday), but wait for the technician from the phone company, all our phone lines are dead since Thursday afternoon. Since each member of the family has cellphone, it is not a big damage, the most notable damage is that the wireless internet at home is linked to the phone (ADSL), however, we have a second home LAN (wired) which is based on the cable TV, since none of the internet connections are reliable, they also serve as a backup for each other. The phone company was a government owned company that was privatized, but retained a strong employee union as a legacy, and as such, its service is not that good, so we'll see when the technician will bother to come.
Anyway -- the picture above is of stream of events going out directly from the human brain, it is taken from a recent article in Scientific American. There is a lot of recent work about brain-computer interface, the brain itself is, of course, an event processor, it receives events through the senses, makes some decisions, and acting with some actuators. There are also a lot of events being created in the human body that has benefit to be processed elsewhere, either due to fact that we want to have "extension" to the brain to help sort out the events and detect situation that the human brain may miss, or that the human body is just one source, and there are other sources for events that this particular human is unaware of.
I see a lot of future in taking what was done in event processing and utilize it outside enterprise computing, and I'll investigate this theme more in the next few blog postings.
An old Jewish folk story is about husband and wife who came to a Jewish Rabbi for marriage counseling, the Rabbi listened to the husband and said: you are right, then listened to the wife and said again: you are right. After they left his assistant asked him: "Rabbi - why did you say to both of them that they were right, it is not possible that both of them are right", and the Rabbi answered again - "you are also right". I recalled this story after reading Mark Palmer's Blog entitled: "how broad is the appeal of CEP". The story starts with the CEO of SAS claiming that CEP has limited appeal continuing with the rebuttal of Progress and TIBCO, saying that the SAS guy underestimates the potential and there are many applications in multiple industries. Mark Palmer has a counter-opinion saying that relative to BI, EP is still small, but has a potential to grow, and it is hot within a single industry - capital markets, with relatively modest interest in other industries - Chad Gadya!
So why does it remind me the Rabbi's story? -- since each of them is right in something.
EP is indeed a smaller market relative to BI, and much younger and less mature. Some of the BI people regard EP as a "non issue" since one can do anything with BI tools, or database queries, and the only reason to use EP is when there are latency and throughput requirements that cannot be satisfied. However, as noted many times, the bigger appeal of EP is the abstractions that enable to develop and maintain applications of that type more easily and substantially reduce the TCO. The observation that EP market is much smaller than the BI market today is certainly true, maybe the fact that the SAS guy bothered to react on it is a sign that they start feel some presence in their territory. BTW -- I don't view BI and EP as competitive technologies, each of them are destined to do different things, some applications need both.
Next -- inside the EP house, it seems that there is some agreement, about the potentially bright future, and some disagreement about the present and even more about the scope. I guess that appeal is in the eyes of the beholder, but let me make one observation: when Streambase is looking at the universe it looks at the market for a stand-alone event processing product; when some of the bigger companies look at the universe, they look at event processing capabilities as part of a larger enterprise computing infrastructure. When we did, a few years ago, the business evaluation work for IBM, towards the decision to enter this market, we have identified both of them as opportunities, the stand-alone one has its own existence, but is more limited in scope, for the "embedded" one -- possibilities are much higher, but more work is needed to make event processing capabilities as part of facets of enterprise computing infrastructure -- decision support platforms, business process management platforms, messaging oriented middleware, sensors and actuators frameworks and even business analytics framework. These two view points of the universe provide different perspectives to the beholders, and so each of them is right from his own perspective. More - later.
In 1998 I have edited a book of articles about temporal databases (together with Sushil Jajodia, and Sury Sripada), this followed a Dagstuhl seminar we held in 1997 about temporal databases, an area that was hot at that time in the research community, and somewhat cooled off. Today a Master student I supervised took her final exam on "final work" (which is less than a thesis, a track that require to take more credit work), and did an implementation of a temporal database model from a paper in this book that was co-authored by Arie Segev, Avi Gal and myself. This is somewhat more expressive model than the TSQL based models, and had its own interesting featured like: ability to freeze and unfreeze data, ability to distinguish between modification and revision, ability to deal with simultaneous value. In fact some of these ideas found themselves into our work on event processing (e.g. policies when there are repeating events that may match the same pattern).
Temporal databases as an area started in the Israeli army. Kobi Ben-Zvi who went from the Israeli army to do PhD in UCLA has invented the area, by formalizing the terms, and there has been a lot of work later in the research communities in the 1990-ies. There was even big fight about how to extend the SQL standard to support temporal databases between two parties, I don't really remember the details, in the book you can find the position of the two sides of this battle, as the Dagstuhl seminar was one of the battle fields. The end result is that it never became part of the SQL standard, partly because of the fights, and more importantly since at that time the DBMS vendors have higher priorities on their mind -- e.g. Web related stuff, XML data etc.. There are some features, but it did not get fully into the mainstream of databases, although there are quite a few of specialized implementations. One of the future directions of event processing will involve getting back to temporal databases as an infrastructure, which is the area of retrospective event processing, I'll write more about it in the future.
Then they asked me to provide some impressions
We were briefed beforehand and were told that we'll not be able to use slides, just use voice, so I have given some insights and impressions in a way that was described later by people as a "stand-up performance" (well, I was sitting down), this is consistent with one of my old habits - Ridendo Dicere Verum, when we started working on event processing in 1998, we did not know that we are doing event processing, we did not actually used that name, we also did not realize that 10-12 years later it will be an established and well-recognized area. Getting new ideas in a big corporate are never easy, and I have written before about some best practices to navigate.
Getting a recognition is always fun - but as my motto in life is based on the poem "IF" by Kipling, I'll end with one citation from this immortal poem:
If you can meet with Triumph and Disaster
And treat those two impostors just the same;
This award has reflected past work, but the challenge for the present is to work on the future of this area, my recent talk in the OMG conference reflects some of the work towards the future.
