The mythical "event per second"

Still in Orlando, the Gartner Event Processing Summit is now behind us, and the Gartner folks are happy from the turnover, and promised another summit in August 2008. The closing keynote speaker was David Luckham who needs no introduction in this community, who talked about the past, present and future of event processing. David has positioned himself as the prophet and the person setting challenges to the community, and talked about "creeping event processing" (present), "event processing as first class citizen in computing" (five years from now) and ubiquitous event processing, where event driven architecture and event processing will be fundamental to computing infrastructure, with the vision of dynamic event processing networks, that consist of Internet-scale amount of agents, that are dynamically created and destroyed. Another interesting talk was the talk of Robert Almgern, who has a long academic record, but currently works as Managing Director, Head of Quantitative Strategies for Banc of America Securities. He has provided excellent introduction to algorithmic trading, one of the most pervasive applications of event processing products today. There has been one sentence that attracted my attention in his talk : "vendors are talking about 200,000 events per second, this is great, but far from what we need, the typical load in our case is 7,000 events per second, with a peak of 13,000 events per second, and any product in the market can deal with this loads with no problems". He further pointed out that the main effort has been invested in connectivity with other applications, and not in setting up the algo. trading system. This brings me to earlier thoughts about the "mythical event per second" . Old timers may remember the mythical man month that discussed the fact the problematic in estimating time durations in software products. While some vendors attempt to make the "event per second" as a main property - there are two questions: the first one -- what does it mean ? is that throughput, well one can have very large buffers within the boundaries of the system, and make sure that all events are there in the buffer and will be processed eventually - this is a good property, but does not make it high performance. It is how many events can be processed in a second that matter (thus, latency is impacted), but what do we mean by process -- this is like talking about "transaction per second" without specifying what is there between the beginning and end of the transaction, this can mean anywhere from filtering out the event, and using it (and its descendants) in 2000 pattern detections. Thus, without a benchmark, this term is meaningless. However, the second question is even more interesting --- how critical is high performance ? some vendors have interest to make it a major requirement, and if you'll ask customer - do you want high performance, nobody will say no -- which reminds me that once I have interviewed customer and asked whether he has a need of high throughput, the answer was - "yes, of course", then I said about the quantity, and the answer was " around 10,000 events per hour", and if you think of it, for a human this is a high throughput. While there are certainly some applications which need "high performance", the evidences show that majority (say 95%) of the candidate applications do not require very high performance, since the main value of event processing is the abstractions to mitigate the complexity, and not the throughput. I remember a discussion with CIO of a bank, who said -- well - today we are doing these things in batch, let's learn how to walk, before start dancing. I think that we need to teach the customers how to walk first - use the right abstractions, integrate with their systems etc, as a higher priority. Bottom line -- I think that the "high performance" in event processing is somewhat hyped, and is only one of a series of considerations, certainly not the most important for most applications. I have started with David Luckham's talk -- and will end with saying that EP frameworks like the dynamic event processing network mentioned before will, by definition, resolve all scalability issues in the framework level, rather than in the engine level.

More on frameworks vs. engines - later.

The role and prospects of standards in event processing

Still in Orlando, we have finished the EPTS meeting yesterday, and I'll summarize it within the next few days, staying for the Gartner Event Processing summit. The fact that Gartner made a commercial conference indicates that the level of interest in event processing has crossed a meaningfull threshold. In order to participate in the Gartner meeting, I also have an IBM "booth duty", which requires me to dress in a way that I am not accoustomed to (disguised as a civilized business person) -- somebody told me to put a picture on the blog of how I am looked, since in reality, there is a rare event, so you have low probability to see me this way, so this is the picture you see on the top. Anyway, one of the topics that have been mentioned in the Gartner meeting by Roy Schulte is that SOA became a reality only after web services standards started to be pervasive, and advocated for standards as a major enabler (and lack of standards as a major obstacle). We had long discussions in the EPTS meeting about standards, it seems that there is an agreement to advance towards standars, but with some caution. First, some people were not sure that our understanding of event processing is mature enough for standards (this is IMHO also true for the SQL extension proposal that is going on, I have some doubts about the maturity of their thinking)... On the other hand, we need to show the customers that we are working towards standards, and try to make the thinking more mature (more on when I'll summarize the EPTS meeting), anyway -- we can classify the standards to three main groups.

(1). Modelling standards.

(2). Interoperability standards.

(3). Language standards.

OMG is about to issue RFP for modelling standards, and I guess that we'll deal a lot in this issue over the next year, on the language standard -- I have already discussed the idea of meta-language first, and it seems to get some momentum, interoperability standards are more difficult since there are a lot of related standards both on event structures and on event transport, and we'll probably defer this discussion for later.

Is there a way to accelerate the process of getting to standards ? -- probably yes, but it requires level of investment from the community, much higher than invested so far --- and this is the challenge we'll have to deal with --- more later.

Classification of event processing applications - the challenge

Hello from Orlando, two out of the three days of the EPTS meeting is behind us, and there is half a day to go, before the start of the co-located Gartner Event Processing Summit.
I'll summarize the meeting some later this week (well, it should end before summarized). Just will report on two interesting insights. The first was made by Richard Tibbetts who talked about two types of event processing functionality: transformation and orchestration. This has some similarity to classification of rules to rules that create additional facts (derivation or inference) and behavioral rules that trigger actions (or prevent actions from happening). This classification is valid, and joins some other classifications made of event processing functionality.
The second insight, which will probably generate action item tomorrow in the EPTS closing meeting is "classification of applications". During the three EPTS meetings, the Dagstuhl seminar, and some other related work, we have collected a long list of use cases in many domains and types, and now we have enough sample to start analyzing them. What we'll probably need to do is to get back to all the presenters in all these events with a request to provide information using some templates, and then analyze the information and draw some conclusion about --- what event processing in reality is really is... stay tuned...

Other than that, good meetings - both inside and outside the regular sessions. Some newcomers, and most of the old players...

EPTS (Event Processing Technical Society)

Hello from Orlando, Florida. I am here to attend two meetings that directly relate to event processing: the EPTS third event processing symposium and the Gartner event processing summit, next week there will also be the EDAPS workshop of the VLDB conference in Vienna - which will make it all together six days dedicated to event processing meetings of various forms. I'll be writing about these events in the blog, but as an introduction I would like to provide some background about the EPTS and what we are trying to do. I have got to think about the need to have an event processing community, in 2005, when I was assigned by IBM to do a requirement survey in various industries to understand what event processing is from various viewpoints. There were two points that came across - one is that many potential customers are not aware of what event processing really is (though they may be familiar with some terms), and the second is that there is confusion in terms, and general conception that this is a proprietary, immature, area. Clearing the terminology, start thinking about standards, and get general awareness, cannot be done without a community, and in fact, the reaction of customers of having such a community is very positive. It took a while to organize the first meeting, and there was a concern about the success - since there are some vendors who claim individually to have invented this area (we'll talk about the professional history and roots other time), it may be difficult to bring them all together to agree to anything. I succeeded to get IBM to host the first meeting, and found partners that got to the same conclusion that community is needed, Mark Palmer who started the Yahoo mailing list, David Luckham who is a long time evangelist of this area, and Roy Schulte from Gartner who helped bringing some of the others - and then TIBCO, SAP, Oracle and Streambase joined the organizing committee for this meeting. The meeting has been a social success, we learned to know the people in this area, we understood that terminology is the first issue to handle - and got a first working group to look at glossary, and heard some position presentations and some use cases. We had even better second meeting hosted by Oracle (Shilandera Mishra and Dieter Gawlick), in this meeting we have decided to form EPTS, as a formal consortium, and work on formalization has taken a while, and the final proposal will be discussed in the business meeting today. The idea is to have an open consortium of both organizations and individual members that will work through working groups in various areas, the aim is to incubate standards proposals - and hand them over to standard organizations (the coming OMG RFP on meta-model will be first instance, probably), host community meetings, work on industry- academia relationships, common terminology and better understanding of this area in various aspects (use cases, use patterns, architectures, application types etc..). We are starting the 3rd meeting today - and I'll report back on interesting insights about this meeting. More - Later.