Acting faster than the speed of thinking

Chris Taylor from TIBCO has written in a Blog with the nice title: "a place for good ideas in a fast changing world" entitled "getting there faster than your opponent".  In the same spirit of TIBCO's two second advantage, it makes the point that event processing is vital for getting things faster than others.  Chris enlists the famous OODA loop originated in the US Air Force, for mentioning that faster decision can impact the combat field.   Note that OODA is one of the variations of  control loops, other variation I have written about is the 4D variation.    

It is interesting to note that speed of reaction has been one of initial reasons for using event processing technology in applications like high frequency trading, where trading programs compete on speed.  I guess that military applications also gain from competition in fast reaction, as well as cyberspace wars.   

While these are notable applications, a common misconception is that event processing is restricted to these type of applications, however, there are many other applications in which competing on speed is not an issue that can benefit from the use of event processing due to the benefits in reducing cost of development and maintenance due to higher level abstractions.  In fact one of the first posts in this Blog, almost five years ago tried to answer the question whether the only motivation to use event processing is high performance?  in this post I have discussed the Total Cost of Ownership as a function,  I guess it is also applicable today.

The two  main observations are:   acting faster than the speed of human thinking issue an important type of event processing applications,  but not the only reason, also in many cases event processing is not enough, and some real-time decision mechanism (reactive or proactive) need to be applied to achieve autonomic action, since the required speed requires the elimination of human from the loop.

"The only motivation to use EP COTS is to cope with high performance requirements" - true or false ?

Somehow, I find myself using my collection of analysts' presentations to help me make some points, this time, I am showing a slide from Roy Schulte's presentation in the Gartner EPS summit - I'll return to the content of this slide shortly, but will start with some discussion I had yesterday about the reasons that enterprises are using COTS for event processing. I have heard (and not in the first time) the assertion - the only reason that one will want to use EP software and not hard-coded solution is the ability to cope with high throughput / low latency - in short, to deal with high performance requirements. If there are no high performance requirements there are other solutions, e.g. the database guys think that in this case one can insert all events to a database and use simple SQL queries for CEP patterns, or just using the good old C/Java programming for this purpose. This is somewhat inconsistent with my own experience, where customers that did not have "high performance" requirements were eager to use CEP technologies. Indeed, high performance is a reason to use CEP COTS, however, as indicated in Roy Schulte's slide above - it is actually a minor reason. According to Gartner, the high end is somewhere between 5-10 percent of the candidate applications, while looking at the prediction for 2012 - the use in the high end will be 8% - out of the 27% total use; note also that Roy Schulte defines the high end as 250 events per second, which is really far from the definition of "high performance", so the numbers are even lower. It seems that the market for "non high performance CEP" is much larger, and will grow faster. If that's so - where does the misconception that EP equals high performance always ? I think there are two sources - the first, the early adopters were from the capital markets industry, where some (not all !) of the candidate applications has indeed high performance characteristics. However, with the growth of the market, and use of EP software in other applications and other industries, these type of applications, while continue to grow, will not match the higher growth of non high performance applications. The other reason is that some vendors make the high performance as their main message, and trying to get the market believe that this is indeed the most important property.

So - if high performance is not the only reason to use EP COTS what are the other reasons to use EP COTS ? this is a matter for investigation, but IMHO the main one is the "high level programming" and agility - in short - the ability to reduce the Total Cost of Ownership.

I'll provide more insights about the TCO issue in a future post.