Saturday, March 5, 2011

Event processing - manual, build or buy ?

The various chapters of the event processing manifesto, published yesterday, hold some interesting insights about the state of the practice and the future of the event processing area. Chapter 1 written by a team lead by Robert Berry has provided some insights and directions about why use event processing systems.  The figure below taken from the manifesto (figure 1.5, page 20) is talking about a frequently asked question - whether an event processing computerized system should be used at all, and if yes - should it be implemented as part of the conventional programming or using a dedicated  COTS event processing system.  

As seen in this figure - this is a function of complexity.  There are various complexity factors, defined earlier in the document:

1. Degree to which the application is expected to change over time, e.g., with new
    event sources, new interactions and new responses expected
2. Numbers and types of event sources
3. Numbers of consumers of information communicated in the events
4. State and context management
5. Opportunity to create new value, e.g., by introducing reflection and introspection

Back to the decision - in some cases the complexity is low, and there are no really processing required except for getting the events and send them to some person directly or displaying them on a dashboard.

 In this case the most cost effective solution is "manual", namely, a human is the event processor and no computerized system is required to do any processing, except for routing, which can be done by any messaging or DBMS system. 

The first vertical bar (from left) is the complexity break-even point between the "manual" approach and the "build" approach, beyond this bar it is cost-effective to construct an event processing computerized solution, but the complexity of such system is quite low, typically doing filtering, some transformations, maybe simple aggregations, typically without strict timing constraints.  In such cases learning and using a COTS system might be an overkill, and it is relatively simple to develop as part of regular programming.

The second vertical bar is the complexity break-even point between the "build" and "buy", beyond this bar, it becomes cost-effective to invest in COTS (the right one which satisfies the requirements, of course).

Note that sometimes the decision should be forward looking, if there is a plan or prediction that the complexity of the application will increase within the short and medium range, then it should also serve as consideration to avoid re-writing of the system within a relatively short period of time.

I think that we need some empirical research to determine how to measure these two break-even points in a more exact way.

Our next mission is probably to devise best practices for the community on this and other issues, and it is already proposed to have it as one of the next working items for EPTS.  

Friday, March 4, 2011

The event processing manifesto has been published today

Today, the Dagstuhl publication service published the "Event Processing Manifesto" which is the result of the work started in the Event Processing Dagstuhl seminar that took place in May 2010 (after long pregnancy).
This 60 pages document is a seminal work that is a result of the collaborative work of the 45 participants seen in the picture below, and quite a lot of editorial work that I have done with the help of Sharon Geva and Chani Sacharen, from our local technical editing department.  The document has six chapters (five original workgroups, one of them decided to generate two chapters).  The chapters deal with: why use event processing, what is the functionality of event processing, how is event processing related to other disciplines, what is the current related standards and what should be the future roadmap, what is the grand challenge in the entire industry level, and what are the shorter term research issues that will promote the area.  I'll dedicate a posting to each of these chapters in the next few days.  The EPTS virtual symposium, planned for March 24, will provide an opportunity to get perspectives on each chapter from the person who lead the construction of this chapter.