On the criteria to evaluate event processing products

This is a map of Finland, the location of our family vacation for this year. The vacation is planned to start in Saturday, and I'll be disconnected from the cyberspace for 15 days. Working late at night to advance in the EPIA book we are writing.

It seems that this is the time of the year of analysts report, the community blogland was full of references to the Forrester wave report - Complex event processing platform, Q3 2009, dated August 4, 2009.

I will not comment about the grades that they gave the different products, the reason for that is that I decided that in my role as chair of EPTS, I see my role to work on the coop side of the coopetition, and leave the competition side of the coopetition to others. I think that the main competition is not between the different vendors (though they have point competitions), but against the barriers that the event processing area has to fulfill its potential and become a pervasive main-stream technology.

The Forrester reports starts the executive summary by saying: "Forrester evaluated nine complex event processing (CEP) platforms using 114 criteria".

Without getting into the long list of criteria (not part of the report itself, but I managed to look at it), I have some doubts about the ability to get a meaningful information to customers by weighing 114 criteria. There are two reasons, one is practical and one is methodological.

On the methodology side, the compensatory model of decision making advocates weighing of many criteria, however, experience shows that the actual decision making model is lexicographic, meaning ordering the criteria according to importance, and making the decisions according to the most important criteria. People may use a compensatory model of weighing a lot of criteria, if their organization require them to work this way, but this is done only as justification to decision that has already been made by the lexicographic model.

Let's move from decision making theory to the event processing universe. The event processing universe is diversified from both functional and non-functional requirements point of view. I really don't believe in a "one size fits all" in anything related to this area, and this goes also for set of evaluation criteria. Getting criteria that are good in variety of cases and weighing them together may not get a good solution to any particular case. The more practical approach is to set a collection of relatively small sets of important criteria, and also segment the space of application, and assign a set of criteria to each segment. Anybody that will manage to do it, will help customers more to make the best decision for a particular case. I hope that EPTS, through its use cases workgroup will be able to provide this segmentation, and this will be the starting point for analysts to come with the right criteria for each segment.