Monday, November 24, 2008

On evaluation criteria for EP products

Typically, I refrain from reacting in this Blog to any marketing material presented by vendors, a restriction I have taken upon myself as the chair of EPTS. I am not deviating from this rule, but since my friends in Coral8 have posted their article entitled: Comprehensive Guide to Evaluating Event Stream Processing Engines on David Luckham's site, as a vendor-neutral service to the community, I am taking a freedom to put some footnotes to this paper.

On the positive side, I think that this type of work is useful, and discussions about it is also useful, and many of the criteria presented are valid. We in IBM have devised in the past criteria for evaluation for internal purposes that included many of the mentioned criteria, I have to check if we can expose them.

On the critic side - here are several comments:

1. The first claim is that the authors view "event stream processing" and "complex event processing" as one and the same, saying that customers do not make distinction between terms, and saying that there is no agreed upon terminology. I am referring the authors to the EPTS glossary as a reference for terminology. But regardless of that, I would agree that customers typically don't care what TLA is used, the substance is more important.

2. Giving the statement that the coverage of this document is ESP and CEP which are one of the same, have created the feeling that this document is general, however, reading further I find out among the criteria that define what is ESP engine the following condition: "...process large volumes of incoming messages or events". This criterion confuses me -- is that a fundamental property of ESP/CEP engine -- I have heard in the recent year some analysts talks saying that actually most of the potential EP applications are not the "high volumes" ones, furthermore, the customers I know have various degrees of event volumes, some of them high, some low -- so maybe this is not part of the definition of what is an engine, but an evaluation criterion for certain amount of applications.

3. Reading further I see terms like: continuous queries, windows -- terms that already assume a certain type of implementation (indeed --- query-based stream processing), this fits the title of "event stream processing" assuming that there is an agreement that this is what ESP is, however, it does not represent the entire spectrum. Continuous queries is a technique that is intended to achieve some functionality, that can be achieved in other means.....

Personally I believe that "one size fits all" does not work, and that different event processing applications have different functional and non-functional requirements. There are applications in which various performance aspects are more or less important, note that there is also no standard benchmarks yet. I hope that the work of the EPTS work group on use cases that is planned to result in classification of event processing applications will result in a finite, manageable number of application classes, so the evaluation criteria can be partitioned by type.

And -- if possible, hands on experience indeed makes the evaluation more accurate and removes noise of preconceptions and false assumptions... More on evaluation - later.


Marco Seiriƶ said...

You have indeed an excellent taste when it comes to cars ;) I saw two of these beasts a couple of weeks ago here in Sweden. The factory is pretty close to where I live. Some day I'll buy one...

Anyway, this "independent" guide is ofcourse marketing. The main goal seems to get out the message that event processing equals to SQL based stream processing. It's a good guide if you evaluate SQL based stream processing tools. So for comparing Coral8 and StreamBase it might be useful.

It's obvious and fair that the goal of Coral8 is to make sure that everyone thinks that event processing can only be done using the SQL based concept.

It would be naive to think that it's anything more than a part of the marketing package of Coral8. I'm sure Coral8 owners would not want their marketing spend time in promoting competing products in a truly independent evaluation guide.

Opher Etzion said...

Hi Marco. I actually have driven a series of Japanese cars, but will keep in mind those made in Sweden.

Naive or not, I am taking the approach that since this has been posted as service to the community on the complexevents site, it can be critisized as such.



Paul Vincent said...

To be fair, David does say
"Here is one vendor’s “vendor independent” analysis of the issue".

I'm pretty sure he realizes it is not independent at all. Probably something he should have posted for discussion on the forum rather than the front page. I'm sure the Coral8 guys wouldn't mind independent analysis... :)

harvey said...


All discussion is useful, even if prompted by marketing... As long as we recognize it as such :-)

Opher - on the lighter side, did you see that Israel took second place in the recently concluded Chess Olympiads?

Now _that_ is an EVENT - LOL!

Anonymous said...

Hello Opher, hello EP community

I read the evaluation guide and do not see it that critical. I think the main concern of the paper is to give users an idea of criterias which are adequate to look at. Not every single one of them suites every usecase, but you can expect a user to decide weather he/she needs a fast engine, high troughput, all the adapters or not. What are the points you do not share? The SQL based syntax? They also mentioned other techniques like graphical modellation and java/c++ style syntax. What would you concider plausible evaluation criterias instead. Do you have any other criterias in mind? I would apreciate any suggestions on that topic, because I will start my own evaluation very soon. Is there any list of CEP (not ESP) vendors in the web. All I found is the forrester research paper but I am not sure how they draw the line.

Kindest regards,
Tobias W.