Tuesday, August 24, 2010

More on event processing as business and technology and how this is related to CEP 2.0?

It seems that the question whether event processing is a stand-alone technology or embedded technology in other area that I've written about recently following Philip Howard's Blog, it spreading over the community Blogland.   Louie Lovas, in his relatively new role in OneMarketData, takes advantage of this discussion to highlight his company's solution about integration between EP and tick databases to yield a certain type of application.  Rainer von Ammon, in response to my Blog, provides his opinion that typically organizations are purchasing an industry oriented solutions and not technologies (which is true in many cases, but not all),  Paul Vincent asks in his Blog whether "CEP is just a supporting act?"  concluding from TIBCO's experience that the answer to this question is negative.     Marco Sierio asks where is CEP heading and preaches for CEP 2.0 that will be based on abstractions and will not have rules or SQL as a basis.  Some other comments to the Blogs ask about metrics to determine when EP technology should be used and when other technologies.

So - what can we learn from all these?     I am not sure that we can learn anything new that was not already discussed, actually such discussions return in cycles from time to time.   When we complete the document that we started to generate in the Dagstuhl seminar (it will probably take 2 months or so), we'll articulate some of this.   

There are three basic issues:  Does event processing has vitality as a technology?   Is it define a stand-alone market?  and how do we go about the 2.0 generation?

If we look at the considerations for the use of event processing as a technology, alternatively to hack in other technologies or just hard code the functionality ad-hoc for applications, one of the consideration that was mentioned is the ability to deal with high throughput of events, which is not a trivial task to achieve with hard-coding or regular technologies.    However, it seems that experience got us to realize that the more noticeable benefit is the TCO.
Dan Galorath produced the TCO chart seen above related to software,  there are also evidences that the software development in event-driven application achieved substantial reduction (sometimes in ratio of 1:4) relative to conventional solutions.    In the last DEBS conference somebody remarked about the Fast Flower Delivery use case that is discussed as a pivot example in the EPIA book that this is a BPM example, since the  event processing network looks to this person as a workflow (it has totally different semantics!), so my challenge is -- go ahead and implement in with a BPM system and then we'll compare the development time.  
As seen from the chart, the software maintenance contributes much more to the TCO than software development, and here the use of higher level abstractions that leads to ease of change intensifies the difference.   Rainer quotes somebody who says that there are charlatans in this area, true - there are charlatans in every area,  when relational databases emerge, all of  a sudden, implementations of relational databases were provided by people who did not understand what a relational database is (not just flat files!) and did not understand that they did not understand what a relational database is -- nothing is new.   Investment in development in various solutions is something that can be measured.

As for the other question - whether event processing is a business as a stand-alone,  I have already referred to it in the previous post, the answer is -- yes, for certain types of customers and applications, and embedded technology both in other technologies and applications.   My guess is that the embedded mode represents higher portion of the market that will be still growing.

As for CEP 2.0 --- here I agree with Marco that the next generation should not be incremental.  In the EPIA book we have introduced some abstractions that are independent of the implementations of the first generation, we are exploring them now as a basis for the second generation.  I guess that this is also a challenge for the research community.  More -later



Rainer v. Ammon said...

(Before I start, please let me say that you should change colours, at least with IE it's nearly not readable)

Just a comment regarding EPIA book and the application FastFlowerDelivery:

The colleague at DEBS seems not be really wrong when he said that this application would actually be a BPM application.
In BPM we distinguish between "Orchestration" and "Choreography", but not all BPM tools and older execution languages like older BPEL releases support choreography as a standard.
Let your students try to model FFD with all the needed (sub) processes e.g. by BPMN, also by swimlanes and pools, or by other approaches like Subject Oriented BPM from jCOM1 in order to model the collaboration of the different parties and then discuss the result and what should be missing.

If you would like to implement a FFD de Luxe:-), you would need a (future, so far not standardized) edBPM platform according to our first sketch of a taxonomy, which we tried together with your former Master student Alex Kofman and my two Thomases-colleagues, once again http://www.citt-online.com/downloads/62750370.pdf.

What means, that CEP alone would not have a (business) value and also not really an advantage regarding speed of implementation etc., if we would have to implement the needed processes hard-coded e.g. in Java as some of your students did allegedly, in order to react on event patterns. Even the maintenance, adapatability, flexibility of changing a business logic would be impossible in real applications. Only edBPM would make sense.

Perhaps a big customer/adopter will allow me to report their use cases or applications in this round within the next days and why they decided to base all these challenges on an edBPM platform in the future. We intend to present this also at our workshop in Ghent in December "From Event-driven Business Process Management to Ubiquitous Complex Event Processing", if the workshop is accepted.

Opher Etzion said...

Hi Rainer. I am still playing with the new editor of Google Blogs, in my Google chrome browser the colors look OK, but based on your comments I changed to dark blue.

About the FFD -- the issue is not of just programming-in-the-large, but programming-in-the-small; if you implement it in any of the approaches you mention it means that you'll need to hard-code the pattern part, as there is no built-in event pattern matching in any of them; of course, you can always write is in Java, but this is inferior solution from TCO perspective.

If we look at the larger picture, there might be BPM system that will manage workflows and event processing system which does the event processing functions. Each of them has existence independent from the other one. You are right that there are quite a lot of cases in which doing some kind of integration between the two is required as you need both to satisfy the requirements. This is true for other combinations: EP + BI, BPM + DBMS, EP + BRMS and more.

So - repeating my basic claim, event processing is a technology that is intended to reduce TCO for certain kind of functions. If the entire application consists of these functions, then a stand-alone EP will be sufficient, however since most applications have hybrid requirements, it is beneficial to have EP integrated with other technologies, one of them is certainly BPM.



Rainer v. Ammon said...

colours are perfect now, also with IE:-)