Revisiting the FFD example in the EPIA book - call for contributions

In 2009 when Peter Niblett and myself wrote the "Event Processing in Action" book we came with an example called "Fast Flower Delivery" that accompanied the book.   it seems that the book is still being used (it was cited so far 276 times),  one idea that came recently in discussions with the publisher is to refresh the implementation page.   We now issue a call for implementations.   All owners of event processing tool (commercial or open source) is called to show how this example is developed in their tool so the reader (and interested general audience) will be able to view it.   In order to qualify one has to create a webpage that contains - source code of the use case with any additional reference to the language and product behind it.  

The book's webpage will include links of all such solutions.  For more details -- please contact me.  

The full details are the FFD example are linked here. 

Another implementation of the "Fast Flower Delivery"

In the EPIA book, we had a running example used for demonstrating all constructs in the book, the example described a scenario called:  "Fast Flower Delivery".    During the book writing we approached the event processing community and issued call for implementations, there has been six implementations that were ready  during the book's writing:  Aleri (currently Sybase),  Apama(Progress),  Esper, Etalis,   ruleCore and Streambase.    It seems that more implementations are being devised,    I was asked for permission to use the "Fast Flower Delivery" scenario as the running example in an upcoming book teaching the use of one of the products,  will write about that when this book will be out. 


Recently,  an implementation of this scenario in IBM Websphere Business Events (WBE) was posted on IBM developerWorks as a tutorial to teach the use of that product.  
Seems to becoming the "Hello World" of event processing.  

On the production phase of the EPIA book

I have co-edited some "collection of articles" books that are common in the research community, there is certainly some logistics associated with it, but relative to producing a "mass market" book it is a children's game comparing with the production of the EPIA book that I have been writing together with Peter Niblett. The publisher of our EPIA - Manning, has various ways to ensure quality. Manning is a publisher focusing on computing related book, thus it views book development as a software development project (and indeed many of its books are code-driven). While the book was initiated by Manning and not by us, they have sent the outline we sent to bunch of reviewers, all of them people with deep knowledge in event processing. Some of the advices we got were very useful. Since we wrote in the outline that we are going to base the book around a single example, the advice of one of the reviewers was that we'll use an example that everybody can understand, the reviewer added that if we would chose an example from the financial services domain (as many of the EP papers are doing), we might create a communication obstacle with some of the readers, who will not be familiar with the terms. We followed this advise and created the "Fast Flower Delivery" (FFD) that already received multiple implementations in many languages. For each 1/3 of the book, we had a milestone in which they sent the book to a bunch of reviewers (in one of them there was a response from 14 reviewers). The reviewers were mix - some people who have deep knowledge in event processing, and some who don't know what it is, both types of reviewers sent various types of comments, some of them resulted in restructuring of the book (originally we planned 15 chapters and 3 appendices, we ended up with 12 chapters and 2 appendices, but with around 100 pages more than we originally planned)- This was just the development process, and it last for 15 months. A few weeks ago we have started the production phase -- it has a workflow (supported by electronic content management system): Another technical review by a "technical proofreader" - here we looked at PhD student in this area, and thus addressed faculty members active in this area; the person who did this job is Samujjwal Bhandari, a PhD student in Texas tech university, he has read all chapters, made comments and caught various cases of inconsistencies among the different chapters, then it moves to the copy editor who makes editorial modifications, and returned to us with many comments and questions that we had to answer, after doing this round, each chapter is going to the proofreader, who is doing another editorial pass, and then comes back to us with questions and comments, so we are doing another pass on all chapters. After that it goes to the typesetting. In the background there is a graphical work to redo our amateurish figures in a professional way, and then hopefully the book will be ready. Currently we passed the phases of the copy-editing, and now working with the proofreader. There are a lot of editorial rules, and style issues that have to be dealt with (Peter is much better than me in noticing the small details), and in fact -- amount of work is much higher than I anticipated, it consumes much of my free time for over a year now, as we both are doing it in addition to our daily work -- but I hope the result will justify the investment. More about the book - later.

Revisiting race condition with FFD example

In the past I have written about race conditions and this triggered some responses. We recently realized that in the example we created for the EPIA book (the Fast Flower Delivery and has got already around ten different implementations, six of them can be viewed on the book's webpage, some more will be added) there is an case that if will not be handled carefully may yield wrong results due to race conditions. Here is the case:

There is an aggregate EPA per driver and day that collects assignment events for a driver and in the end of the day creates a derived event which counts the number of assignment per driver, there is a second EPA per day that collects all the drivers count at that day and calculates mean and standard deviation for the number of assignments per active drivers in that day; there is a third EPA, again per driver and day, which gets the derived events from the first two EPAs and calculate for each driver its deviation from the mean, in standard deviation units. These three EPAs are all aggregation type EPA which has some order among them, until now -- no problem. Now, the issue is that all these calculations occur at the end of the day, and have causal dependencies. If we are not careful, the first EPA calculates the count per driver at the end of the day, but until it finishes the calculation the time is say, 12:01, so the result is classified to the next day, but it is required to calculate the statistics for this day, and then if it gets into the statistics of the next day, then we get some inconsistency in the system. Obviously a naive implementation will get wrong results here. There are various ways to handle it and ensure correctness, however the main issue is whether the developer needs to be aware of it while designing the application, or the compiler that takes the definition of these EPAs and creates the actual implementation should be the one which will do the job. My opinion is that if the developer will have to take care of such things in hard coding, the life will be quite difficult, as this is only one case of race condition, and it is better that it will be transparent to the developer. This will eat the cake and have it too --- both using high level tool that makes the programming easier and lower the total cost of ownership, and fine tune the semantics in a way that require typically dedicated, and even complicated programming. More about other aspects of semantic fine tuning - later.

On the Fast Flower Delivery example and various programming styles of event processing

As explained in one of the previous postings, we are using a single example in the EPIA book as an example that accompanies the book, this is part of the book's methodology. I'll write more about the methodology more, as we are now writing the preface for this book explaining the methodology (among other things). Actually today I received an additional review of the book from the publisher and the reviewer has criticized the example claiming the since most of the readers are men, example that relates to flower may be considered as too feminine, and realizing that it is too late to change, suggested that we'll select another example in the second edition.

Well, I am thinking what will appeal to the real machos.

Maybe we should go for Poker example -- this is a real macho staff...

On second thought, the real machos are engaged in boxing, so maybe we should have an example around boxing match... go figure...

Getting serious now. The FFD (Fast Flower Delivery) example is explained in the book using our building block approach, it is also demonstrated using several languages. I have approached the entire community earlier this year, and there has been a very good willingness of participating in this game, implementing the FFD example in various languages by the "language owners". We have six languages participating in the game now. Languages implemented by four commercial products:

• Aleri (actually the CCL language originally Coral8)
• Apama (owned by Progress Software)
• Rulecore (a Sweden based company)
• Streambase

and two open sources:

• Esper
• Etalis

The reader will be able to look at the example implemented in these six languages; furthermore, will be able to download a full or demo version of the engine implementing this language. As written before the logistics of constructing this website, validating the solutions etc... are done by students taking my event processing course. Some examples will be brought in-line to the various chapters of the book to provide the readers some glimpse of the different styles.

We should have a "Beta version" of this website within a couple of weeks.

I'll update about this experiment more.