Event processing = big data + real time ?

BIG DATA is a hot phrase, one can also be labelled as "BIG DATA NERD" by purchasing the shirt shown above.   The term BIG DATA refers to the explosion of data in the universe, especially methods to store and process huge amount of data.   


Paul Vincent poses the question:  CEP = real-time bid-data?     Actually he doesn't answer this question on his blog posting, but the spirit of the postings makes the impression that the "=" relations is evaluated to "True".    


The answer is that it is not really an equality, but more like an intersecting concepts.   


Event processing of course does neither necessarily works on big data nor it necessarily has to satisfy real-time constraints.


On the other hand,  real-time big data, is not necessarily event-driven and process events at all, as big data can be fairly static.    As an example for real-time big data, I'll return to an invited talk in DEBS 2011 on Watson, in which there has been a live demonstration of  the Jeopardy! game with the Watson computerized system, you can see in the picture that Paul Vincent is playing (very skilfully) the host.  

Watson is a "big data" crunching application, it has very strict real-time constraints, so it certainly can qualify as "real-time big-data", alas, it has nothing to do with event processing, in fact the data exists prior to the game  and no additional data is added during the game.


While the equality does not work,  there is an obvious relation;   in some cases there are substantial number of data inputs, which much of it can be reduced, or filtered out.   Event processing can be used to process streaming data on the fly, and not follow the paradigm of store now process later.   This is important especially one the processing is required in real-time.


More on event processing and big data -- later

On event processing critical success factors

Some other input related to student's work, is the posting of Sascha Retter, based on his diploma thesis (the term discloses the origin in Germany).   The observation is about critical success factors of event processing systems -  integration with workflow management system, and modeling tools.   There is some truth in both observations, but I would like to take broader view. 


There is a market for stand-alone event processing systems, but we have observed that the larger market is  event processing embedded in other systems,  workflow management systems, or in the more modern name "Business Process Management" (BPM) is one of these systems, but this is not the only one, there are both other systems in which event processing can be embedded inside, e.g. analytic frameworks, sensor networks and more.  It may also be embedded within packaged applications - like trading platforms, supply chain management and much more.  


Modeling tool are a vehicle for usability, and indeed usability and ease of use has been identified as a critical success factor.   This is now a well recognized fact.


There are other critical success factors for various applications, like non-functional requirements (performance).  I also believe that standards are critical success factor for the entire area.


The EPTS use case survey provides more insights into properties and preferences of event processing customers. 


  

On terminology again

While the EPTS glossary team is about to produce additional version of the glossary,  I received by Email a question about terminology from a student investigating this area, whether "event driven architecture" and "event processing"  (both titles of books) are indeed the same thing  


The answer is -- not really, but the difference is simple:   


Event Driven Architecture is an architecture in which there are event producers, event processors, and event consumers, with the following principles:  1). All players are decoupled;  2). All the communication between players is that they send events to one another:  3). All communication between players is asynchronous. 


Event Processing is a type of processing that takes as input one or more events and produce as output one or more event applying functions such as: filtering, aggregation, transformation and pattern matching.   


The concepts are related in the sense that it is common to implement event processing on top of event driven architecture, but it is not tightly coupled:
Event Driven Architecture can implement  pub/sub systems that are doing routing only and do not do any processing of events;
Event Processing can be be implemented in synchronous mode and not on top of  EDA.

Another implementation of the "Fast Flower Delivery"

In the EPIA book, we had a running example used for demonstrating all constructs in the book, the example described a scenario called:  "Fast Flower Delivery".    During the book writing we approached the event processing community and issued call for implementations, there has been six implementations that were ready  during the book's writing:  Aleri (currently Sybase),  Apama(Progress),  Esper, Etalis,   ruleCore and Streambase.    It seems that more implementations are being devised,    I was asked for permission to use the "Fast Flower Delivery" scenario as the running example in an upcoming book teaching the use of one of the products,  will write about that when this book will be out. 


Recently,  an implementation of this scenario in IBM Websphere Business Events (WBE) was posted on IBM developerWorks as a tutorial to teach the use of that product.  
Seems to becoming the "Hello World" of event processing.  

On Watson and Event processing

This is a picture taken in the DEBS conference last week, we had an invited talk by Eddie Epstein from the Watson team, and he ran a round of Jeopardy!, two of the conference participants against Watson, here you can see that Watson has -800 point (click on the picture to see it clearly), but it recovered and won.   We invited the talk on Watson, not because it somehow related to event-based systems, but because the conference took place in the Yorktown auditorium, where the famous Jeopardy! game in which Watson succeeded to beat two of the all time champions was recorded.


Today I've found a Blog posting by Shalin Shah from Vitria, with the promising title:  

IBM’s Watson: What Does Complex-Event Processing Mean For Customer Experience Management?

So I tried to understand what the author thinks is the relationship between the  two,  the answer according to the posting - both of them can be used for operational intelligence.

Indeed, there are now efforts within IBM Research, to determine what are the next steps, since in essence Watson is "deep question answering machine" there are some areas that seem to be killer applications of this technology, among them are: medical diagnosis and helpdesk/contact center in which agents need to answer questions in a lot of areas. There are some others as well.   

From technology point of view, Watson works in a different paradigm relative to event processing.  It is not event-driven, but is based on a knowledge stored in books, encyclopedias, and other sources.  What it does in real-time is - question understanding and question answering using statistical reasoning, and massive computational power.   

The interesting question is what can be a synergy between question answering machine and event processing,  here we can think of two sides:   an event processing system is being assisted in Watson-like system in order to determine contextual information that can be used for evaluating assertions, or classify events into context instances.   On the other hand a question answered can trigger event.  Or the question answering system can be monitored by an event processing system.     One can also think about real-time update of Watson's knowledge-base as a result of event, which is not the way Watson currently works.    I think that there are various more synergies between the two types of systems.

More - later.

Event processing as analytics

Recently, I hear more and more that people are classifying event processing as a kind of analytics. 
This is partially due to hype that exists around analytics, and partially due to taking the word analytics in more broader term  that denote general use of  computerized quantitative tools beyond the traditional use of statistical processing.   In sense it also reflects the fact that event processing is in many cases used as OEM inside sophisticated solutions, and not sold as a middleware  per se.     Is it the right classification?  --- there are pros and cons,  but linking to a hype seems to be a good marketing strategy especially towards people who don't know what it is.    From research point of view, it is certainly a distinct discipline, though there are synergies.  I'll write more about the differences and synergies in follow up postings.

DEBS 2011 awards

Back in my office now from the DEBS trip,  after spending Saturday in NYC and watched the matinee' show of Wicked, a wonderful musical. 

Last remaining fact about DEBS 2011 is that it is the first instance of DEBS to grant awards, the award granting ceremony occurred at the conference banquet's on Wednesday evening.  The awards are noted on the DEBS 2011 webpage.  Here is the list of awards and award winners:

Best Paper Award:	Gabriela Jacques Da Silva, Buğra Gedik, Henrique Andrade, Kun-Lung Wu, Ravishankar K. Iyer.
	Fault Injection-based Assessment of Partial Fault Tolerance in Stream Processing Applications.
DEBS Challenge Award:	The ETH team:Lynn Aders, René Buffat, Zaheer Chothia, Matthias Wetter, Cagri Balkesen, Peter M. Fischer, Nesime Tatbul.
Best Poster Award:	Nihal Dindar, Peter Fischer, Nesime Tatbul.
	DejaVu: A Complex Event Processing System for Pattern Matching over Live and Historical Data Streams.
Best Demo Award:	Sinan Sen, Ruofeng Lin, Bijan Fahimi Shemrani.
	Complex Event Pattern Evolution based on Real-Time Pattern Execution Statistics.
Best Idea in the DEBS Gong Show:	Mike Lefler.