Saturday, March 10, 2012

Event processing as reducer in Map Reduce

A relatively recent posting from MSFT.   In this posting there is a question about the relationship between the batch oriented Map Reduce,  and the on-line oriented event processing.   The answer, according to MSFT is - event processing can be used as a reducer in the Map Reduce, where there are multiple copies of an event processing engine perform the reduce function.

I have written before about offline event processing,  with the insight that event processing is useful not only in online, but in offline, since it provides both efficient implementation and high-level abstraction in certain functions (pattern matching, aggregation and more) that makes it also attractive to use in batch.  

Of course, another synergy may be using event processing within real-time hadoop, like Darkstar.  as is frequently articulated by Colin Clark.