Event processing as reducer in Map Reduce

A relatively recent posting from MSFT.   In this posting there is a question about the relationship between the batch oriented Map Reduce,  and the on-line oriented event processing.   The answer, according to MSFT is - event processing can be used as a reducer in the Map Reduce, where there are multiple copies of an event processing engine perform the reduce function.


I have written before about offline event processing,  with the insight that event processing is useful not only in online, but in offline, since it provides both efficient implementation and high-level abstraction in certain functions (pattern matching, aggregation and more) that makes it also attractive to use in batch.  


Of course, another synergy may be using event processing within real-time hadoop, like Darkstar.  as is frequently articulated by Colin Clark.    

EPTS activities in 2012

In Wednesday we held an open conference call of EPTS members.  The call discussed EPTS activities for 2012.  While in the past EPTS concentrated in working groups that have created glossary, reference architecture, investigated use cases and languages, and some other stuff, the idea is that in 2012 EPTS will attempt to address larger audience by creating online magazine which will consist of:  news (including connecting to social media),  general material about event processing (starting point: FAQ taken from analysts in this area),  and industry specific content.   We are now looking for volunteers to set up all aspects: orchestrating the content ("editorial board"),  soliciting readers and topics they are interested ("reach out board"), and a coordinator for sponsorship and advertisements.    Anybody interested to participate - please let me know.   I'll post updates in this Blog

Book review: The Live Web by Phil Windley

I am browsing through the book "The Live Web" by  Phil Windley, whose sub-title is "Building Event-Based Connections in the Cloud".   Phil sent me the book (with handwritten dedication), and starting from the back cover I liked his motivating examples: "Imagine a world in which your phone automatically mutes the ringer when you start watching a movie" (actually today my mobile phone rang when I was giving a talk, forgot to mute it) or another one  "imagine a workd in which your alarm clock sets itself based on your schedule and other information like weather, traffic, and your past behavior" (I don't need much sleep so getting up early without alarm clock, and some members of my family have the habit to turn of the alarm clock and get back to sleep, but I am sure it is helpful for some people). 
This reminds me of many examples that I used over the years to explain either event processing or autonomic computing (e.g. the refrigerator invited the technician and more).   Phil's answer to realize these scenarios is what he calls "the live web", which according to him is the evolution beyond Web 2.0 which is still static web. I think terms like "the active Internet" or "The event web" was used before in a similar context.  Much of the book deal with projection of event processing to the web interfaces,and working in web environment. Some of it is dedicated to events, their semantics, and operations around them.  He also dedicates chapter to the current hot topics - cloud and mobile and connects them to the story of the live web.   Phil also founded a start-up called Kynetx to implement these ideas, the book also describes their system and language in detail.
Overall - interesting book, both for those who want to understand the principles and those who wish to drill down on the details of how to build such applications (or learn the Kynetx stuff)...   It is also another book that explain the principle of programming with events -- although we somewhat different perspectives of the other books. 

On transdisiciplinary education

One of my IBM colleagues, Jim Spohrer, the IBM University Relations Director,  has written about the need of students to receive transdisciplinary education.   Indeed we find more and more that doing things require not just multidisciplinary skills, of people from several disciplines working together, but also transdisciplinary skills, where people have good skills in several disciplines and a single person knows how to integrate these disciplines.  This is, of course, contrary to the academic tradition, where promotion processes for faculty members advocate the association with a single well-defined discipline.    We can see much more  transdisciplinary programs today then 20 years ago.  I believe that a program in which a student learns combination of computer science with another discipline (biology, business, mechanical engineering, cognitive psychology and more...),  not only at the intersection between the disciplines, but rather at the union, of having deep understanding in both disciplines as the type of skills required in the marketplace.  
  

The pilot decision making process

While surfing the web to find something,  through the wonders of Google search, I reached the site of the Langley flying school and came across the illustration above under the title "the pilot decision making process". 
The cycle starts with a situation, which is defined as a critical situation which requires action.   The detection and identification of such a situation is what starts the process, thus the decision process is inherently event driven, then the pilot needs to assess the options,  sometimes the options are very obvious, and sometimes there is a creativity in that process,  then there is a need to chose among the options, act according to the selected action and assess whether the situation has been resolved.   Note that this may be either reactive (situation already happened) or proactive (situation is expected to happen).  Of course, when possible it is better to be proactive, it is much better to eliminate crash of the aircraft then try to rescue the crew and passengers when the crash already happened, both from economic point of view and from safety to human life point of view.     Note that the pilot needs to react to critical situations, for the process-driven business as usual case, an autonomic pilot is sufficient.    Can we extend the autonomic pilot also to deal with critical situations?  

On enrichment - and the difference between BRMS and EP

A recent article in the IBM developerWorks discusses two ways to enrich data used for rules from external databases,  one of them is doing the enrichment in the request level, before calling the "decision server" (which is the current name for using BRMS system using the request-response protocol), the other one is doing enrichment during the rule processing itself. The article describes how each of these options is done and also discusses pros and cons,  the benefits of enrichment by the request level are - less complexity, and better performance of the rule component; the benefits off enrichment at the rule level are - handling dynamic data and more specialization for the exact data that is being used by the rule.


Thinking about event processing -- there is similarity to the BRMS case,  event can be enriched both by the event producer and as part of the event processing itself, the arguments are not far from those in the BRMS case,  there is one fundamental difference in event processing --  the work is not done using the "request-response" protocol,  moreover, the different part of the system are decoupled, thus the event producer does not necessarily know what purposes the event is going to be used, thus there may be different types of enrichment needed for different uses.   The dynamic aspect is applicable here, and there may be some race conditions in highly dynamic systems between updates in the database that was enriched and its use in enrichment, unless the event processing enrichment system locks the data in the database until it is being used in the event processing system, which requires the event processing system to exhibit a transactional behavior for part of it, but I'll not get now into this issue,


Bottom line:   The considerations in event driven architecture are somewhat different than the request-response systems that are the most common one in computing.   

Timo Elliot's presentation on "Business in the moment - from reactive to proactive"

Timo Elliot from SAP gave a recent talk in the Gartner BI meeting in London entitled "Business in the moment -from reactive to proactive". You can download the presentation from a link in Timo's Blog posting.   In a following post on his Blog,  Timo refers to an FreshDirect explaining the proactive behavior:

“FreshDirect has an operations center that manages its fleet of delivery trucks. In a large metropolitan area like New York, traffic doesn’t always flow predictably. A traditional approach to BI would be to print a report showing the level of on-time deliveries (OTDs) the day before and then ask the transportation department what went wrong for the orders that were delivered late. FreshDirect uses analytics in a more impactful way.”

“The company monitors the delivery rate of every truck and enters that data into the BI system on an ongoing basis. Every hour, it uses the previous hour’s data to predict how many deliveries will be on-time in the next hour. If the predicted OTD rate is below FreshDirect’s target, the company sends out an auxiliary truck or trucks to help make deliveries. The company holds 10 trucks in reserve for just this purpose.”

I'll bring more proactive stories when I'll find out about them...