On actions and success criteria in event processing

I am now in Durham, North Carolina, the second stop in the USA business trip (that included 4th of July dinner with friends).   A reflection on one of the DEBS 2013 presentation given by Steffen Hausmann, a PhD student supervised by Francois Bry about "complex actions".   The model built specifies an action model for event processing.  The interesting insight is that unlike traditional systems where the success is determined by the successful completion of a task, event processing systems are goal-driven, an action goal is a reaction to some situation, and is intended to satisfy some goal with respect to this situation, for example: resolve a problem.   Thus in the presented model each action has success criterion that determines whether the action satisfied the goal.    In the presentation this criterion is an event, typically emitted by a sensor, but this can be generalized to a more comprehensive model.   The idea to define success criteria for actions as part of event processing system is very useful, and in a sense closes the loop of sense and respond.   

On DEBS 2013 grand challenge

The DEBS challenge was introduced first time in DEBS 2011, since then it also became "grand". 

This year it was around football (or soccer for the USA guys) and consisted mainly on aggregations, but also detection of proximity between player and ball, using sensors.   There was a lot of interest this year, and 14 submissions.  It becomes a "community building" theme.    6 solutions arrived to the final and were invited to present at the conference, a lot of effort has been dedicated, and we saw very nice and creative solutions.  The main criteria was performance that had to be measured on a given data-set.    In the business meeting I raised some ideas to make it even more interesting, by given "unseen" data at the conference itself, and introduce last minute change in the specification.     The conference is now at its last session.  Next year - DEBS 2014 is planned to be in IIT Bombay, India.

More impressions on the last two days in DEBS 2013

The social event of DEBS 2013 was a guided tour in the Dallas Cowboys stadium, and a dinner in a nearby restaurant.  Here are some subjective highlights from the last couple of days -- I captured some isolated points, not the entire picture:

1. An interesting paper on intra-query parallelism, which in my terminology is the optimization of a single event processing agent, after filtering-in only the relevant events.  This paper came from ETH Zurich.

2. Zohar Feldman - member of our own team gave a talk about "proactive event processing in action", discussing application in the logistic area, as part of the industrial session.

3. Another interesting talk in the industrial session was delivered by Mauricio Arango from Oracle -- on mobile QOS management using EP.   Seems that Oracle is advancing its spatial event processing.

4. The second keynote -- on smart grid platform by David Wollman from NIST,  talked about platforms, standards and challenges

5.  There are two topics I'll write in follow-up posts: the grand challenge session, and a paper about complex actions.  

On DEBS 2013: First keynote speaker - Roger Barga from Microsoft

Roger Barga has been an excellent choice (of the organizers) for first keynote in DEBS 2013.

He has a wide perspective from the multiple roles he occupied over the last few years in MSFT.   

Roger mentioned Thomas Kuhn's seminal work on the structure of revolutions - saying that there are two competing forces, those who push for paradigm shift and those who resist them and try to find "good enough" way to resolve everything in the old paradigm where they feel comfortable.   The use of event-driven thinking is just a paradigm shift, this is consistent with our tutorial given in the previous day.  

Some comments I took while listening to Roger:

1. Evolution of analytics:  analytics 1.0 - descriptive analytics based on warehouses, analytics 2.0 - big data, with NOSQL and Hadoop, we are going to analytics 3.0  -"rapid analytics with business impact".    Hadoop will become a niche technology, and real-time analytics (based on event processing) will take over. 

2. Some examples: Rols Royce is giving "engine up-time" as a service instead of selling aircraft engines, with a lot of instrumentation for maintaining the engines.  Other examples in the area of telemetry.  Some companies are making huge investments on real-time analytics.

3. Today there are two types of analytics: operational analytics  based on the speed of business, but use little information to get decisions;  investigative analytics based on the speed of  data scientists, and is based on a lot of information. There is a gap between the two that need to be unified.

4.  People don't know how to make use of new technologies to find new useful applications and tend to apply new technologies to old applications -- example: the first TV area was just visual radio, until the industry learned that TV opens new opportunity.

5. Bottom line: there are velocity pressure to do real-time analytics, but it requires paradigm shift and education.  Very compatible with our conclusions.   More about DEBS 2013 - later.

Our DEBS 2013 tutorial: why is event-driven thinking different than traditional thinking about computing?

The tutorial that Jeff Adkins and myself delivered in DEBS 2013 yesterday is now posted on slideshare.  

Many of the participants reacted with further discussions.  The first take away we wanted to convey is described on this slide.   While the community continues to work on advancing the technology (which is fine!), the main challenge today is not the technology,  but making the paradigm shift in people's mind about 

event-driven thinking.  Indeed, experience shows that a typical developer when facing with an application which we classify as event-driven would tend to implement it in a traditional way:  treat the events as data, store them in a database, and process them in request/response mode.   The tutorial discusses the differences and explains the ontology of event-based system as we see it.  It also has a short description of our recent work around the "Event Model", on which I'll write more in the future.  

More about DEBS 2013  --- soon. 

DEBS 2013 -- the tutorial day

Arlington Texas.

DEBS 2013 started today with the tutorials day.   In the morning I have attended a tutorial given by my IBM colleagues (past and present) members of the System-S team who gave a tutorial about stream optimizations.   The presentation was not posted yet -- so I'll write more when it does.   The interesting and useful concept they introduced is a catalog of optimization techniques along with the conditions when such an optimization might be applicable.   Some of the methods in the catalog are: operators reordering,  fusion, operator separation, and fission (defined as - replicate operator for parallel processing).     Very interesting talk.  As an observation, the talk covered "black box optimizations" - where the code of the operator itself is touched.   There are other efforts on "white box optimizations" like the one we presented in DEBS 2011. 

In the second half of the day - Jeff Adkins and myself presented our tutorial about event driven thinking.  

I'll write about it after the tutorial slides will become public.   

Tomorrow --  the first paper and industry sessions.