On actions and success criteria in event processing

I am now in Durham, North Carolina, the second stop in the USA business trip (that included 4th of July dinner with friends).   A reflection on one of the DEBS 2013 presentation given by Steffen Hausmann, a PhD student supervised by Francois Bry about "complex actions".   The model built specifies an action model for event processing.  The interesting insight is that unlike traditional systems where the success is determined by the successful completion of a task, event processing systems are goal-driven, an action goal is a reaction to some situation, and is intended to satisfy some goal with respect to this situation, for example: resolve a problem.   Thus in the presented model each action has success criterion that determines whether the action satisfied the goal.    In the presentation this criterion is an event, typically emitted by a sensor, but this can be generalized to a more comprehensive model.   The idea to define success criteria for actions as part of event processing system is very useful, and in a sense closes the loop of sense and respond.   

On DEBS 2013 grand challenge

The DEBS challenge was introduced first time in DEBS 2011, since then it also became "grand". 

This year it was around football (or soccer for the USA guys) and consisted mainly on aggregations, but also detection of proximity between player and ball, using sensors.   There was a lot of interest this year, and 14 submissions.  It becomes a "community building" theme.    6 solutions arrived to the final and were invited to present at the conference, a lot of effort has been dedicated, and we saw very nice and creative solutions.  The main criteria was performance that had to be measured on a given data-set.    In the business meeting I raised some ideas to make it even more interesting, by given "unseen" data at the conference itself, and introduce last minute change in the specification.     The conference is now at its last session.  Next year - DEBS 2014 is planned to be in IIT Bombay, India.

More impressions on the last two days in DEBS 2013

The social event of DEBS 2013 was a guided tour in the Dallas Cowboys stadium, and a dinner in a nearby restaurant.  Here are some subjective highlights from the last couple of days -- I captured some isolated points, not the entire picture:

1. An interesting paper on intra-query parallelism, which in my terminology is the optimization of a single event processing agent, after filtering-in only the relevant events.  This paper came from ETH Zurich.

2. Zohar Feldman - member of our own team gave a talk about "proactive event processing in action", discussing application in the logistic area, as part of the industrial session.

3. Another interesting talk in the industrial session was delivered by Mauricio Arango from Oracle -- on mobile QOS management using EP.   Seems that Oracle is advancing its spatial event processing.

4. The second keynote -- on smart grid platform by David Wollman from NIST,  talked about platforms, standards and challenges

5.  There are two topics I'll write in follow-up posts: the grand challenge session, and a paper about complex actions.  

On DEBS 2013: First keynote speaker - Roger Barga from Microsoft

Roger Barga has been an excellent choice (of the organizers) for first keynote in DEBS 2013.

He has a wide perspective from the multiple roles he occupied over the last few years in MSFT.   

Roger mentioned Thomas Kuhn's seminal work on the structure of revolutions - saying that there are two competing forces, those who push for paradigm shift and those who resist them and try to find "good enough" way to resolve everything in the old paradigm where they feel comfortable.   The use of event-driven thinking is just a paradigm shift, this is consistent with our tutorial given in the previous day.  

Some comments I took while listening to Roger:

1. Evolution of analytics:  analytics 1.0 - descriptive analytics based on warehouses, analytics 2.0 - big data, with NOSQL and Hadoop, we are going to analytics 3.0  -"rapid analytics with business impact".    Hadoop will become a niche technology, and real-time analytics (based on event processing) will take over. 

2. Some examples: Rols Royce is giving "engine up-time" as a service instead of selling aircraft engines, with a lot of instrumentation for maintaining the engines.  Other examples in the area of telemetry.  Some companies are making huge investments on real-time analytics.

3. Today there are two types of analytics: operational analytics  based on the speed of business, but use little information to get decisions;  investigative analytics based on the speed of  data scientists, and is based on a lot of information. There is a gap between the two that need to be unified.

4.  People don't know how to make use of new technologies to find new useful applications and tend to apply new technologies to old applications -- example: the first TV area was just visual radio, until the industry learned that TV opens new opportunity.

5. Bottom line: there are velocity pressure to do real-time analytics, but it requires paradigm shift and education.  Very compatible with our conclusions.   More about DEBS 2013 - later.

DEBS 2013 -- the tutorial day

Arlington Texas.

DEBS 2013 started today with the tutorials day.   In the morning I have attended a tutorial given by my IBM colleagues (past and present) members of the System-S team who gave a tutorial about stream optimizations.   The presentation was not posted yet -- so I'll write more when it does.   The interesting and useful concept they introduced is a catalog of optimization techniques along with the conditions when such an optimization might be applicable.   Some of the methods in the catalog are: operators reordering,  fusion, operator separation, and fission (defined as - replicate operator for parallel processing).     Very interesting talk.  As an observation, the talk covered "black box optimizations" - where the code of the operator itself is touched.   There are other efforts on "white box optimizations" like the one we presented in DEBS 2011. 

In the second half of the day - Jeff Adkins and myself presented our tutorial about event driven thinking.  

I'll write about it after the tutorial slides will become public.   

Tomorrow --  the first paper and industry sessions.   

Prelude to the DEBS 2013 tutorial

The notable event of this week in our family was the wedding of our eldest daughter Anat. In the picture you can see Anat, her new husband Adi with my wife and myself, in fancy dresses.

It was also very loaded week at work, one of the items has been submitting extended abstract of the  tutorial that Jeff Adkins and myself are  planned to deliver in DEBS 2013.     So here is the tentative outline we submitted in the original proposal:

Outline  

Topic I:  Introduction – Brief history of event processing  

Topic II:  The major differentiation factors of event-based thinking

Topic III:  The ontology of events and event influence

Topic IV:  Anatomy of reactive systems 

Topic V: Pragmatics – a computational independent model for event-based systems 

The tutorial starts with follows some of my previous postings about "event oriented thinking",   and Jeff's 4D classification.  It analyzes the way that people think about systems in the conventional way vs. the way people think in event-driven fashion, and gets into the role of events in language, and in system modeling.  

Unlike the implementation oriented thinking we usually employ -- this is a computational independent thinking, it looks at the event-based systems from the customer's perspective.     See (at least some of) you in Arlington.

DEBS 2013 - keynotes and tutorials were published

DEBS 2013 will take place in the campus of the University of Texas Arlington, in June 29- July 3.

Today the keynotes and tutorials were published on the conference's website:

The keynote speakers this time will be: Roger Barga from Microsoft, one of the first persons in Microsoft Research who worked on event processing, and crossed the line to the product organization and deals now with product management.  Roger will talk about "the rise of the velocity pipeline in enterprise computing" that will focus on the velocity part of big data that makes batch solutions like Hadoop as inadequate.

The second keynote speaker is David Wollman who manage the smart grids standard activities in  NIST  will talk about smart grids.

There will be four tutorials - all of them of known people active in the area.

The first one will be given by my IBM colleagues from the System S team who will talk about "stream processing optimizations".  

The second one will be given by Christoph Emmersberger  an Florian Springer (both of them I know from their past association with Rainer von Ammon), who will talk about the event processing capabilities of Apache Camel.

The third one will also be given in German accent, by Boris Koldehofe and  Frank Dürr from University of Stuttgart. They will talk about "Software defined networks".

Last -- keeping the tradition, I'll be giving a tutorial this year also, this time together with Jeff Adkins, on a topic that we are both dealing - "why is event driven thinking different from regular thinking about computing". I'll write about this tutorial at a later phase (well, we have to prepare it first), meanwhile you can read the short abstract on the site.  Hope to meet old friends and colleagues in Arlington.   

More - later.

DEBS 2013 grand challenge was announced

DEBS 2013 grand challenge was published today.  This is one of the new items we introduced in DEBS 2011 which continues to the next generations. The idea is to publish an event processing application, view various implementation and evaluate based on announced criteria.

This time the application is around football (soccer in American English), where the input events come from sensors attached to players' shoes and hands and to the ball, and some referee events.   The events are spatiotemporal by nature. 

The goal is to calculate various statistics about players performance, ball's location etc.

Criteria -- application's performance (latency,  throughput),  while keeping correctness. 

I hope that various teams will answer this challenge and will provide interesting solutions.