Another type of proactive systems

Today I had a meeting with PhD student and her adviser to help them do sanity checks for a research idea related to event processing.    One of the ideas coming from them is actually looking at another type of proactive systems relative to what we have been looking at.   The difference stems from the fact that event producer can be of two types:  observer and creator.   An observer, e.g. sensor, observes that an event has happened and reports it, but it does not take any part in the event creation.   A creator is a producer that creates the event, and can decide whether an event is created, or in some cases can retract events.  Such a creator can be an instrumented program, BPM,  transaction system etc...

The proactive systems we have been looking at assumed that the producer is observer, thus we cannot control the creation of events but we can try to predict it, and eliminate it, or mitigate the damage it might inflict.   

Contrary, when the event producer is a creator, we can control the event.   There might be a few cases here.

One case can be that we again predict that event is going to happen since there is some causality relations with events that have already been detected, and within a certain context we don't want this event to happen,  we can then tell the producer -- if this event is being produced (again, within certain context, e.g. time window) -- don't produce the event.    Another case can be that the event is being created by a transaction system,  this event, within some context, and maybe in conjunction with other events, derives a situation that we want to eliminate.  The fact that the event is part of transaction that has not committed, gives the possibility to retract the event and eliminate it from being produced to the outside world when the transaction commits.  

The pure view of event processing puts producers and consumers outside the boundaries of the event processing system,  however, in reality there are cases where either the producers or the consumers (maybe even both) are controlled by the same logic, and thus detection of situation may impact the producers and consumers.   I think it gives rise to interesting applications.  I'll get back to this topic later.

DEBS 2012 - on Dave Maier's keynote, frames, and fragmantation

I have arrived home 49 hours later than planned,  in Sunday early morning instead of Friday early morning, due to a malfunction of the aircraft which required to replace a part, and a combination of the fact the the airline ELAL, does not fly in Saturday, thus, although the aircraft has been fixed in Friday, it was too late to take-off,  and the German liking to paperwork and red tape, which inflicted a long process of releasing this part from the German custom on Friday, and also delayed the takeoff on Saturday night -- since half of the passengers dropped out and found alternative ways to fly,  the aircraft was half empty, and the crew let people occupy the business class,  however, it turns out that the German authorities inspected the aircraft and found out that the number of people in the business class does not match the flight authorization request and wanted that the paperwork will be done again,  so after some minutes of trying to convince them. the crew asked all passengers moved to the business class to return to their seats until after the takeoff.  We in Israel has the completely opposite mentality, we are good in improvising, and find that doing things by the book is boring.  I must also say that ELAL treated us much better than a similar case I had in Delta Airlines,  which caused me not to fly Delta. 

Anyway -- back to DEBS 2012.  The organizers quickly assembled all presentations, and they can be found on the conference's website.   There are also pictures from the conference posted on Facebook, you can read through the presentations and find anything you are interested at.

I'll concentrate in the keynote of the third day - Dave Maier.  Dave is a senior figure in the database community, and got involved in recent years also in data streams research, he is also involved in related work with Microsoft.    

Dave gave an interesting talk whose title was:  "capturing episodes - may the frame be with you".  The rationale is that the notion of window in stream processing is too limited and the event stream is partitioned according to slices of times or event count,  but in different cases there is a need to slice the event stream differently,  such as: as long as an episode holds.    

While the notion of frame is indeed a useful abstraction if we take the pure data stream management primitives as given,  it seems that the fragmentation that exists due to people's starting point, results in duplicating work within different sub-communities, without being aware of work in other sub-communities.  More specifically, I think that the notion of frame has strong  relation to the notion of context that we have introduced, furthermore, I have written before that the notion of context also makes  the concept of punctuation redundant., The notion of punctuation also  came from Dave Maier (with his student Peter Tucker).

I am going to get deeper into Dave's paper and make more thorough comments on the relationship between "context" and "frame"  (and can add "fluent" in event calculus as a third term for comparison).  I also sent Dave some material on context (following a corridor chat), and he mentioned it during his talk. 

One of the good things that the relational model brought to the database community, was that it was accepted as a starting point (years before the standard approved),  and then let the research community focus on doing absolute new work, instead of relative new work,  which was one of the reasons to the big influence of the research community on reality in the database area.    We should strive to do the same in the event processing domain -- more later.  

DEBS 2012 gong show -- event processing as key to immortality

Peter Pischer, who chaired the "gong show" in DEBS 2012 was looking for volunteers to give a "gong show" talk.  The gong show idea, borrowed from other conferences, and added to DEBS last year, is a forum to present futuristic, compelling ideas  (which not need to have scientific proof).   As Peter approached me in the break to see whether I would like to present, I thought about a past discussion I had with Rainer von Ammon about the singularity idea,  and devised a quick presentation on a topic I don't understand anything about...  I have uploaded the presentation to slideshare --   enjoy! 

DEBS 2012 - our presentation on "basic proactive"

I have written before about the proactive event-driven paradigm that we are investigating in the last couple of years.   Today I have presented some new results in this area, the "basic proactive" model which provides end-to-end solution for a restricted type of applications with several characteristics.  The paper co-authors by Yagil Engel, Zohar Feldman and myself is in the proceedings (will probably be in the ACM digital library soon).   The presentation can be found on slideshare.

Since there has been a slight delay in the schedule, the session chair allowed only 2 questions, one by Alex Buchmann who wondered whether it is wise to introduce new buzzword  like proactive, or position it otherwise.  I think that there are pros and cons, but the buzzword is already there and I'll write about it.

Adrian Paschke, who is interested in the formal aspects, remarked that classical logic will not be sufficient here since conclusions are uncertain and can be retracted.  I answered that we'll probably need non-monotonic logic, maybe combined with some kind of quantitative logic of uncertainty (such as probabilistic logic).  If somebody want to take the challenge and work on the formal model -- let  me know.

More -later. 

DEBS 2012 - highlights of the first conference day

I'll start from the last session - the poster and demo session.  During this session I wore the glasses shown in the picture, and it was the first glance towards the world of the glasses that Google is taking us.   It has a potential, but getting used to it is not intuitive,  one can see menu and then has to put a paper in the place to select among the menu options.   The demo was of an EU project that does human attention detection and  detect whether a person is interested in museum artifacts.  There have been some other demos and posters, many of them from EU projects.  Another interesting ones were one that provide safety alerts for monitoring underground trains, and another one which use events to help collaboration between team of software developers who work together.  

Getting back to the beginning -- the day started with the interesting keynote of Sethu Raman (the industrial keynote), already written about yesterday.    The next session has been an industry session.  It is interesting to note that around 1/3 of the participants here are from industry and not academia,   and the industry track became integral part of the program (not all academic people like it though!). 

In the afternoon there were two scientific papers sessions -- one on pub/sub, the original topic of DEBS, now reduced to a single track per conference,  the other one on "complex and spatial events".   I'll write a few sentences about it - slides can already be found on the conference's site: 

• The first paper was presented by Martin Hirzel from the IBM System S team about parallel complex event processing, Martin started with the old assertion that CEP is part of stream processing, since it is doing only pattern matching, while stream processing can also do aggregations.  I was never sure why this distinction is important, furthermore, as I have written before CEP is used by different people to mean different things, thus I would say that "event processing" is the name of functionality that does all.  In any event, the talk about parallel incremental computation was interesting.
• The second paper was presented by Alex Artikis from NCSR and event recognition (pattern matching) based on event calculus.   Formal approaches can be useful for that domain, and the event calculus is one of the first attempts to do it.  I have some terminology dispute with Alex, who equates the notion of fluent from event calculus to composite event.  I think it is actually refers to a state that can be initiated and terminated by events, and may serve as context or state. 
• The third paper was presented by Michael Olson,  Mani Chandy's PhD student in CalTech.  He presented the latest on their going on project on geo-spatial events (relate to seismic events)
• The fourth paper was present by Boris Koldehpfe from University of Stuttgart, who also active in the community for several years.  Boris talked about range queries in distributed event processing systems, where range queries relate to spatial operators on events.   It seems that the spatial dimension which was also discussed in the previous talk is gaining more traction. 

 I'll write about the sessions of today at a later point.

DEBS 2012 - keynote by Sethu raman

This morning started by some statistics about paper acceptance by Patrick Eugster, the PC chair. It seems that the acceptance rate in the scientific track was lower than previous years.  

The first keynote was given by Sethu Raman, who was until recently the "distinguished engineer" in Microsoft responsible for StreamInsights.    Sethu gave a relatively short talk (with a lot of questions and discussions) entitled "event processing solutions in the enterprise - a waypoint or launchpad". The illustration above is taken from his opening slides, he had such nice illustrations and citations all over.

You can download Sethu's presentation from the DEBS 2012 webpage (the organizers is doing a good job by copying the presentations from the presentation computer and posting it on the website quickly). 

The main message of Sethu is not surprising (as seen from the information management universe):

•  Event processing has a big potential to play in multiple roles -- real-time analytics,  ETL.  distributed query framework 
• It also plays in many different industries and is not a niche play
• There is a major issue is of conception.  There is aggressive messaging of the "analytics" community which drowns the event processing message in some cases,  The analytics community claim that they can solve all problems of the universe by database centered way, and that event processing is required only for small niche of very low latency requirements that cannot be satisfied by DBMS technology.
• The reality is that the value proposition of event processing is not restricted to the low latency, it is actually reducing the high cost of developing applications - however this message is not heard in a clear enough voice, relative to the other aggressive voices.
• The term "complex event processing" is a confusing term,

I have discussed it with Sethu in the break,  I actually agree to all the assertions.  The over-hype of analytics, and the messaging that comes sometimes from the analytics people (which is not really backed up by technical facts) makes the impression that analytics subsume the functionality that is proposed by event processing.   I am not sure how ling the over-hype of analytics will continue (I think this is quite high in the hype-cycle and will get to the disillusionment phase, since it is indeed over-hyped),  but the messaging of the event processing community about its ROI as complementary and sometimes contrasting to analytics should be cleared heard.   How? -- this is part of the discussion about the EPTS online magazine.

More - later 

DEBS 2012 tutorial on event processing design patterns

In the morning I had to attend the tutorial that I have co-presented,  but in the afternoon I was free to chose among three tutorials.  Out of sheer laziness I stayed in the same room and listened to the tutorial given on behalf of the EPTS reference architecture workgroup.   The gang of four that presented this tutorial were:

• Paul Vincent (who gave his performance wearing a papillon  tie - I hope that somebody took a picture of him, I have put a picture of some papillon tie as a place holder)

      Paul was the main presenter, and also presented the TIBCO interpretation and the Oracle one (covering  

      for Alex Alves who couldn't arrive).

• Adrian Paschke - who presented the Prova interpretation (Prova is an open source that he co-developed)
• Catherine Moxey - who presented the IBM Websphere Operational Decision Management (WODM) intepretation 
• Martin Hirzel - who represented IBM Infosphere Streams

The tutorial was uploaded to the DEBS 2012 site along with the other tutorials  

I need to review this tutorial carefully in order to comments,  but several comments off-hand;

First - I think that the team has done a very thorough work of trying to classify functions, map them to various implementations, they also attempted to map them to the EPIA book that Peter Niblett and myself have composed.   

I need to think more carefully about some terminology issues --- first, I am not sure that what they presented are really design patterns as defined in software engineering by the original gang of four.  I think that what was done is more classification of functions not really design practices.  Their classification is to four major classes: event reaction, complex event detection, event analysis and event preparation. 

The complex event detection contain three functions:  consolidation, composition and aggregation.  I am not really sure that the definitions are intuitive, but I am taking as an action item to look at the definition and right a detailed review of them.   There are some other stuff that worth reading and reviewing, including validating the relationships that they made with the EPIA patterns (again in EPIA these are functions and not patterns).

The mapping to the different languages is also very interesting.  Work on real best practice of patterns and anti-patterns will be a good next step.

More -later