Reflection on blogging in 2012

The year 2012 is going to expire today,  this has been the first year since 1985 that I have not visited the USA (I have been several times in Europe, though), somehow I don't think this will be true for 2013.

Looking at this Blog,  I had less posts this year (this is post 92nd for the year, the record year was 2009 with  162), but the flow of readers was bigger this year than the previous years,  I recently  came across an article in HBR Blog entitled "If you're serious about ideas, get serious about blogging".   

Looking at the popularity test,  the most read post this year was entitled the pilot decision making process,  which shows that the mental thinking of a pilot is situation driven.  One of the main areas that I have investigated this year is how to make people think in a situation driven way when coming to IT systems, which are dominated by the request-response thinking.   

Another popular post was   not about event processing but the one dealt with the question: Is computer science a science or engineering?   This question was triggered by the fact that my daughter Daphna participated in a science day in the high school she was going to attend (and is attending now) and it seems that while this school has computer science major, it does not regard it as a science.    My opinion is that computer science is neither science or engineering but a thing of its own.

Additional ones are again a more generic post on presentation skills.  This is a  soft skill that I think is extremely important in today's world.  I am putting emphasis in all the courses and seminars I am teaching, my source of inspiration, as I have written is Steve Jobs style of presentation.

Several popular professional posts: 

  On temporal extensions to  SQL 2011.      I am following temporal databases for many years, and the eventual extension to SQL is long overdue.   

On event server as the 21st century application server - following Paul Vincent,  I think we are seeing this shift happens.

Last but not least of the popular post was my review of Dave Maier's keynote in DEBS 2012, where I observed that the fragmentation in research make even distinguished researchers to reinvent wheels.

Let's see what blogging topic will be interesting in 2013 -- happy new year.

On{X} Recipes - cool event based Android application

Thanks to Guy Sharon I've learned about cool Andorid application developed by Microsoft Israel called On{X}.  It enables to specify and use an event- condition-action recipes (the name sounds tastier than rules).

Examples of recipes are: 

Ring on the third call from the same person when my phone is silent 

Set mode to silent between 11:00PM and 7:00am after the phone has not been unlocked for 2 hours

Text my wife "my phone is dying" when the battery goes below 15%

All of these look to me as event patterns.  Looking at the On{X} Blog I learned that new features include - dynamic regions, noise detection and more.  There is a collection of recipes and one can program more using JavaScript 

A very interesting event processing application  -- I may buy smartphone one of these days :-) 

More on request driven vs. event driven

In the table above (taken from a recent presentation I am working on) I have summarized some of the main differences between request driven thinking and event driven thinking.   It is interesting to note that many of the activities we are doing in life are event driven,  however, we are programmed to think that computer should be approached in request driven way, and I have noticed that even if the application itself is event driven by nature, people will tend to convert it to request driven.  Event driven action is being activated not due to explicit request but since an event has occurred, or a derived event was concluded.  This may happen in unknown time and unknown frequency.  Furthermore, a request getting into a system should always entail response (which can be error message),  an event getting into the system may be ignored, since it is out of context, just increment internal state, or close the circle of detecting derived event which can be either internal to the system, or trigger external action or notification,  Note that only the last case has visible response to the outside.    One of the challenges is to educate people to think in event driven way rather than request driven.  I'll write more on event driven thinking in the sequel. 

Container fleet management with event processing

An interesting application of event processing technology is reported by Jack Vaughan on container fleet management done in  Orient Overseas Container Lines (OOCL), Hong-Kong based company.

It tracks various events such as monitoring the load time to avoid penalties.   You can read more in the article itself.  

On event oriented thinking

The title of this Blog is "event processing thinking", and much of the posts are my own thoughts about event processing, but today I would like to write about another topic: event oriented thinking.  

An interesting observation from recent discussions is that  in daily life we can very easily think in an event-driven way,  when the phone is ringing we are either answering it or ignoring it, when we hear about traffic jam - we try to find another way, and so on...      However - when it comes to thinking about computerized systems,  many people are programmed to think in a request-driven way, which means:  a person sends a request and the computer responses.   Thus people are trying to use what they know even if they have scenarios that are event-driven.  A very typical line of thinking is:  event is a data, we should insert it in a database, and then ask query that reflects the situation that we wish to detect.   The question is -- when are we going to ask this query.  We can ask it on any update -- but if the event is just one component in a pattern we wish to detect (example: we wish to detect when a customer sent three requests within a single day -  assuming that most customers don't send more than one request, then we can have 99.99% of the queries redundant),  we can ask the query periodically - but then again, we can both ask redundant queries, and moreover, not able to react on time, since the granularity of our timing is wrong.   There is also a notion of continuous query -- but this is not really a query, since it is not a result of a specific request.  

The nature of event-driven scenarios are: we don't know when they are going to happen, we even don't know whether they are going to happen,  but when they happen we want to do something - sometimes very fast (e.g. earthquake detection).   Furthermore, the situation we would like to detect can be realized in a pattern of many events,  and each individual event may trigger reaction, just increment some state, or even be ignored or filtered out.

The fact that people are trying to model and implement event-driven scenarios using the traditional request response is a thinking mismatch, and creates added complexity.  I'll follow up with an example in one of the next posts.  More - later.

On fifty years of databases

This drawing is taken from the ACM SIGMOD Blog post by Thomas Haigh entitled 
"Fifty years of databases".   It tells the story of  IDS (Integrated Data Store) as the first database developed 50 years ago in GE.  IDS was designed by Charlie Bachman, who received Turing award for his pioneering work on databases, and as Haigh remarks  - Bachman was the first Turing award winner who did not have PhD, and actually spent his life in industry and not in academia.  It is worth remarking that the two persons who followed Bachman by receiving Turing awards in the database area, Tedd Codd  and Jim Gray were also from industry, in fact, the academic database community did not have until today any Turing award winner.  Bachman's Turing award talk "programmer as navigator" was very insightful.  Bachman compared himself to Copernicus who said that the earth is revolving around the sun and not vice-versa, and said that the computing world will revolve around data - where programming will be side effects of operations associated with data.   We are not quite there, but for me it was an inspiring goal.
IDS is far from the current databases we have today, but it laid the principles, and started from pure engineering position, theory came later.     I am trying to compare the database area development to those of event processing, and think that in many respects we are still were databases have been 40 years ago,  so the challenge is to advance it further...  more on that -later.

On decision latency

James Taylor wrote in his Blog an article on decision latency, the illustration above is taken from his article. 

The latency for getting a decision consists of three time intervals:

The Capture latency is the time taken from the event occurrence in the universe until it is detected (captured) by the processing system

The Analysis latency is the time taken from the time that the event is detected by the system, until the time it is  processed and creates the derived event/data that triggers an action, and the Decision latency has two parts - which I think should be distinct,  the time to make the decision how to react, and the time it takes to react.

These four type of latency are actually orthogonal.  For the Capture latency what is required is an infrastructure of instrumentation, sensors and communication; for the Analysis latency -- the event processing system latency, and possibly additional analysis required by querying historical databases,  for the decision latency --  the decision tools (either a decision management system or optimization system), and for the action latency -- the speed in which it can be applied. 

Why the decision latency is important? -- here we are getting back to the real-time issue.  In some cases the time is critical for decisions,  an example that we saw a few weeks ago is in the Israeli anti-rockets systems

known as "Iron Dome" (Kipat Barzel).  The system detects that a rocket was launched, and then it needs to analyze its course and determine whether it is going to hit a populated area, in order to decide whether it is cost-effective to send a missile to hit the rocket (this decision is needed since a missile is very expensive), and if yes,  determine when and where the missile will intercept the rocket and launch the missile at the right time to the right direction.  Latency in all the four phases is critical., and the success of this system saved many lives.    

Decision latency is not only related to defense systems,  it exists in different other areas - healthcare is certainly one of them, but also business.  The time scale does not have to be seconds,   decision latency can span minutes or hours,  

We can look at the value of decision as function of its latency,  and can see that there are soft real-time cases which the value of decision goes gradually to zero, Firm real-time where it goes to zero at a certain deadline.

Hard essential in which it goes to a certain penalty (such as missing SLA), and hard critical where it is going to a minus infinity (a disaster case), the Iron Dome case is of that type.  

Of course, there are no miracles, and the cost of a system grows when bounded latency need to be guaranteed (for example cost of extremely robust communication).  For each type of event-based system there is a need to calculate the trade-off between the value of latency and the cost.