Container fleet management with event processing

An interesting application of event processing technology is reported by Jack Vaughan on container fleet management done in  Orient Overseas Container Lines (OOCL), Hong-Kong based company.

It tracks various events such as monitoring the load time to avoid penalties.   You can read more in the article itself.  

On event oriented thinking

The title of this Blog is "event processing thinking", and much of the posts are my own thoughts about event processing, but today I would like to write about another topic: event oriented thinking.  

An interesting observation from recent discussions is that  in daily life we can very easily think in an event-driven way,  when the phone is ringing we are either answering it or ignoring it, when we hear about traffic jam - we try to find another way, and so on...      However - when it comes to thinking about computerized systems,  many people are programmed to think in a request-driven way, which means:  a person sends a request and the computer responses.   Thus people are trying to use what they know even if they have scenarios that are event-driven.  A very typical line of thinking is:  event is a data, we should insert it in a database, and then ask query that reflects the situation that we wish to detect.   The question is -- when are we going to ask this query.  We can ask it on any update -- but if the event is just one component in a pattern we wish to detect (example: we wish to detect when a customer sent three requests within a single day -  assuming that most customers don't send more than one request, then we can have 99.99% of the queries redundant),  we can ask the query periodically - but then again, we can both ask redundant queries, and moreover, not able to react on time, since the granularity of our timing is wrong.   There is also a notion of continuous query -- but this is not really a query, since it is not a result of a specific request.  

The nature of event-driven scenarios are: we don't know when they are going to happen, we even don't know whether they are going to happen,  but when they happen we want to do something - sometimes very fast (e.g. earthquake detection).   Furthermore, the situation we would like to detect can be realized in a pattern of many events,  and each individual event may trigger reaction, just increment some state, or even be ignored or filtered out.

The fact that people are trying to model and implement event-driven scenarios using the traditional request response is a thinking mismatch, and creates added complexity.  I'll follow up with an example in one of the next posts.  More - later.

On fifty years of databases

This drawing is taken from the ACM SIGMOD Blog post by Thomas Haigh entitled 
"Fifty years of databases".   It tells the story of  IDS (Integrated Data Store) as the first database developed 50 years ago in GE.  IDS was designed by Charlie Bachman, who received Turing award for his pioneering work on databases, and as Haigh remarks  - Bachman was the first Turing award winner who did not have PhD, and actually spent his life in industry and not in academia.  It is worth remarking that the two persons who followed Bachman by receiving Turing awards in the database area, Tedd Codd  and Jim Gray were also from industry, in fact, the academic database community did not have until today any Turing award winner.  Bachman's Turing award talk "programmer as navigator" was very insightful.  Bachman compared himself to Copernicus who said that the earth is revolving around the sun and not vice-versa, and said that the computing world will revolve around data - where programming will be side effects of operations associated with data.   We are not quite there, but for me it was an inspiring goal.
IDS is far from the current databases we have today, but it laid the principles, and started from pure engineering position, theory came later.     I am trying to compare the database area development to those of event processing, and think that in many respects we are still were databases have been 40 years ago,  so the challenge is to advance it further...  more on that -later.

On decision latency

James Taylor wrote in his Blog an article on decision latency, the illustration above is taken from his article. 

The latency for getting a decision consists of three time intervals:

The Capture latency is the time taken from the event occurrence in the universe until it is detected (captured) by the processing system

The Analysis latency is the time taken from the time that the event is detected by the system, until the time it is  processed and creates the derived event/data that triggers an action, and the Decision latency has two parts - which I think should be distinct,  the time to make the decision how to react, and the time it takes to react.

These four type of latency are actually orthogonal.  For the Capture latency what is required is an infrastructure of instrumentation, sensors and communication; for the Analysis latency -- the event processing system latency, and possibly additional analysis required by querying historical databases,  for the decision latency --  the decision tools (either a decision management system or optimization system), and for the action latency -- the speed in which it can be applied. 

Why the decision latency is important? -- here we are getting back to the real-time issue.  In some cases the time is critical for decisions,  an example that we saw a few weeks ago is in the Israeli anti-rockets systems

known as "Iron Dome" (Kipat Barzel).  The system detects that a rocket was launched, and then it needs to analyze its course and determine whether it is going to hit a populated area, in order to decide whether it is cost-effective to send a missile to hit the rocket (this decision is needed since a missile is very expensive), and if yes,  determine when and where the missile will intercept the rocket and launch the missile at the right time to the right direction.  Latency in all the four phases is critical., and the success of this system saved many lives.    

Decision latency is not only related to defense systems,  it exists in different other areas - healthcare is certainly one of them, but also business.  The time scale does not have to be seconds,   decision latency can span minutes or hours,  

We can look at the value of decision as function of its latency,  and can see that there are soft real-time cases which the value of decision goes gradually to zero, Firm real-time where it goes to zero at a certain deadline.

Hard essential in which it goes to a certain penalty (such as missing SLA), and hard critical where it is going to a minus infinity (a disaster case), the Iron Dome case is of that type.  

Of course, there are no miracles, and the cost of a system grows when bounded latency need to be guaranteed (for example cost of extremely robust communication).  For each type of event-based system there is a need to calculate the trade-off between the value of latency and the cost. 

Launching the online magazine: Real-time Business Insights: Event Processing in Practice

There are two events today -- the 15th birthday of my daughter Daphna (in the picture taken in our recent family vacation in Malta)

Daphna's birthday will help me remember also the other event happening today -  launch of the online magazine.  I have written about the idea of the online magazine before,  a few months ago - took some time to accomplish - assemble and editorial board, solicit material from authors and find a publisher who believes in the idea to take care of the logistics.   

After some deliberations - the name chosen is shown on the Logo below.  The idea is to start with several pilots issues, the first of them is being launched today.  The magazine consists of several sections:

Business Strategies presents the customer's perspective on how other enterprises gained value from processing streaming data and events in a variety of use patterns.

Tools and Tactics presents the practitioner's perspective with best practices and lessons learned from those who develop event-based solutions.

Frontline presents the community perspective with provocative and thought provoking opinions.

Industry Insights provides in-depth education about the use of real-time business insights in a particular industry.

On the Horizon provides the researcher's perspective on what's coming next in

applied research projects and new technologies.

 EP News keeps readers abreast of the latest industry developments

Each of these section has a section editor, I volunteered to serve as editor-in-chief and section editor of the blogging section ("On the Horizon") - to set a personal example, I have also written the first one, expressing an opinion which will come at no surprise to this Blog's readers.  

The online magazine can be downloaded from here:   (registration is required by the publisher), 

You are encouraged to download, react (the magazine's website links to the EPTS LinkedIn group), distribute to anybody who might be interested, and contribute to any of the magazine's section.

Enjoy and react!

On behavioral programming

Yesterday I took the train to Rehovot and visited the Weizmann Institute.  I gave a talk in a local seminar and  was hosted by David Harel's research group.  An interesting work they are doing is on behavioral programming, which is a formal language independent model to specify reactive applications.  They have been experimented with games, robots and toy helicopter.   They are actually specifying event processing systems;  I see a possible benefit in their approach that their formal model is a basis for validation which in event based systems is tricky,  due to the temporal nature.   I'll  work with them on some use cases  from our universe to learn more about their model.    Can be interesting.  

On combining event processing with neural networks

This picture is taken from a recent article by Kamalkumar Mistry which explains the benefits of possible combining event processing with neural networks, starting by explaining each of these technologies separately and then providing a patient monitoring example. In this example the combination of the two technologies is that a patient is monitored in real-time, the signals from the monitors are getting to an event processing system as an input, and after processing is done the output is fed into a neural networks, which recommends action based on an  individual model of each individual patient.

This is a valid use case,  I actually thought about additional use of neural nets and this is to tune up the monitoring, in event processing terminology - set up the patterns to be monitored.   Taking back the monitoring patients example,  which is actually one of the use cases we analyzed in the past,  in fact it is example no.1 in the examples we put in the introductory chapter of the EPIA book, (chapter 1, page 7).  In our example, the physician can tune up the system to have individual monitoring patterns for each individual patient, since in different patients, different combinations of signals over time may mean different things.   The event processing pattern can be recommended by the neural network system based on the same patient model mentioned by Mistry, and in this case the relationship between the neural net and the event processing system are reversed.

Anyway - it is interesting to investigate this combination further.

A promo: stay tuned for the inauguration issue of the online magazine soon...