On the OODA loop and the 4D

Richard Veryard asked in a comment to my post on  the recent tutorial given by Jeff Adkins and myself if our 4D scheme (which was introduced by Jeff)  is related to the famous OODA loop.  I have mentioned the OODA loop in the past in connection with the need to act faster than the speed of thinking. the strategy of air combats, later he also made claims about the generality of this method.    The 4D is certainly from the same family, and the four stages are indeed similar.   Interestingly Boyd's had event-driven thinking.    

The OODA loop was aimed to describe event-driven decision by a human - the human has to observe that an event happened, perform mental self orientation to analyze the meaning and implication of the event,  decide what to do, and act accordingly, have feedback loop to see whether the observation has changed.

The 4D describes a computational process, where the event is detected (not necessarily directly observed),  situation is derived (by computational means and not by mental process), and then a decision is taken (autonomic or manual) and an action is performed.   

The mapping is not 1-1:    

detect is always mapped to observe;

derive can be mapped to observe - as the detected situation is a derived event, and sometimes to orient - as it may derive a conclusion.

decide can be mapped to the combination of orient and decide in the OODA loop

do seems to be always mapped to act. 

More thoughts about the 4D and related stuff - later.  

The 4D - where is the starting point?

In reaction to my recent post, Gagan Saxena responded with "decision first", stating that the starting point is decision. Goals and decisions need to be managed, and the event processing is a means to achieve these decisions.

Taking the 4D (detect-derive-decided-do) approach, there might be multiple starting points:

The detect starting point:  Start with instrumentation and sensing. Create event, and publish them.  The event utilization and processing comes second, based on what is available

The derive starting point: Start with the situations that need to be identified,  then determine what need to be detected in order to derive this situation, and the decision is on reaction to the situation.

The decide starting point: As proposed by Gagan, start with the goals and decisions needed to achieve these goals, the determine what is need to be derived and detected to achieve this goal.

The do starting point: Start with the action, which can be a workflow,  and treat the rest as trigger to the action.

Each of these approaches may be valid in certain scenarios and circumstances. 

I argue that in event-driven systems goals are more linked to situations than to decisions.    In cases where the decisions are request-driven, a decision is requested due to getting to a certain point in the workflow that requires a decision to achieve some goal.  In event-driven systems the decision is required to react to a situation, thus the first goal is associated with the situation,  then there is another goal of how to react to the situation.  The decision might be simple (notify somebody), or complex (optimize some objective function based on predicted events).     

Each of the 4Ds have to be managed as an entity.  The detect part need to be managed in order to determine  sensors, frequencies, fault tolerance etc..,  the derive part need to be managed, and this is the glue to the rest of the Ds,  decisions need to be managed, and actions need to be managed.  It is a benefit if all of them can be managed in a coherent system and not in four different systems.  More on this - later.

Event processing at the core of automated commerce systems

This illustration is taken from Chris Taylor's Blog, where he posted a piece entitled:  "There's a technology tsunami hitting the world of commerce".  Chris makes distinction between the hype technologies:  cloud, analytics, big data, and mobile, and claims that all of them have role in the story of commerce, however the core technologies doing the heavy lifting are: integration, managed file transfer, in-memory data, log file management, event processing and business process.

As for event processing, Chris believes it is the core, and issues the nervous system of the commerce which enables to automate it, as commerce is event driven by nature - buy and sell are events, and so are payments, fairness rules, credit decisions, fraud detection, and management of the commerce system itself. 

Note that illustration follows the 4D: detect (channel), derive (pattern detection), decide (decision), do (actions).  The "decide" part is not necessarily "IF---THEN" rules, but can also be optimization or another event driven pattern. 

On event model and decision model

In the article about "event model" by Luckham and Schulte, on which I have reported last week,   the term "decision model" was referenced as one of the complementary models to an event model.  This has created some discussion within the DMN LinkedIn group, the discussion was created by Paul Vincent who is one of the persons behind the DMN standard work.   I am following the DMN work, though not in an active way, and participated in the kick-off meeting in March 2011,  and also written on "decision models and event processing"  following the talk on "The Decision Model" by KPI at the same event. 

In order to view the relationship between "event model" and "decision model", we need first to understand what is a decision that a decision model is modeling 

 The DMN presentation defines decision as "A Decision is the act of applying decision logic to one or more inputs and produce one output".    This can relate to modeling a collection of business rules, which gets a request with an associated input and returns a result  ("concluded fact") or to a result of some quantitative model (such a scoring model or optimization model). 

In comparison event model is modeling event-driven logic, which is based on deriving situations or derived events based on filtering, transformation, aggregations, and pattern detection over a collection of input events.   

The question is what is the relationship between these two terms?  

we can start by looking at the relationship between business rules and event processing,  there have been long discussions about it in the past, the position I expressed was that related to a discussion with James Taylor in 2008, stated that event processing and decision management are different issues.  A decision may not be related to event, and an event processing may not trigger any decision.  However, there are also relations between them.   

In my recent presentations about event-driven applications I am using the "4 D" that coined by Jeff Adkins: 

 I found out that this is an easy way to explain both IT and business persons who are not familiar with event-based systems what it is in a nutshell.      Referring to this picture, an event model deals with the "detect" and "derive" phases.    The derived event may also by itself denote a decision (since it is obvious what to do if this situation is detected and does not require further decision),  or that the detection of this situation triggers a decision that can be modeled by a decision model (note that the situation may trigger a task, and then the decision may be embedded inside a business process model).  In the example referred to in the slide above,  the  "derive" phase detects that a traffic jam is coming, and the "decide" phased determines how to reset the traffic light policies in order to reduce the traffic jam -- in this case they are complimentary.

Another interesting point is that "decision" in the common interpretation is "request driven",  I have recently written about the distinction between "event driven" and "request driven".   One may claim that a request is an event (which is sometimes semantically doubtful), or that there is always some event that causes a person or a system to make a request (may be metaphysically true, but the event is not explicitly exposed).  However, request-driven is taking an action by request (a passive approach), while event-driven is taking an action without a specific request, due to either occurrence of event or detection of situation (an active approach).    An event can take 

Bottom line:  decision models and event model are complementary models.   I'll discuss similarities and differences in the content of what is modeled in a later phase. 

Acting faster than the speed of thinking

Chris Taylor from TIBCO has written in a Blog with the nice title: "a place for good ideas in a fast changing world" entitled "getting there faster than your opponent".  In the same spirit of TIBCO's two second advantage, it makes the point that event processing is vital for getting things faster than others.  Chris enlists the famous OODA loop originated in the US Air Force, for mentioning that faster decision can impact the combat field.   Note that OODA is one of the variations of  control loops, other variation I have written about is the 4D variation.    

It is interesting to note that speed of reaction has been one of initial reasons for using event processing technology in applications like high frequency trading, where trading programs compete on speed.  I guess that military applications also gain from competition in fast reaction, as well as cyberspace wars.   

While these are notable applications, a common misconception is that event processing is restricted to these type of applications, however, there are many other applications in which competing on speed is not an issue that can benefit from the use of event processing due to the benefits in reducing cost of development and maintenance due to higher level abstractions.  In fact one of the first posts in this Blog, almost five years ago tried to answer the question whether the only motivation to use event processing is high performance?  in this post I have discussed the Total Cost of Ownership as a function,  I guess it is also applicable today.

The two  main observations are:   acting faster than the speed of human thinking issue an important type of event processing applications,  but not the only reason, also in many cases event processing is not enough, and some real-time decision mechanism (reactive or proactive) need to be applied to achieve autonomic action, since the required speed requires the elimination of human from the loop.

On the 4Ds -- past version and the proactive version

The climate in Israel this year is quite strange, it is December, and today I still saw people going in the street with short dress, the summer just did not go away.  However, the forecast for the next three days, starting tonight is of a major winter storm (which here means a lot of rain, not snow) and much colder weather, so getting the winter clothes ready.   


Today I've read a blog posting by Jeff Adkins,  one of the people with most practical knowledge about event processing, who relatively recently joined IBM  GBS (Global Business Services).  Jeff blogged about the 4Ds-- detect, derive, decide, do.    These four are part of smart systems that sense and respond, what is known as reactive system.  I knew that this looked familiar, so went back to my archive and found that seven years ago we  were engaged with a project called "active integration"  (the actual application was in the insurance area, but we have generalized the concept), roughly what was known by Gartner as "real-time enterprise".    Here is the original flow from that project:

  
I don't think it has been original invention, it was a variation of concepts from control theory, but it looks very similar to the 4Ds, with a more detailed granularity.


The first D: Detect spread into two phases on our model: sense and detect, where the sense dealt with instrumentation and sensing of raw events, and detect with a detection of the meaningful situations by pattern detection.


The second D: Derive is the same:  created a derived event (and sometimes also derived data) as a result of this detection


The third D:  Decide is partitioned to three phases:  Analyze - determine the possible alternatives and recommend,  Collaborate - in case of "human in the loop" within the decision, and Decide -- apply some decision procedure to select among the alternatives (simulation, analytic methods, predetermined rules).


The fourth D: Do we called "effect", since it had to effect a running system.  


The loop indicates that the after effecting the system, it feedbacks through its instrumentation mechanism and the sense phase is looking again for things that needs reaction.


I agree that the 4D is much more catchy then our more detailed drawing.


And one comment:  when we did this work, we worked on REACTIVE system - something has occurred, we detect it, and then do something to repair.   This drawing is also a good description of our current project that deal with PROACTIVE systems, but the semantics is somewhat different:  the detection is of predicted undesired states instead of situations that require reaction since something already happened.