On transdisiciplinary education

One of my IBM colleagues, Jim Spohrer, the IBM University Relations Director,  has written about the need of students to receive transdisciplinary education.   Indeed we find more and more that doing things require not just multidisciplinary skills, of people from several disciplines working together, but also transdisciplinary skills, where people have good skills in several disciplines and a single person knows how to integrate these disciplines.  This is, of course, contrary to the academic tradition, where promotion processes for faculty members advocate the association with a single well-defined discipline.    We can see much more  transdisciplinary programs today then 20 years ago.  I believe that a program in which a student learns combination of computer science with another discipline (biology, business, mechanical engineering, cognitive psychology and more...),  not only at the intersection between the disciplines, but rather at the union, of having deep understanding in both disciplines as the type of skills required in the marketplace.  
  

The pilot decision making process

While surfing the web to find something,  through the wonders of Google search, I reached the site of the Langley flying school and came across the illustration above under the title "the pilot decision making process". 
The cycle starts with a situation, which is defined as a critical situation which requires action.   The detection and identification of such a situation is what starts the process, thus the decision process is inherently event driven, then the pilot needs to assess the options,  sometimes the options are very obvious, and sometimes there is a creativity in that process,  then there is a need to chose among the options, act according to the selected action and assess whether the situation has been resolved.   Note that this may be either reactive (situation already happened) or proactive (situation is expected to happen).  Of course, when possible it is better to be proactive, it is much better to eliminate crash of the aircraft then try to rescue the crew and passengers when the crash already happened, both from economic point of view and from safety to human life point of view.     Note that the pilot needs to react to critical situations, for the process-driven business as usual case, an autonomic pilot is sufficient.    Can we extend the autonomic pilot also to deal with critical situations?  

On enrichment - and the difference between BRMS and EP

A recent article in the IBM developerWorks discusses two ways to enrich data used for rules from external databases,  one of them is doing the enrichment in the request level, before calling the "decision server" (which is the current name for using BRMS system using the request-response protocol), the other one is doing enrichment during the rule processing itself. The article describes how each of these options is done and also discusses pros and cons,  the benefits of enrichment by the request level are - less complexity, and better performance of the rule component; the benefits off enrichment at the rule level are - handling dynamic data and more specialization for the exact data that is being used by the rule.


Thinking about event processing -- there is similarity to the BRMS case,  event can be enriched both by the event producer and as part of the event processing itself, the arguments are not far from those in the BRMS case,  there is one fundamental difference in event processing --  the work is not done using the "request-response" protocol,  moreover, the different part of the system are decoupled, thus the event producer does not necessarily know what purposes the event is going to be used, thus there may be different types of enrichment needed for different uses.   The dynamic aspect is applicable here, and there may be some race conditions in highly dynamic systems between updates in the database that was enriched and its use in enrichment, unless the event processing enrichment system locks the data in the database until it is being used in the event processing system, which requires the event processing system to exhibit a transactional behavior for part of it, but I'll not get now into this issue,


Bottom line:   The considerations in event driven architecture are somewhat different than the request-response systems that are the most common one in computing.   

Timo Elliot's presentation on "Business in the moment - from reactive to proactive"

Timo Elliot from SAP gave a recent talk in the Gartner BI meeting in London entitled "Business in the moment -from reactive to proactive". You can download the presentation from a link in Timo's Blog posting.   In a following post on his Blog,  Timo refers to an FreshDirect explaining the proactive behavior:

“FreshDirect has an operations center that manages its fleet of delivery trucks. In a large metropolitan area like New York, traffic doesn’t always flow predictably. A traditional approach to BI would be to print a report showing the level of on-time deliveries (OTDs) the day before and then ask the transportation department what went wrong for the orders that were delivered late. FreshDirect uses analytics in a more impactful way.”

“The company monitors the delivery rate of every truck and enters that data into the BI system on an ongoing basis. Every hour, it uses the previous hour’s data to predict how many deliveries will be on-time in the next hour. If the predicted OTD rate is below FreshDirect’s target, the company sends out an auxiliary truck or trucks to help make deliveries. The company holds 10 trucks in reserve for just this purpose.”

I'll bring more proactive stories when I'll find out about them...

Killing elephants - is MapReduce dying?

My English teacher in the last grade of high school had an interesting taste in literature, and taught us the story on  "Shooting and Elephant" by Orwell.   I was not a very good student in English and forgot about it until reading Colin Clark's Blog posting entitled : "It's time to kill the elephant".   From time to time there are various people claiming that various things are dead or dying.   Some of the readers may still remember the discussion about whether SOA is dead.   Recently the Forbes Blog has announced the death of ERP.  Colin's contribution to the hunt is the observation that MapReduce is dying (or should be dying) and the batch processing should be replace by more real-time processing.  His evidence is that Google is dumping MapReduce and using Colossus for its search technology.   While this fact is certainly true, I think that there are still many types of analytic procedures that are done off-line using batch processes, so while the use of real-time analytics will substantially increase given supporting infrastructure, I am not sure that batch will die soon (the same goes for SOA and ERP)...   Old soldiers never die - they just fade away (s-l-o-w-l-y).

Crash course to build simple EP application using Esper

A crash course claimed to take less than an hour entitled "A simple introduction to complex event processing" has been posted.  This is done by example, which seems to be indeed very simple, finding "decreasing" or "increasing" pattern over two consecutive events and setting the color as green or red.   The main emphasis is on the setting - how to obtain, define and use events, and configure the engine - threadpools, listeners etc...  However not much about what event processing actually can do -- this is probably the next lesson.


  Esper is contrasted with commercial products since its open source model allows developers to play with it, use it for toy examples, and for daily usage that is not necessarily a commercial application of big enterprise, in our days of enterprise computing this approach has certainly a role to play, it should be noted that Esper is not the only open source in this area, and that some of the commercial products allow free development version (not access to the source code, but enabling developers to use the product for these purposes for free).


Anyway -- if you wish to learn Esper, it is a good start. 

On revision and compensation in event processing

The event processing course that I taught in the Technion has ended (except for the make-up exam in March). This year the students' project has been different from the last couple of years (well, I am bored having the same type of project all over again).  The students were asked to select a research project.  My Teaching Assistant was very skeptic about the students' abilities to do such projects, but after the presentations they did in the last class summarizing what they did so far she changed her mind.    I'll write about these projects after they'll be submitted (due in early March).   One project is investigating the issue of compensation and revision in event processing,  I have written about revisions before,  both in general,  and specifically related to event processing.  
There are couple of motivations of why I have returned to be interested in it now. 


One of them is the investigation of uncertain events,  since more information may be acquired with time about events that are uncertain,  a revision might be needed.
The second is the work on future events,  since future events are obtained using a forecasting process, this forecasting process is not a one-time process, but can be sensitive to additional events that happen between the forecast and the occurrence of the future event, thus the forecast itself may be revised.  
The issue of revision may entail the need for compensation for decisions and actions already taken.

• In some cases it is easy, example when no action was taken and it is still possible to take action
• In some cases it is impossible, if an action has been taken, and this action cannot be retracted, 
• In the remaining of the cases, it might be possible, however not always cost-effective, since it might have cascading effect of compensating for large amount of actions. 

After getting the students' work, I'll write again about this issue.