Acting faster than the speed of thinking

Chris Taylor from TIBCO has written in a Blog with the nice title: "a place for good ideas in a fast changing world" entitled "getting there faster than your opponent".  In the same spirit of TIBCO's two second advantage, it makes the point that event processing is vital for getting things faster than others.  Chris enlists the famous OODA loop originated in the US Air Force, for mentioning that faster decision can impact the combat field.   Note that OODA is one of the variations of  control loops, other variation I have written about is the 4D variation.    

It is interesting to note that speed of reaction has been one of initial reasons for using event processing technology in applications like high frequency trading, where trading programs compete on speed.  I guess that military applications also gain from competition in fast reaction, as well as cyberspace wars.   

While these are notable applications, a common misconception is that event processing is restricted to these type of applications, however, there are many other applications in which competing on speed is not an issue that can benefit from the use of event processing due to the benefits in reducing cost of development and maintenance due to higher level abstractions.  In fact one of the first posts in this Blog, almost five years ago tried to answer the question whether the only motivation to use event processing is high performance?  in this post I have discussed the Total Cost of Ownership as a function,  I guess it is also applicable today.

The two  main observations are:   acting faster than the speed of human thinking issue an important type of event processing applications,  but not the only reason, also in many cases event processing is not enough, and some real-time decision mechanism (reactive or proactive) need to be applied to achieve autonomic action, since the required speed requires the elimination of human from the loop.

On open access in scientific publications

A recent article in "The scientist"  issued a blunt attack on open access in scientific publications, call it: predatory publishing.  This article claims that open access publications will not allow people to distinguish between "science" and "non-science", since it will not guarantee the quality of publications through the rigorous process of peer reviews, and then gives the main reason for objection --- "implications for tenure and promotion will be significant".    Some of the comments compared the arguments with those made against open source software.   

I think that there are some orthogonal issues here:  first -- I am for open access, actually every person today can put any paper on a website, and now all we need is to index it so it will be accessible to all.   Scientific publication is not different in principle from newspaper article, blog, TV show, book - in all of them we can find low quality and high quality examples.  

The review process is highly subjective and noisy,  and the amount of false positives and false negatives is quite high, it provides a quality control which is better than nothing, but it does not guarantee anything.  There can be some quality evaluation models, but this is a different discussion. 

I have written before about the fact that  the need to measure everything is taking over work in the corporate world,  this is also the case in academia,  in many cases universities count papers,  or use some other quantitative metrics, which fits the current publication system.    A new system will require to change this criteria which is an earthquake in the academic system (which needs earthquakes from time to time)...

I think that open access in scientific publication is happening and resistance is futile.....

Sensors as actuators

Phil Windley, the CTO of Kyntex, has reported recently in his Blog about "talking back to a thermostat".  Phil has constructed a thermostat as event producer,  The example he talks about is changing the display color, however one can think on even more meaningful actions like turning on and off the air-condition.  The thinking of sensors as vehicles for human awareness is not enough,  in our universe, there will be more sensors than people, and especially much more sensors than people who are able to look at sensor reading treat is as data (small or big) and make sense of this data.  We are more and more move to era of autonomic computing, the Mars rover "curiosity" is largely autonomic, since the speed of light barrier creates a time delay that makes it virtually impossible for remote control.   We'll see more and more sensors who will also serve as actuators, or smart sensors directly talking with smart actuators (e.g. thermostat to the air-condition

controller).    The required event processing  (filter, aggregation,  maybe even patterns that takes into consideration other sensors' input, like presence of people in the house)  can be distributed between the smart sensor and the smart actuators without any middleware.    More on embedded event processing - later.  
  

How can the level of uncertainty be determined?

You may identify this formula as Heisenberg uncertainty principle,  nature is full of uncertainty and so is the world of business and any other world.  It is not difficult to convince people that uncertainty representation and handling is needed, but people wonder how in practice people will be able to consume and digest uncertainty based systems.   I'll try to refer to it in a series of posts,  the first of them deals with the following issue:  uncertainty handling models assume that the level of uncertainty can be quantified.  Quantification can be in the discrete world, a collection of values, and in the continuous world -- probability, measure of belief, and similar metrics.    The question is -- how can we determine the value that represents uncertainty.

There are three ways to determine uncertainty:   prior knowledge,  observation and  statistical reasoning.

The prior knowledge often exists due to physical properties:  the accuracy of sensor may be a property of the sensor reported by the producer, mathematical models can determine the error rate of physical measurement due to friction or other physical phenomena, this can also be rooted in statistical analysis, but for a various system it is given as prior knowledge.

Observation:  In some cases and observation includes some uncertainty, such as:  I left home somewhere between 8 - 8:30am,  there was an accident somewhere in main street that made me late for the meeting, I heard that the accident was caused due to a drunk driver (but not sure this rumor is true),  I arrived to the meeting a few minutes after the meeting start, and waited a long time to the elevator that took me into the 35th floor.     

This observation is full of inexact fact: time, space, event attributes and more.  People typically does not know how to quantify it, but can use fuzzy terms that can be translated into quantified values either in the discrete space or in the continuous space.

The statistical reasoning path is based on learning mechanism that is based on the ratio between historical input and the real value.  The assumption here is that eventually the real value is known and can be compared to the reported value.

There are some interesting questions about the representation formalism, the coverage that can be obtained by these methods, and the methodology for value assignment.    I'll write about these questions in a following post. 

On what you need vs. what you can consume

In a recent review on our uncertainty project,  the major question that came up was -- will user be able to consume uncertain data, or uncertain decision making?   The question of consumability is a major question -- and it seems that in some cases consumability is considered as a barrier of introducing new technologies.   This is true for all technologies based on statistical reasoning that have the reputation that in order to use them  one has to be PhD in statistical reasoning...   Consumability comes in all levels --- there are products that come with hundreds of configurable parameters to control its performance, some famous expert systems have been conceived to configure these parameters, as the quantity of humans who can successfully do it is too small,  Consumability comes to application developers that use languages and tools.  in many cases the amount of language commands that are not used is high, on the other hand, the developer is working hard to achieve the desired functionality, since it is not supported in a simple way.  Then of course consumability of any product to consumers.  Today we have very sophisticated washers and dryers with many programs, which totally not understood by the consumer,  or just look at a remote control for any home appliance -- TV, Air-condition and more -- do we really know what are all these buttons for?   

Back to uncertainty --- it is quite clear that the world is uncertain, and ignoring uncertainty in decision making  may yield undesirable results.  It is also clear that in the daily life we often make decisions under uncertainty,  the main problem is how to move this intuition to a computerized form in a way that people will be able to utilize.   The major questions are:  

• how uncertainty quantification (such as probability) are obtained?
• how uncertain results are consumed in decisions?
• how quality of uncertain decisions are evaluated and taken into account?

I'll write more about it soon - in general, consumability issue in event processing application design and construction will occupy much of my agenda in the near future.  More - later. 

Is Business Intelligence dead?

Neil Raden posted recently an article entitled " BI is dead! long live BI".  This adds to the trend of declaring some technology or trend as dead:    Earlier this year I wrote a post in this blog, entitled: "Is mapReduce dying?" in which I also indicated some references to claims such as  "SOA is dead" or "ERP is dead". Actually nothing in the IT industry dies quite easily,   I guess that COBOL, which is about my age, is still alive  today, and may outlive me.   What Neil claims is that the success of BI was much less than the hype created around it, and its adoption was around 10-20% in large organization depends on the survey.   This is consistent with other opinions,  James Standen reported  on a survey that found that the the actual adoption is lot less than the percentage claimed  by BI vendors are. The picture at the top of this page is borrowed from his post "Business Intelligence adoption low and falling".     

Neil claims that BI does have a future, but its future is in doing it as part of a decision management framework, not as a stand-along service consumed by people.   When talking about decision management, it has a role to analyze historical data, but doing it within a context of a specific decision.  This can be used to provide input to decision or to reinforce or refute assumptions.  

I think that this is an interesting perspective and in a way has some similarity to event processing which also transitions from stand alone technology to part of a bigger game.   I have discussed it in the past, and still stand by the conclusion that stand alone event processing will stay as a niche, but the mainstream will be part of bigger games.  I have investigated recently the current list of "bigger games"and will write on it in one of the future posts,  however, I see the same observation as valid for BI,  current BI will remain a niche, while the mainstream will move towards embedded BI.   More -later. 

Event processing in practice

My old friend Roy Schulte from Gartner, who has been the first analyst that published analyst reports about event processing, and one of the founding fathers of the event processing community and EPTS has published a "personal blog" on the complexevents site with the provocative title:  " does anybody care about event processing?".   Roy expresses his opinion (that I've heard from him several times before) about the event processing concept and market. Roy's short answer to the question he asks is YES, but many of those who are doing event processing are not aware that they are doing event processing.   He mentions two main issues:

• Terminology:  the term event is manifested in many different ways, while the community centered around the current products have narrow interpretation
• Market:  Most of the event processing is custom code, or built into vertical products, and not the general purpose event processing platforms, thus he had to scale down his prediction for revenue that count the event processing platform market.

Roy suggests that EPTS will decouple itself from the focus on the product oriented functionality and concentrate on event processing as technology in the broad sense.   

I concur with Roy's observations (we have discussed it last year when I visited him in the Gartner office);  I have written before about the build-buy trade-off,  based on chapter 1 in the event processing manifesto, a chapter that Roy was one of its authors.      I also noted in some of the feedback to my book that I got from developers who told me that the book helped them a lot in architectural thinking and clarity of concepts, but they also said that they'll continue to develop event processing applications in custom code.  

Roy's Blog has been written in the context of the discussions on EPTS and online magazine,  and I have expressed my opinion in the same spirit as Roy --  the online magazine (and EPTS) should appeal to anybody that are doing anything with any kind of event in any form.  This should capture all the "event processing in practice" in the universe.  More news about the online magazine -- soon (the editorial board is now being created, which contains both old members of the community and new blood of practitioners).