On the reactive manifesto

Today I am writing from Luxembourg, where I participated in the negotiation meetings of two EU projects that are being launched.   I'll write about them when they will actually signed.  

Not far from here, in  Dagstuhl 2010, we worked on the "event processing manifesto" .   Today I discovered another manifesto that I think can be tracked to EPFL in Laussanne (not completely sure since the manifesto authors don't identify themselves), the manifesto is called "The REACTIVE MANIFESTO"  and is dated July 15, 2013.   The picture above is copied from the manifesto, and as you can see they define REACTIVE as event-driven, scalable, resilient, and interactive.   I wonder what is the background and motivation about it,  perhaps one of this Blog readers will be able to shed light...

Gartner's Hype cycle on emerging technologies 2013

Hype cycles time of the year, and Gartner published the emerging technologies hype cycle for 2013. 

Some insights: 

1.  "Complex Event Processing" is still near the peak of inflated  expectations, actually moved a little bit below the peak.  This means that it started the process of filtering out the hype, and getting to realistic contributions.
2. On the analytics front -- predictive analytics is now at the plateau of productivity, and on its way to being a commodity, while prescriptive analytics is on the rise, but still in the innovation phase.
3. Human augmentation, brain-computer interface, quantum computing and mobile robots are on the rise.  In fact, the Gartner's press release emphasizes the human - machine relationships.
4. In adjacent technologies to event processing; Internet of Things is getting closer to the peak, In memory DB and context analytics are also getting past the hype peak. 
5. Big data is still in the height of the hype --- as we saw in other sources,  it is now recognized as a catch-all hype, and I guess that it spawn several distinct concepts in the future.
6.   Mobile phones/tablets etc are not mentioned explicitly as part of the emerging technologies, I guess that mobile by itself is not a technology -- it has influence on all other technologies (the same as the WEB is not a technology).

New market research on the event processing market by Markets&Markets

It seems that there is a new comprehensive market research on the event processing market in the years 2013-2018 by Markets and Markets.  I don't have the market research itself (it is quite expensive), but the site gives some details, according to the report,  Markets&Markets forecast that  the "CEP  market"  is expected to grow from $764.5 million in 2013 to $3,322.0 million in 2018.  I wonder what this figures represent, it seems that this is beyond the accumulative sales of event processing platforms. 

They also classify the market according to the following verticals: 

BFSI: algorithmic trading, electronic transaction monitoring, dynamic pretrade analytics, data enrichment, fraud detection, governance, risk and compliance (GRC); 

Transportation and Logistics: asset management and predictive scheduling and toll system management; healthcare: self-service proactive monitoring and alerting and governance, risk and compliance (GRC); 

Telecommunication: mobile billboards, revenue assurance, network infrastructure monitoring and predictive CDR assessment; 

Retail: inventory optimization, shoplifting detection and real-time marketing and customer engagement; 

Energy and utilities: oil and gas operation management and nuclear crisis and smart grid energy management; 

Manufacturing: shop floor automation and operational failure detection, infrastructure management and supply chain optimization; 

Government, defense and aerospace: Intelligence and Security, emergency response services and geo-fencing and geospatial analysis; 

Others: includes education and research

Hope to get more insight towards this research. 

On concept computing - take one

We think in concepts.  We study concepts, we reason about concepts.   

Now we also have "concept computing", the term was coined by Mills Davis.  It does not appear in Wikipedia yet, but it is an interesting and useful idea.  Mills Davis uploaded his AAAI keynote talk on Slideshare recently, and the slides below is taken from there.   The work we are doing now is somewhat the projection of this idea for the event-driven world.  I'll write about it in the future.  Meanwhile -- this presentation is recommended 

Big data analytics will never replace creative thought

The claim expressed in the title of this posting is the title of  a piece in "Data Quality News" by Richard Jones.   It claims that the "data craze" - the conception that data mining alone is sufficient to get decisions in all areas, is a misconception in some areas.  Jones provides two examples:  marketing - where statistical reasoning gives a great value, but it deals with the small details, however  human creative thinking deals with the big picture, and data mining alone cannot get it,  and healthcare - again, data mining can be of great value, but interactions with the patient and personal examination by a physician is vital.    
I guess that the research into AI should also deal with how to create artificially creative thinking.  As I've written before Noam Chomsky has criticized the AI community by making statistical reasoning its mainstream and deserted the strive for  "solid model of the universe" .  I guess that after some disillusionment from the "data craze" the industry will settle on getting data mining its right place, as a supporting technology.

More on this - later.

On machine learning as means for decision velocity

Chris Taylor has written in the HBR Blog a piece that advocates the idea that machine learning should be used to handle the main issue of big data - decision velocity.  I have written recently on decision latency, which according to some opinions - real-time analytics will be the next generation of what big data is about.

Chris' thesis is that the amount of data is substantially increasing with the Internet of Things, and thus one cannot get a decision manually in viewing all relevant data,  there will also not be enough data scientists to look at the data.   Machine learning which is goal oriented and not hypothesis asserting oriented will take this role.     I agree that machine learning will take a role in the solution, but here are some comments about the details:

Currently machine learning is off-line technology, case sensitive, and cannot be the sole source for decisions.

It is off-line technology, systems have to be trained, and typically it looks at historical data in perspective and learns trends and patterns using statistical reasoning methods.  There are cases of applying continuous learning, which again done mostly off-line, but is incrementally updated on-line.    When a pattern is learned it needs to be detected in real-time on streaming data, and here technology like event processing is quite useful, since what it does is indeed detect that predefined patterns occur on streaming data.  These predefined patterns can be achieved by machine learning.    The main challenge will be the online learning -- when the patterns need change, how fast this can be done in learning techniques.  There are some attempts at real-time machine learning (see presentation about Tumra as an example), but it is not a mature technology yet.

Case sensitive means that there is no one-size-fits-all solution for machine learning, and for each case the models have to be established in a very specific way for that case.  Thus, the shortage in data scientists will be replaced by shortage of statisticians,  there are not enough skills around to build all these systems, thus the state of the art need to be improved to make the machine learning process itself more automated.

Last but not least - I have written before that get decisions merely based on history is like driving a car by looking at the rear mirror.  Conclusion from historical knowledge should be combined with human knowledge and experience sometimes over incomplete or uncertain information.  Thus besides the patterns discovered by machine learning, a human expert may also insert additional patterns that should be considered, or modify the machine learning introduced patterns.

On event-driven, request-driven,stateful and stateless

This slide is taken from our DEBS 2013 tutorial, explaining what are the differences in thinking between the traditional request-driven way and the event-driven way.   It shows the differences by answering three questions.     This goes back to the differences between business rules and event processing, and old topic, on which I have written first time around 6 years ago!    One of the claims that I've heard several times is that the distinction between them is that business rules are stateless and event processing is stateful.     I think that the main difference is that business rules are treated as request driven,  the rule is activated on request and provides a response, while event driven logic is driven by event occurrence as shown in the slide above.

While it is true that there is correlation between event based/request based and stateful/stateless,  these are really orthogonal issues.

Event-driven logic can be stateless.  If we only wish to filter an event and trigger some action,  this can be stateless (most filters are indeed stateless), but it has all the characteristics of event-driven, including the fact that if the event is filtered out - no response is given.   

On the other hand -- a request-driven logic may be stateful, there are many instances of session oriented and other stateful request-response protocols.    One can also implement stateful rule engine in a request-response way, where invocation of rule is based on result of previous rules that are retained by the system.  

Bottom line:  stateful vs. stateless is not equivalent to event-driven vs. request-driven.