Big data analytics will never replace creative thought

The claim expressed in the title of this posting is the title of  a piece in "Data Quality News" by Richard Jones.   It claims that the "data craze" - the conception that data mining alone is sufficient to get decisions in all areas, is a misconception in some areas.  Jones provides two examples:  marketing - where statistical reasoning gives a great value, but it deals with the small details, however  human creative thinking deals with the big picture, and data mining alone cannot get it,  and healthcare - again, data mining can be of great value, but interactions with the patient and personal examination by a physician is vital.    
I guess that the research into AI should also deal with how to create artificially creative thinking.  As I've written before Noam Chomsky has criticized the AI community by making statistical reasoning its mainstream and deserted the strive for  "solid model of the universe" .  I guess that after some disillusionment from the "data craze" the industry will settle on getting data mining its right place, as a supporting technology.

More on this - later.

On machine learning as means for decision velocity

Chris Taylor has written in the HBR Blog a piece that advocates the idea that machine learning should be used to handle the main issue of big data - decision velocity.  I have written recently on decision latency, which according to some opinions - real-time analytics will be the next generation of what big data is about.

Chris' thesis is that the amount of data is substantially increasing with the Internet of Things, and thus one cannot get a decision manually in viewing all relevant data,  there will also not be enough data scientists to look at the data.   Machine learning which is goal oriented and not hypothesis asserting oriented will take this role.     I agree that machine learning will take a role in the solution, but here are some comments about the details:

Currently machine learning is off-line technology, case sensitive, and cannot be the sole source for decisions.

It is off-line technology, systems have to be trained, and typically it looks at historical data in perspective and learns trends and patterns using statistical reasoning methods.  There are cases of applying continuous learning, which again done mostly off-line, but is incrementally updated on-line.    When a pattern is learned it needs to be detected in real-time on streaming data, and here technology like event processing is quite useful, since what it does is indeed detect that predefined patterns occur on streaming data.  These predefined patterns can be achieved by machine learning.    The main challenge will be the online learning -- when the patterns need change, how fast this can be done in learning techniques.  There are some attempts at real-time machine learning (see presentation about Tumra as an example), but it is not a mature technology yet.

Case sensitive means that there is no one-size-fits-all solution for machine learning, and for each case the models have to be established in a very specific way for that case.  Thus, the shortage in data scientists will be replaced by shortage of statisticians,  there are not enough skills around to build all these systems, thus the state of the art need to be improved to make the machine learning process itself more automated.

Last but not least - I have written before that get decisions merely based on history is like driving a car by looking at the rear mirror.  Conclusion from historical knowledge should be combined with human knowledge and experience sometimes over incomplete or uncertain information.  Thus besides the patterns discovered by machine learning, a human expert may also insert additional patterns that should be considered, or modify the machine learning introduced patterns.

On event-driven, request-driven,stateful and stateless

This slide is taken from our DEBS 2013 tutorial, explaining what are the differences in thinking between the traditional request-driven way and the event-driven way.   It shows the differences by answering three questions.     This goes back to the differences between business rules and event processing, and old topic, on which I have written first time around 6 years ago!    One of the claims that I've heard several times is that the distinction between them is that business rules are stateless and event processing is stateful.     I think that the main difference is that business rules are treated as request driven,  the rule is activated on request and provides a response, while event driven logic is driven by event occurrence as shown in the slide above.

While it is true that there is correlation between event based/request based and stateful/stateless,  these are really orthogonal issues.

Event-driven logic can be stateless.  If we only wish to filter an event and trigger some action,  this can be stateless (most filters are indeed stateless), but it has all the characteristics of event-driven, including the fact that if the event is filtered out - no response is given.   

On the other hand -- a request-driven logic may be stateful, there are many instances of session oriented and other stateful request-response protocols.    One can also implement stateful rule engine in a request-response way, where invocation of rule is based on result of previous rules that are retained by the system.  

Bottom line:  stateful vs. stateless is not equivalent to event-driven vs. request-driven.   

On the role of chief evangelist

Recently there has been an announcement that Theo Pristley joined Software AG as "chief evangelist". 

I thought this is an interesting title and wondered what is behind this title. Today Theo Pristley published a Blog post with his interpretation of the term.  According to Pristley the three main point of this role are (according to my own interpretation):  telling the story, connecting the dots and forming influential opinions. 

There has been some posts before on technology evangelism, for example, the one from which I copied the above picture,  saying that  "Evangelist is born to learn, speak, share, sell and inspire the masses towards the technology or product they are passionate about".

 I always believed that it is important to a person to work on things that they are passionate about. Early in my career I have taken some management course in which the instructor said: "if you are not getting up every morning being enthusiastic about what you are doing - you are not in the right place, do something else!".  

I realized that this assertion is true for me, and I went to do something else (study for Ph.D) - true story!

Some people have used the term about me claiming that I am evangelizing "event processing" (although, unlike the official evangelists nobody ever paid me to do it). While I find it difficult to identify myself with a term associated with religion, I think that I do preach event processing for various aspects for years in different ways (of course, I am not the only person doing it).  

Actually, in the last slide of the tutorial that Jeff Adkins and myself delivered in DEBS 2013,  we put the following statement:

BTW - Jeff has on his business card the title "connecting the dots", which is kind of part of the evangelist role according to Prisetley.    I guess that this is different from the role of chief evangelist in a software vendor level that is looking across the vendor's portfolio.  I generally believe that people with bright eyes are happier and working better - and that people who can inspire people to brighten their eyes are great asset.believe that people with 

New name and face-lifting to David Luckham's site: complex event processing & real time intelligence

In this picture, taken seven years go, you can see me (somewhat heavier than I am today) with David Luckham (in the middle) and Roy Schulte (in the right hand side).  I have talked with David Luckham recently and found out that he is still maintaining his website, and now also face-lifting it.   On the website he also explains why he has changed the name.   According to Luckham, the name "complex event processing" is strongly identified with the event processing platforms, dedicated software to do event processing,  but this is a tiny fraction of the market, where the bigger market is embedded event processing inside other software.  This observation has been done before by various people (including myself), so I believe he is right. Luckham calls the bigger game as "real time intelligence", some people call it "real time analytics", but  there is an agreement that it is part of the big data game (and other games) and that event processing is in its backbone.   It will be interesting to see if such branding will catch on, and how exactly "real time intelligence/analytics" will be defined -- I'll write more about it.  

Taking the complex out of complex event processing

The quote of this week is taken from an article in InformationAge that talks about operational intelligence. 

The article explains what operational intelligence means, and you can read it to see if you find anything new.

The point of this post is a quote done by Ivan Casanova from TIBCO:  

“We should all be focused on taking the 'complex' out of complex event processing" 

This quote is in the context of explaining the acquisition of Streambase by TIBCO.    I don't know Mr. Casanova personally, but what I have learned from his statement is that he believes that going forward, the programming model and tools represented by Streambase are better fit and less complex to use that TIBCO has done before, where it extended RETE based business rules system to handle stateful event processing cases, while retaining the rule-based programming model.    Streambase is using an "event flow" model that is some variation of event processing network.    Without getting to analysis of specific products (a restriction I have taken upon myself in this Blog), I would say that overall I believe that as a conceptual model for event processing I believe in the EPN model (which is of the family of data flow models),  and in visual working environment (better than textual working environments) to design and program.   This reduces the complexity for IT developers, which I think is very important trend.   The ultimate reduction of complexity requires one more step -  event processing modeling in the level of the business user level and automatic translation to an implementation language.  

Bottom line: I agree with the statement in the quote -- actually this is my main area of interest nowadays. 

On continuous compliance

The Waters special report sponsored by Apama that was published recently,  ranked "risk and compliance" as the number one application of event processing in financial institutions.   Today TIBCO also published in its Blog under the "event processing" category its view on compliance - entitled: "The Y in comply".   It gives the view that people comply better when they understand the rationale of compliance, the example given is compliance with regulation in the food industry that employee has to wash hands before putting on gloves.  

I agree that people comply better when they think that the regulation makes sense,  however, my own subjective impression is that in the current culture, compliance became goal of its own whose sole reason as somebody described it "to get the regulator off my back".   Thus employees are required to do things that they don't see any reasonable explanation why they should do it, and IT systems check that they are doing it. Here, event processing is used  since in various cases auditing is event-driven, it is time sensitive an often involves calculation in time windows, and in some cases online auditing on the fly is required. The business value sometimes is just pleasing regulators, but this is often a great motivation for managements to take it very seriously...   Anyway, technologies are used for strange reasons sometimes...