Michael Jordan on the delusions of big data

Michael Jordan (the one from Berkeley, not the basketball player) gave an interesting interview to IEEE Spectrum.  it is recommended to read his own words. 

Some of the highlights of Jordan's opinions are:

1. Using brain metaphors for computing is misleading:   computing does not work like the brain, this is also includes one of Jordan's expertise areas - neural nets.
2. He says that the advances in computer vision lead us to be able to solve some kind of useful problems, but we are very far from giving machine the vision capabilities of a human
3. "Big Data" is over-promising.  One can prove many false hypotheses using big data methods.  This is similar to building bridges without a theory of how to build bridges,  some may  survive, and some will collapse... 
4. If he will have $1B to spend on research, he will invest in natural languages processing...

I think that it adds to some other observations about the overhype of "big data" (for example, see my posting on Noam Chomsky's opinion couple of years ago, or Tim Harford's recent article). 

Event processing platforms - reboot?

Doug Henschen, the editor of Information Week wrote a commentary entitled "big data reboots real-time analysis" .  Henschen says that event processing was in the height of its hype in 2008, but the economic crisis stopped the growth of this area.  He sees indications of "reboot" in the recent acquisitions of Apama by Software AG and Streambase by TIBCO, and attributes the reboot to the need of big data to evolve from its batch origins to detect patterns on moving data.  

As I have written before, the barriers to growth stem from some external factors (certainly the general financial situation), but also the over-hype of request-response or batch oriented analytics (see my post on Sethu Raman's keynote in DEBS 2012).  Another reason, as observed by Roy Schulte last year,   is that many enterprises developed in-house solutions.    I assume that Henschen is right in the sense that big data gives additional opportunities to event processing technology, and that the recent acquisitions will create waves of interest in the market.   As I have written before, the next frontier is not improving the technology, but making it accessible to the business users and convert the enterprises to think in an event-driven way.   Jeff Adkins and myself will discuss this issue in the coming DEBS'13 tutorial, on June 30.  More - later. 

On disciplines and marketing devices

Yesterday I participated in the "parents teaching" program in my third daughter's junior high (8th grade) and gave the children a short introduction to the issue - does a computer think ? I did not give them an answer for this question, but gave them several basic puzzles and explained them how we can teach a computer to solved them -- one of them has been the old good missionaries and cannibals problem.



From the question --- does a computer think, I will move to the Blog of Hand Glide who phrased his posting in a form of a question -- CEP is a marketing device, so what does it say about CEP products ?

The answer is --- not much.

Let's change the TLA from CEP to SOA and ask the same question, the answer is that there are good and bad products that are marketed under the TLA of SOA, some of them have been here before SOA, and maybe some of them will be here if another TLA will dominate.

I have blogged before about the various interpretation of CEP, and the observation about what is called "CEP products" is that there is a variety of implementations that call themselves CEP, this does not teach anything about the quality of these products, their benefits to the business etc...

While TLAs became the property of marketing people to position products, somehow disciplines consist of one or two words such as: data management, image processing, graphics, information retrieval and many more - that's why I consistently use "event processing" when talking about the discipline.

Disciplines normally start in multiple places that try to solve similar (but not necessarily identical) problems, first generation of product is developed, and sometimes also hype is created and this is consistent with the "hype cycle" concept of Gartner. In the EPTS conference Brenda Michelson has argued that if anything this area is under-hyped and not over-hyped. There are some other indications that support her observation.

The early phases of a discipline lacks standard, agreed upon theory, and coherent thinking.
In the OMG meeting, March 2008, I have used the following slide as an example of what are the indications/conditions for a discipline to succeed:

The fact that EP is not in the maturity level of relational databases or some other more mature discipline is obvious, however, while there are people who made a career out of criticizing and complaining that what other people are doing is not good enough, I think that our challenge is to advance ---- it took years until there was an agreement what a relational database is, during which all databases suddenly became relational (to anybody old enough to remember, there were some funny situations of products that claim to have relational extension, when they did not understand the term), we need an event processing manifesto, and a collection of standards, but they will not be constructed in a single day, so we also need patient and persistence... I believe that EP will be 10 years from now one of the major disciplines of computing, and that we have the challenge to get there...

BTW - I agree with Hans that if products have business value for customers, they will be used regardless of the fact if at the end they will be classified EP or not. more - later

event processing meets artificial intelligence

Bedford, MA, USA.

In the EPTS symposium last week, Alan Lundberg from TIBCO, who moderated the "business panel" made the analogy to AI, especially to "Experts systems", saying that there was a hype in the beginning, and people believed it will solve many of the world problems, and in the reality, it did not recover from sliding down in the hype cycle, this triggered the (somewhat surprising to some) response of Brenda Michelson, that actually EP is under-hyped, and its place in the hype-cycle is much lower in the climbing phase than the Gartner analysts draw, this is the diagram that Brenda presented with "event processing" in orange, way below SOA (in blue), BPM (in red), and Web 2.0 (in green).

Anyway - this is not the topic of today's Blog, but going back to the AI issue. The term AI is interesting, in the sense that it has spawned several disciplines (e.g. robotics, image processing, information retrieval, data mining and more) which are based on AI principles, but when they mature they stop being AI and become disciplines of their own. This is the same phenomenon we have for philosophy - the mother of all arts and sciences - many disciplines has emerged from philosophy, but when they depart, they are not considered as philosophy anymore. Event processing as a young discipline, is a descendent of multi disciplines as stated in the past, AI is certainly one of them.

What are the current topics in which AI touches event processing?

1. Modeling: the basic term "situation" and "context" have been taken from AI (situation calculus), conceptual modeling is important for design of EP applications, AI techniques can help here

2. Discovery: Prediction of events, mining of patterns - these are all derivatives of machine learning in AI.

3. Reasoning: Defining precise semantics of both event processing languages and execution models. Evidently from the recent discussions in the community, this becomes an important topic - again, precise reasoning of both the regular case of event processing, and the extended case of handling uncertain events.

As my colleague Guy Sharon described in the research session of the EPTS meeting, we in IBM Haifa Research Lab (together with some colleagues in IBM Watson Research Center) are engaged in the "Intelligent Event Processing" project that concentrates now on the discovery aspects, however, the idea is to extend the activity probably through collaborative work with the academia, as part of this collaboration we are organizing the "Intelligent Event Processing" workshop which will take place as one of the AAAI spring symposium series that will take at Stanford University, March 2009. The idea is to have the EP community meet the AI community and create partnerships to deal with these issues... so - target this conference for paper submission and/or attendance. More - later.