A recent article in the science section of TIME magazine reports that prediction using "big data" techniques is not as easy as portrayed. It analyzes the Google Flu Trend case, in which the assumption has been that there is a strong correlation between the spread of flu, and the searchers for flu related terms in Google. It seems that this does not produce accurate results. The article claims that while using the big data methods is useful, they should be combined with traditional "small data" methods. There are various definitions of what a small data is - for example, the one from "small data group" : Small data connects people with timely, meaningful insights (derived from big data and/or “local” sources), organized and packaged – often visually – to be accessible, understandable, and actionable for everyday tasks.
I guess that this also relates to the discussion about understanding causality in addition to statistical correlation that I've discussed before on this blog.
There have been some visible floods in recent years, like the one in Australia. Prediction of the course of floods can be tricky, since rivers can have multiple splits. Today IBM announced that IBM Research together with UT Austin a prediction model to predict the course of flooding. This combines IBM analytics research with UT research in the physics of rivers.
Note that disaster management is an area where proactive computing hold significant potential, however in order to realizing this potential, strong prediction abilities are required.
There are several works that attempt to use social networks to detect human sentiments. In a recent article, John Bates (Progress Software CTO and one of the founding fathers of the event processing discipline and community) reacts to some claims that using Twitter one can predicts trends in the stock market with high accuracy. While the article deals with stock market applications, I'll give some comments within a broader context of detecting human sentiments in any domain. I'll quote John's claims in italics and different color and mine in this font and color.
John states three problems with that assumption:
- Social networks are not secure, thus the results may not be accurate, and may also be malicious or just wrong. His example is that somebody twits about the fact that USA started a war on France. If somebody uses it to effect the stock investment, or even travel plans, the results may not be that good. I agree that this is a major issue.
- Social networks are unlikely to contribute anything that is not being disclosed in mainstream news. I am not sure that I totally agree with this one, sometimes individuals on twitter or facebook reveal news before the mainstream media detect it, we saw some examples recently all over the world.
- The time it takes to process the sentiments may make the results obsolete, since the reality may move faster. I guess that it might be true in some cases, but with real-time analytics, it may be good enough in other cases.
Overall, the first problem stated is the most serious one. Perhaps when there will be some validation mechanism to social networks postings, then it can be more reliable. One may claim that if we can assume that most posts on social networks are correct, then there can be noise removal filter. But still social networks may not be reliable sources (BTW - mainstream media news may not be accurate either!).
It should be noted that I've learned about John's article since he twitted about it on Twitter. So maybe the fact he wrote this article by itself is not accurate? More on this -later.
An exhausting week, followed by over-sleeping during the weekend (our weekend is Friday and Saturday), and a 2.5 hours session with the dentist in the middle.
This week I heard in one of the IBM internal meetings some impression from a customer meeting in Europe.
The topic was data mining, and the insight has been that mining is trying to learn patterns based on the past behavior. This works in some cases, but might not be enough in other cases; the reason is that in some cases what we actually need to learn is human behavior, and assuming that past behavior is a good indicator may be wrong, thus the domain should be extended to analyzing and predicting human sentiments and human behavior. This is, of course, not a new area, and the behavioral sciences people are studying it for years, one notable work is the work on prospect theory, that brought Daniel Kahneman the Noble prize (the co-inventor of the prospect theory, Amos Tversky, passed away before the Noble prize was given, and the prize is being awarded to living persons only). There are other works in this area, and some are being used in reality. Getting multi-disciplinary work is quite important for enabling intelligent systems and enable predictive analytics. More on this - later.
The last few days in Israel were extremely hot, one local newspaper claimed that Friday was the hottest day in Israel within 112 years. Now it is somewhat less hot, but still very hot. Relief is expected later this week.
I am playing now with the new editor of the Blog editor, which looks like Wiki editors, it seems that web editors are starting to converge into some form.
Anyway -- recently I have read some "business intelligence" stuff - ("analytics" is now a hot buzzword in IBM, and probably outside IBM as well). In business analytics terminology people talk about three phases: descriptive, predictive and prescriptive, while in event processing we also talk about three phases: responsive, reactive and proactive. So I was asked - are those terms equivalent. The answer -- not exactly.
Let's start with business intelligence, or analytics in general. The main starting point is: we have historical data, we can present it in different ways, we can learn from it something that can provide observations, and can predict future data (e.g. by trends) and then we can propose actions to bridge gaps towards our goals.
The basic starting-point -- analyzing existing past data. The first phase is descriptive -- describes what is seen in the data, this is the most common use of business intelligence.
The second phase is - predictive, find trends and extrapolate into the future, predicting future values of the same data.
The third phase is prescriptive - given the predicted data, and possible gaps between this predicted data and the enterprise's goals -- propose a way to bridge the gap, e.g. change inventory policies, change risk policies, even getting to change business processes.
Event processing is starting from different viewpoints - there are events happening now, and we would like to react to them -- the metaphor is -- a dangerous bear is approaching and I need to react.
In event processing the evolution is starting in "responsive" - in this case, indeed event is treated as data, information about events arrive using queries, search, or even applying any kind of analytics, this is the regular mode of programming, but it is data-driven rather than event-driven. It may be applicable to some applications, will not be very helpful in the case that the bear is chasing you. Event driven architectures and programming has enabled the next phase in the evolution - reactive programming, in which predefined alerts or actions can be triggered by the fact that an event has detected, or that an event pattern has been identified. Currently the state-of-the-practice in what is defined under event processing applications fits this category. The next step in the evolution is proactive, which means that by computerized means we'll be able to identify predicted events, and then a decision of how to mitigate or eliminate the event is being taken, for example when a bear is chasing me, I need quickly to decide whether my best bet is to hide, escape, or shot tranquilizing darts at the bear
Having explained the basic terminology, back to the original question, how are these terms related.
First, the goal of business intelligence and event processing are typically distinct, however there are some points of overlap. From the BI perspective, reactive event patterns can be used as a component of predictive analytics. Proactive event-driven processing can be thought as a type of prescriptive analytics. The overlap occurs when the analytics system has real-time component, which requires that the prescriptive analytics will be done on-line and with some timing constraints, this turns it from being data-driven to be event-driven, but one can think of prescriptive system that is totally off-line - analyzing data in batch, predicting shift in trends, and change the policies for the next year/quarter.
From event processing perspective, analytics tool can be used in populating the event patterns, but this is not that easy -- I'll write soon about some thoughts on the feasibility of patterns learning.
