Showing posts with label analytics. Show all posts
Showing posts with label analytics. Show all posts

Sunday, March 9, 2014

Big Data analytics by Robin Bloor

Today I had to give students in a seminar introduction to big data analytics -- I chose a recent presentation by Robin Bloor (from slideshare). Bloor states that the term "data science" is a misnomer, since all science is empirical and involves analysis of data.   This is true for many of the sciences, still if my memory does not mislead me Einstein did not use empirical analysis of data to come with the relativity theory.  It also goes to the discussion of causality vs. correlation in science.   In any event, Bloor asserts that data science is actually a multidisciplinary efforts involves software engineering, statistics and domain knowledge. 

BI, according to this presentation, is partitioned to:  
  • Hindsight: regular reporting
  • Oversight: dashboards etc,
  • Insight: data mining & statistical analysis
  • Foresight: predictive analytics
He does not get as far as prescriptive analytics,  and puts the heavyweight on the insight. 
The second part of the presentation gives fast introduction to machine learning.       Overall, it gives introductory level insights on insights from big data, and is well presented as such.

Friday, December 20, 2013

Gartner's recent predictions about business intelligence and analytics



A recent set of predictions by Gartner, state that Business Intelligence and Analytics will remain top focus for CIOs through 2017. 

It mentions two interesting observations:   One is that the confusion in the market about the term "big data" and its tangible results constrain the spending and limit the growth of BI and analytics software.   The second observation is that By 2017, more than 50 percent of analytics implementations will make use of event data streams generated from instrumented instrumented machines, applications and/or individuals.  

This is consistent with the Gartner's term -  "two tier analytics",  where event processing is the second tier, after historical data analytics.  While the need to consolidate analytics and event processing is becoming more pervasive, the utilization barrier and the need to battle complexity is still a common denominator. 

Tuesday, August 20, 2013

Big data analytics will never replace creative thought


The claim expressed in the title of this posting is the title of  a piece in "Data Quality News" by Richard Jones.   It claims that the "data craze" - the conception that data mining alone is sufficient to get decisions in all areas, is a misconception in some areas.  Jones provides two examples:  marketing - where statistical reasoning gives a great value, but it deals with the small details, however  human creative thinking deals with the big picture, and data mining alone cannot get it,  and healthcare - again, data mining can be of great value, but interactions with the patient and personal examination by a physician is vital.    
I guess that the research into AI should also deal with how to create artificially creative thinking.  As I've written before Noam Chomsky has criticized the AI community by making statistical reasoning its mainstream and deserted the strive for  "solid model of the universe" .  I guess that after some disillusionment from the "data craze" the industry will settle on getting data mining its right place, as a supporting technology.

More on this - later.

Saturday, November 24, 2012

The big data hype cycle 2012

I haven't written in the last few days,  I have been in EU project review (as a reviewer) in Brussels and also had some time to be tourist, and climbed the Atomium, Brussels known icon

and  in several museums in center city, taking refuge from the rain 

  including the famous Magritte museum.   I have imported some Belgian chocolate (most of it was already given away)  and a Belgian virus, with whom I am struggling in the last couple of days.

I also came across the Gartner's big data hype cycle for 2012 -- the first time in which Gartner chose to look at big data as an area.


You may notice that "complex event processing" is around the peak of the diagram.

It seems that this hype cycle made Irfan Khan, CTO of Sybase quite furious, his firm reaction was:
"Gartner dead wrong about Big Data life-cycle".    Khan claims that Big Data is not a hype but a reality, and expectations are under-inflated not over-inflated since it can do much more than what people assume.

I guess that there is growing adoption to technologies associated with Big Data, but I don't think that it reached the plateau of productivity, as Khan's claims,  since this is not around whether there are mature products (by the vendors' conception), but around the utilization in industry, and it is difficult to say that most organizations had good exploitation of such technologies.  Furthermore, Khan's claim that Big Data is under-inflated actually shows that the plateau of productivity has been reached.   

In any event,  the event processing angle is interesting.  Note that originally event processing appeared in the hype cycle of enterprise architecture for several years.  In 2012 event processing does not appear explicitly, 
Big Data appears as one block in the top.  This shows that event processing has migrated (at least in Gartner's mind) from the middleware world into the analytics world,  and this is also compatible  with some of the current trends, but this should be a subject of another posting - coming soon. 

Wednesday, October 10, 2012

SAS announcement on event processing


SAS announced today that a new "SAS DataFlux Event Stream Processing Engine" will be available in December.  It is described as: "the new software is a form of complex event processing (CEP) technology...incorporates relational, procedural and pattern-matching analysis of structured and unstructured data".     Welcome to the event processing club,  this seems to be an indication that the analytics guys see the value of adding event processing to their portfolio, I guess that either the "limited appeal" of event processing has somewhat changed in the last couple of years to justify it.  Anyway - I welcome SAS to the club, and hope that they will also become active  part of the event processing community.  


Tuesday, July 17, 2012

DEBS 2012 - keynote by Sethu raman


This morning started by some statistics about paper acceptance by Patrick Eugster, the PC chair. It seems that the acceptance rate in the scientific track was lower than previous years.  

The first keynote was given by Sethu Raman, who was until recently the "distinguished engineer" in Microsoft responsible for StreamInsights.    Sethu gave a relatively short talk (with a lot of questions and discussions) entitled "event processing solutions in the enterprise - a waypoint or launchpad". The illustration above is taken from his opening slides, he had such nice illustrations and citations all over.

You can download Sethu's presentation from the DEBS 2012 webpage (the organizers is doing a good job by copying the presentations from the presentation computer and posting it on the website quickly). 

The main message of Sethu is not surprising (as seen from the information management universe):

  •  Event processing has a big potential to play in multiple roles -- real-time analytics,  ETL.  distributed query framework 
  • It also plays in many different industries and is not a niche play
  • There is a major issue is of conception.  There is aggressive messaging of the "analytics" community which drowns the event processing message in some cases,  The analytics community claim that they can solve all problems of the universe by database centered way, and that event processing is required only for small niche of very low latency requirements that cannot be satisfied by DBMS technology.
  • The reality is that the value proposition of event processing is not restricted to the low latency, it is actually reducing the high cost of developing applications - however this message is not heard in a clear enough voice, relative to the other aggressive voices.
  • The term "complex event processing" is a confusing term,

I have discussed it with Sethu in the break,  I actually agree to all the assertions.  The over-hype of analytics, and the messaging that comes sometimes from the analytics people (which is not really backed up by technical facts) makes the impression that analytics subsume the functionality that is proposed by event processing.   I am not sure how ling the over-hype of analytics will continue (I think this is quite high in the hype-cycle and will get to the disillusionment phase, since it is indeed over-hyped),  but the messaging of the event processing community about its ROI as complementary and sometimes contrasting to analytics should be cleared heard.   How? -- this is part of the discussion about the EPTS online magazine.

More - later 

Monday, May 14, 2012

Driving while looking at the rear-view mirror



Typically I don't recommend commercial Blogs, but I'll make an exception this time, and cite Mark Palmer's post, since I like the metaphor he made, reflected in this picture.  According to Mark, analytics that refers to the past is like driving by only looking at the rear-view mirror,  which of course can show the road you have already passed.  Since typically we drive forward and not backward it should be more useful to look ahead than to look back.   In many cases the road is fixed, in the sense that the road forward looks exactly like the road backward, and then it might make sense to do it,  however, in other cases, like driving in real traffic and not in a bubble, the road ahead may contain surprises that are not evident from the previous parts of the road. 
The interesting thing is that even law latency event processing system are, in fact, looking at the past,  where the past is almost the present,  looking to the future is not a standard event processing feature.   

Sunday, April 8, 2012

On two-tier analytics


Will Cappel from Gartner has written about two-tier analytics and went back to Immanuel Kant (in the picture above) as support to his thesis.   Kant argued that the human cognition work in two levels:  the first level that grasps objects and raw facts about them, the second level which captures causality between these objects over space and time, applying some levels of simplification to what Kant said, he is right.  Cappel makes the analogy to the analytics world, and says that the first level is satisfied by event processing that process events by filtering, transformation and pattern detection to identify higher level situations.  The second level is satisfied by pattern discovery engines that work on top of the first level.    This is an interesting observation, I think that the picture is somewhat more complicated as there are more tiers.    Event processing detect patterns in real-time, and indeed one of the ways to obtain these patterns are the pattern discovery mechanism over historical data, which may include the results of event processing systems, but should also include many other data items that describe the impact on the environment,  since situation detection triggers actions, and actions impact the environment,  the pattern discovery needs feedback from the outcome along with feedback from the process itself.    The interesting part comes when we add real-time adaptation to the picture, here, in a similar thing to how the cognition works,  the causality relations may change on the fly.   Consider traffic management systems, studies show that these systems are chaotic in nature and one cannot forecast patterns of behavior based on past experience with sufficient accuracy, forecast is limited to about 15 minutes to the future in some cases and the control policies for highway should constantly adapt.    Here we need four tier analytics:

The first tier is the off-line tier which change the setting of the system based on historical learning.  
The second tier is the event processing tier which observes and monitors
The third tier is the real-time forecasting tier which adapts the causalities and make the short term forecasting
The fourth tier is the real-time decision making tier which makes the best decision possible within the time frame allocated for the decision (which may not be the global optimized solution).

Bottom line:  I agree with Cappel about the multi-tier approach,  and pointing out that reality is somewhat more complicated...   


Saturday, July 30, 2011

More on event processing as analytics/business intelligence



This is nice  picture I borrowed from a BI blog,  talking about seven ways to report BI results.  


Last week I have written on the fact that nowadays we see various sources who classify event processing as a kind of analytics,   Gartner, in a recent report, used the term "active analytics" in a report classifying event processing as analytics.    Jeff Wooton from Sybase said in the Sybase Blog that I am pondering whether event processing is analytics, and stated his positive answer to that,    After looking up the meaning of the word "pondering" (I still have some holes in my English vocabulary), I am not sure it is the right phrase to express my attitude.   I have drawn an observation that people did not use to classify event processing as analytics in the past, and started to do it in the last year or so.  Since I did not see a big shift in what event processing is doing at that time, then either there is a shift in how analytics is defined, or that the event processing marketing guys are riding on the analytics hype.  I guess that there is a truth in both.   The analytics guys are talking about event processing as part of analytics from their side.  
Paul Vincent noticed that there is a recent Communication of ACM article entitled "An overview of Business Intelligence technology".    Paul complains about the fact that the authors looks at event processing as equals to continuous queries, while in reality many of the models (and products) are base on other programming models such as rules (Paul has illustrated the distribution into programming models on a genealogy illustration of event processing languages).      Paul is right in the sense that people from the database research community view all the universe as centered around their terminology, and are sometimes ignorant about other approaches.  What makes it more interest is that one of the CACM article authors is Umesh Dayal (currently in HP Labs),  Umesh  (which is one of  my early professional inspirations) is considered to be the father of "active databases", and the HiPaC project in CCA that he managed has coined the Event-Condition-Action architecture, and in a conversation I had with him a few years ago he thought that the ECA architecture had a lot of traction;  Seems that Umesh did not keep track on the descendants of HiPac!    
Anyway -- besides the complains,  this is an indication that the BI guys adopted event processing to be part of BI from their side,  which is somewhat extending the definition of what BI is.   Interesting!





Tuesday, July 19, 2011

Event processing as analytics

Recently, I hear more and more that people are classifying event processing as a kind of analytics. 
This is partially due to hype that exists around analytics, and partially due to taking the word analytics in more broader term  that denote general use of  computerized quantitative tools beyond the traditional use of statistical processing.   In sense it also reflects the fact that event processing is in many cases used as OEM inside sophisticated solutions, and not sold as a middleware  per se.     Is it the right classification?  --- there are pros and cons,  but linking to a hype seems to be a good marketing strategy especially towards people who don't know what it is.    From research point of view, it is certainly a distinct discipline, though there are synergies.  I'll write more about the differences and synergies in follow up postings.

Thursday, June 26, 2008

On Unicorn, Professor and Infant

What is common to - unicorn, professor and infant - and how they are related to event processing anyway? -- These where my associations while reading the amusing post of my dear friend Tim Bass when he writes "On elephants and analytics" in an amusing response to my blog posting from this morning .
I am glad that Tim likes my metaphor of the big elephant, since my visit in Thailand last year, I dream about elephants all the time -). Since I am in a metaphors mood today, I'll use some more.


It seems that both Tim and myself share the opinion that the domain of event processing is much larger than the applications that exist today. In fact, I have stated several times, including in my press briefing for the EPTS launch: I believe that the big challenges are still ahead of us, since the industry barely scratched the surface of the potential use


In my previous Blog I have used the metaphor of an infant to describe this state.








Tim is constantly writing about the hype around CEP, again, in the metaphor level, some people that claim that their infant is a professor, and this of course not really true.



Indeed, there are people who over-hype CEP (a perfectly normal phenomenon, there is a hype cycle for each technology, and CEP is getting closer to the top), no surprise here.
Where I respectfully disagree with Tim, is in his claim that what has been done until today is just hype and hence totally worthless, my experience shows otherwise, there are customers which use current products and get value from using these products, and for some strange reason, the number of such customers is even growing (as a comment - calling all of them "time-series stream data engines" is a gross generalization, there is a variety of products, and not all of them are based on time-series streams, on the other hand, those who are based on time-series streams also have value). This does not say that current technologies solve all potential applications, and satisfy all potential users, but treating all the existing technology as a unicorn,

saying that everything is a hype, and there is nothing concrete, like a mythological creature, seems to me as going to extreme. Which takes me back to the elephant that both Tim and myself like While I agree with Tim that those who describe their infant as a professor may be looking at a toy elephant, the other extreme of treating an infant as a unicorn seems also to miss some part of the elephant.


Getting back to issue that start the case analytics and EP -- in my mind - to say that all EP applications require analytic tools, is like saying that one cannot get value from relational databases without using analytic tool, this assertion is certainly true for some types of application, and certainly wrong for others --- and it seems that the same hold for event processing applications. I'll let every reader to quantify the ratio based on his/her experience.

And last comment: it seems to me that there are more than one person in this community who meet customers and care about the value to customer; nobody has a monopoly on understanding all types of customers, applications and business values, and surprise - some vendors actually
believe that bringing value to custotmers is a better way to do business than selling balloons.

On EP and Analytics



Recently, some of my fellow-bloggers have occupied themselves with the question whether analytics are integral necessary part from CEP, and without it CEP does not really exist. My good friend Tim Bass went further in his current Blog and called the current state of the practice in CEP as Snake Oil , well - here I return to my previous posting with the metaphor of a group of blind people touching an elephant and each of them touches a different side of an elephant and each is confident that he knows what an elephant is, and furthermore he is the only one who knows what an elephant is. One of the benefits (there also some shortcomings...) of working in a large company is the exposure to many types of applications that are very far apart, and yet, all of them can share the same infrastructure (with some variations), talking specifically about the issue of EP and analytics - there are some applications that you cannot really think of without using analytics like "fraud detection" - where we need always to look at patterns that were unknown before, and have been used when our software has blocked the previous set of gaps, thus if the blind person touches the right back leg of the elephant in the "fraud detection" side, he sees analytics as a must. However - likewise there are plenty of other application - probably on other legs, back or trunk that do not require analytics - some simple examples: A physician sets individual alerts based on combination of test results/monitors reading - when to alert the physician or nurse; exception handling in manufacturing process - where the exceptions types are well-defined (and translated again to some combination of events); monitoring security regulation of different persons that need to be accompanied to various places within a plant according to the tag type, and checking if an autorized escorting person exists within the required distance, which monitors regulation and does not require any analytics, this is a small sample of CEP applications I have noticed recently.


Personally I think that analytics are very important, and we'll see more and more applications in which they are required, like the fraud detection one I've already mentioned, smart auditing etc... As stated before, I think that the combination of EP and some intelligent techniques (like: machine learning, prediction, handling uncertain information), which I call "Intelligent Event Processing" is very important for going forward, and we intend to reach to the AI community by doing a first Intelligent Event Processing conference as one of the AAAI spring symposia - stay tunde to note on that,

Having said that, I cannot say that this is the majority of applications, currently most CEP applications do not require analytics, here I join the opinion of Hans Glide on this issue.



This assertion returns me to the picture on the top of this blog - showing the half-full glass syndrom. Some people look at the empty half glass and some look at the full half glass. First - let's take the optimistic "full half" approach -- indications show that there are CEP products that serve as platform to build applications that bring real value to real customers, these applications are in a wide variery of areas, in variety of products, some of the type of "time series" streams, and some are not (next week I'll give a tutorial in DEBS 2008 on patterns, so this will be an opportunity to blog about types of patterns). The fact that they exist, is an indication that the EP indusrty is not just promoting "snake oil"
It is more interesting, as a scientist, to look at the empty half glass - which I think is even more than half. I think that the EP discipline is in its infancny, it already walks some steps, can say a few words, but we need to invest more until it will run, dance, sing and write Blogs... However, as every parent can testify, huge progress happens since from being a newborn until getting to the state I've described, so I don't underestimate the work done so far, and the traction achieved in the market, and think the entire community believe that we just scratched the surface of the potential. More - later.