On DEBS 2013: First keynote speaker - Roger Barga from Microsoft

Roger Barga has been an excellent choice (of the organizers) for first keynote in DEBS 2013.

He has a wide perspective from the multiple roles he occupied over the last few years in MSFT.   

Roger mentioned Thomas Kuhn's seminal work on the structure of revolutions - saying that there are two competing forces, those who push for paradigm shift and those who resist them and try to find "good enough" way to resolve everything in the old paradigm where they feel comfortable.   The use of event-driven thinking is just a paradigm shift, this is consistent with our tutorial given in the previous day.  

Some comments I took while listening to Roger:

1. Evolution of analytics:  analytics 1.0 - descriptive analytics based on warehouses, analytics 2.0 - big data, with NOSQL and Hadoop, we are going to analytics 3.0  -"rapid analytics with business impact".    Hadoop will become a niche technology, and real-time analytics (based on event processing) will take over. 

2. Some examples: Rols Royce is giving "engine up-time" as a service instead of selling aircraft engines, with a lot of instrumentation for maintaining the engines.  Other examples in the area of telemetry.  Some companies are making huge investments on real-time analytics.

3. Today there are two types of analytics: operational analytics  based on the speed of business, but use little information to get decisions;  investigative analytics based on the speed of  data scientists, and is based on a lot of information. There is a gap between the two that need to be unified.

4.  People don't know how to make use of new technologies to find new useful applications and tend to apply new technologies to old applications -- example: the first TV area was just visual radio, until the industry learned that TV opens new opportunity.

5. Bottom line: there are velocity pressure to do real-time analytics, but it requires paradigm shift and education.  Very compatible with our conclusions.   More about DEBS 2013 - later.

On event processing as a paradigm shift

The readers are probably familiar with this picture where it shifts between seeing two faces facing each other (in black) and a white vase. I came across a (relatively) new blogger in this area, Pern Walker, blogging for Oracle's "event driven architecture". The title of the posting is:

Event servers, a disruptive technology. It describes the components of the (former) BEA framework, nothing new here, but the interesting part is the conclusion - event processing COTS is a disruptive technology, it displaces custom code in event processing, since it is more cost-effective.

This reminds me of a discussion we had in May 2007 in the Dagstuhl seminar on event processing, it was a night discussion with wine, and was lead by Roy Schulte, the question that Roy has posed to the participants : "Will Event Processing (EDA) become a paradigm shift in the next few years or not?”.

Today, I don't intend to answer this question, instead I'll post part of the discussion in Dagstuhl that included observations about "paradigm shifts" (thanks to my colleague, Peter Niblett, who documented the entire Dagstuhl seminar). I'll return to this topic again, with my (and maybe other) opinions about the answer, after the EPTS event processing symposium

Observations (from the Dagstuhl discussion):

• Paradigm shifts can’t happen if there are too many barriers; have the entry barriers for "event processing" already been removed? ;
• Paradigm shifts are more likely to happen when adopters decide they need a whole new avenue of applications; they are less likely to happen as a way of re-engineering existing systems. For example the German population will reach 1:2 old: young ratio by 2020 so this requires a paradigm shift of healthcare models. Can we identify new avenues of relevant applications?
• Paradigm shifts usually happen as a result of some external change, not just because of innate strengths of the technology itself. Can we identify such external changes?
• Standardization is not necessary for a paradigm shift, but good, appropriate standards (de facto or otherwise) certainly help

Another question is to where in essence is the "paradigm shift" - is it the decoupled "event-driven" paradigm ? is it the "complex event processing", i.e. ability to find patterns on multiple events? is it the entire processing framework as the Oracle's Blog claim?

More - Later