On flood prediction from IBM Research

There have been some visible floods in recent years, like the one in Australia.    Prediction of the course of floods can be tricky, since rivers can have multiple splits.  Today IBM  announced that  IBM Research together with UT Austin a prediction model to predict the course of flooding.   This combines IBM analytics research with UT research in the physics of rivers.


Note that disaster management is an area where proactive computing hold significant potential, however in order to realizing this potential, strong prediction abilities are required.   

More on disruptive technologies

Hello from Brussels -- I'll spend the next couple of days in reviewing two EU projects, one here in Brussels and the other in Dortmund, Germany -  3.5 hours train ride from here.    


I have written about disruptive technologies before, one of my favorite topics in fact.  
Yesterday there were two events that reminded me of this, the first one was visiting Better Place, An Israeli company,  founded by Shai Agassi,  a person I met when he was still young, and got to be one of the senior leaders of SAP.   His vision is the vision of electric car, and the company, as seen in the slide above, is an infrastructure company which provides charging, automatic battery replacement and some other goodies.  They intend to start commercial use of their cars (currently they work with Renault as a manufacturer) at the end of the year,  this is indeed a disruptive technology over the current models of energy consumption in cars, and also of the leasing model of cars.     We'll see if this trend will catch.


The other event was an interview with Eitan Wertheimer,  chairman of the board of Iscar, and known Israeli industrialist, who said in that interview that whenever he launches a product, he kicks-off a team to establish a disruptive product that will make it obsolete.  This seems to go against the genetics of business -- since the goal of any business is to make money out of its products, and not try to defeat it.     His reasoning was that for every technology somebody will work on defeating it,  and it is better be us and not our competitors.  
I like this line of reasoning,  the technology world moves fast, and if one wants to be a leader, then it must lead   
new developments all the time.    


I always saw our mission in IBM Research to create disruptive technologies rather than incrementally advance existing ones, I think we are doing some of these, but not enough.   Our current project is certainly in that direction.   

Watson - a machine beats human in Jeopardy!

One of the amazing achievements of IBM Research is the recent triumph of the "Watson" program, whose avatar  is seen above in the Jeopardy! game, beating two human champions.  This raises back the question whether computer thinks, and whether we are getting closer to the "singularity" vision between human and machine intelligence.     While this game is a question answering one,  a complete machine intelligence will also require the ability to detect events and contexts, and react within a changing environment.  I suggested (as a joke) that the next game that a humanoid robot should strive to is the "survivor", which requires other capabilities, like social capabilities, but I think that this challenge is somewhat beyond even the current wildest dreams.   

DEBS 2010 program has been published

The program of DEBS 2010 is now published on the conference's website. Note that IBM has become a research power, based on its representation in DEBS.

In total IBMers have authored or co-authored 10 papers out of the 21 papers in the research and industry tracks, and 2 out of the 6 tutorials.

Some details:

Tutorials:

Ella Rabinovich and myself will deliver a tutorial prepared by several members of our team, on: context Aware Computing and its Utilization in Event-Based System

This is the third year in a raw that I am giving (alone or with others) a tutorial in DEBS.

Guy Sharon, another member of our team, participated in the preparation of a tutorial on Measuring Business Value of CEP-Applications.

Papers:

I am a co-author of the paper: Analyzing the Behavior of Event Processing Applications.

Another paper by members of our team is: Industry Experience with the IBM Active Middleware Technology (AMiT) Complex Event Processing Engine (I have not participated in writing this one), both of these papers are planned to be presented by Ella Rabinovich.

Three additional papers are co-authored by other persons in IBM Haifa Research Lab:

Magnet: Practical Subscription Clustering for Internet-Scale Publish/Subscribe

Quilt: A Patchwork of Multicast Regions

On Trade-offs in Event Delivery Systems

Four papers are authored by IBM Watson Research center's guys:

Evaluation of Streaming Aggregation on Parallel Hardware Architectures

Placement of Replicated Tasks for Distributed Stream Processing Systems

Experiences with Codifying Event Processing Function Patterns

Workload Characterization for Operator-Based Distributed Stream Processing Applications

And another paper co-authored by Udo Pletat from IBM Boeblingen

Distributed Heterogeneous Event Processing

This is an indication for the importance that IBM sees for investing in research and thought leadership in the areas of event processing, stream analytics, and pub/sub. More about DEBS -- in July; meanwhile another business trip to Europe is waiting for me in Sunday.

On internal recognition

Today it has been announced that the "event processing" activity done over the years in the IBM Haifa Research Lab has been recognized, internally in IBM, as "outstanding accomplishment" in the category of "emerging market".

IBM Research has an annual process, as one of the way it is measured in IBM, for recognizing accomplishment (from various ranks and classes), most of the accomplishments are either helping IBM products or scientific accomplishment; the "emerging market" accomplishment in the "outstanding" level, was given, first time in the history, as far as we know, which made it quite challenging. This is not a personal award, but a recognition for an activity.

We have started working on event processing (not using this name, we called it "active technologies") in 1998, and moved through a long way inside IBM. Getting a big company to get into a new area is not an easy task, Lou Gerstner's book Who Says Elephants Can't Dance? provides some glance on how decisions are being taken in big corporates; I can write a book of my own about the event processing case.

Impact decisions, when you are part of the research part of the corporation, and is sitting in the back part of a big vehicle, is even more challenging. It was not many years ago when I've heard a senior architect in IBM saying about event processing: "this is just a hallucination of research people".

The recognition we achieved today is a recognition that we were critical factor in IBM's decision to get into an emerging market (recognizing that it is an emerging market). Even when this was done, getting official recognition about it is often not trivial.

Is it a reason for celebration --- not really, recall that I have stated that my motto in life is represented by the poem IF by Kipling. Quoting Kipling:

If you can meet with Triumph and Disaster

And treat those two impostors just the same;

Typically I don't tend to become depressed by the many frustrating moments, and also don't tend to be conceited by the few encouraging moments, it is all just a game.

While event processing has gotten over the years from being considered as "hallucination of research people" to an emerging market recognized by multiple analysts and all big software vendors, it still has a lot of challenges, so the work is far from being completed... there are more disruptive technologies ahead to conquer.

Live from DEBS 2009 II - our paper presentation

Stratification is one of the terms that computer scientists borrowed from Geology. In 2007, Ayelet Biger, my former M.Sc. student has done her thesis about Complex Event Processing Scalability by Partition which looked at semantic partition of an EPN graph to strata, where in each stratum all agents are independent and can run safely in parallel. Today we have reported about 2008 research project that took the stratification idea and developed a system to assign agents to machines in a distributed environment. Geetika Lakshmanan delivered the talk about this project today. I have posted it on Slidshare. Enjoy !

This project is an interesting example for a life-cycle of a project:

• It started as an academic thesis;
• It has flown to a research project within IBM Haifa Research Lab ;
• After showing promising results in the lab, it evolved to a more "down to earth" project that deals with assignment of agents to machines and threads within the IBM product - WBE-XS (Websphere Business Events - Extreme Scale) which enable event processing on grid environment. This project has to take into account products and their implementation, deal with product instrumentation, and other stuff that pure research projects do not deal with.

While starting an idea in the academia and then going to a start-up has its magic, the work in IBM Research enables to get stuff from academic projects until impact through products, and using my hat as an adjunct professor in the Technion, this is possible. However -- as noted before, getting things pushed in a big company are not necessarily easy.