On DEBS 2013 grand challenge

The DEBS challenge was introduced first time in DEBS 2011, since then it also became "grand". 

This year it was around football (or soccer for the USA guys) and consisted mainly on aggregations, but also detection of proximity between player and ball, using sensors.   There was a lot of interest this year, and 14 submissions.  It becomes a "community building" theme.    6 solutions arrived to the final and were invited to present at the conference, a lot of effort has been dedicated, and we saw very nice and creative solutions.  The main criteria was performance that had to be measured on a given data-set.    In the business meeting I raised some ideas to make it even more interesting, by given "unseen" data at the conference itself, and introduce last minute change in the specification.     The conference is now at its last session.  Next year - DEBS 2014 is planned to be in IIT Bombay, India.

A proactive system: NASA grand challenge -- mitigating asteroid risks

NASA issued the next in its series of grand challenges, this time the ultimate killer (or saver) application for the idea of proactive systems -  the challenge consists of the sense part -- sensing all moving objects in space,  forecast part --- forecast the asteroid trajectories, determine if the trajectory might jeopardize earth, and if yes -- use robotic vehicles to alter the trajectory.   An ambitious project, which may be vital for the survival of earth.    This includes breakthroughs  in sensors and actuators and  also lightweight robots that can explore asteroids.  Note that there are 8800 known near earth asteroids.  See also the Wikipedia value about "Asteroid impact avoidance".  The relevant research community will have a lot of resources to work on these issues.

World Stream computing to replace the Web

I came across an interesting article by David Gelernter.  Gerlenter, a Yale Professor, who co-introduced the term "lifestreams" in the 199o-ies, proceeds in the same spirit and talks about "worldstreams" which he defines as a collection of heterogeneous real-time messaging stream  (may be thought as a collection of streams).   His prophecy is that this will be a disruptive technology for the web, browsers, search, operating systems.  We will no longer surf in sites and search the web,  but we'll browse the worldstream, and ask the worldstream to bring us what we want, in the time we want it, instead of search for it.    The event processing grand challenge, which was part of the event processing manifesto,  and talked about the "event fabric" was similar in nature, but the manifesto authors did not claim that the web will be replaced,  it will probably not happen in one step anyway.    However -- certainly interesting observation, and promises bright future...

On big events and big data

The "Big Data" phenomenon gains a lot of traction, interest, and related work in recent years.   The Internet and making everything in digital form has resulted in amounts of data beyond past imagination, and the rate of growth is amazing.   Mark Palmer in his Blog posting made the analog of data as sand, 

saying that  "If every grain of sand in the bucket was 1 byte of data, then:

• The entire work of Shakespeare fills just one bucket of sand (about 5MB)
• A fast financial market data feed (OPRA) fills a beach of sand in 24 hours (about 5TB) 
• Google processes all the sand in the world every week (about 100PB)
• We generate 60% more sand every year" 

Using this analogy - if all data in the world is a sand,  much of the sand is talking about facts, BTW - the fact that a fact appears as a data in the big data universe, does not say that this fact is in fact true.  

Events issue some of this data, but in many cases an event is the fact that a fact becomes true or false, and this fact is not really kept in the data.   

The "Dagstuhl grand challenge",  which is part of the event processing manifesto, is talking about an "event fabric", which will be the Internet equivalent of events instead of data, I guess that the quantities will be on the same cardinality, thus   it will have the same scalability challenge. The main difference is the type of processing -   event processing instead of queries/information retrieval.    Getting to an "event fabric" has indeed many challenges.  In DEBS 2011 there will be a tutorial about this grand challenge.   I'll write more about this challenge in the future. 

(and this is of course Schloss Dagstuhl) 

Briefing a USA federal committee

In this picture you can see me and a tiger during my family vacation in Thailand, in 2007.
Today I met another TIGER - the standing committee for Technology Insight-Gauge, Evaluate & Review which is a committee of the USA National Academics that works on behalf of the USA intelligence community. 

I am still unsure how they picked me up, but they invited me to participate in a meeting with the committee members and some other participants through video conference and brief them on the state of the practice, my vision for event processing in 2020, and the challenges on the way to get there.  This is an indication that event processing has caught the attention of  defense sector as well.    The vision part talked about the Event Fabric devised as the grand challenge in the Dagstuhl event processing seminar,  published as chapter 5 in the event processing manifesto (BTW - in DEBS 2011 there will be a tutorial about this grand challenge). 
Another topic was of course, the proactive world, my favorite topic.  


I was asked what is the killer application that will drive the 2020 vision, and said there is no single killer application, there are multiple of them, and I see many of them outside the corporate IT domain, such as autonomic robots and computing embedded in biological systems.   Of course, defense applications will also be part of the driving forces.    More on this topics -- later

Watson - a machine beats human in Jeopardy!

One of the amazing achievements of IBM Research is the recent triumph of the "Watson" program, whose avatar  is seen above in the Jeopardy! game, beating two human champions.  This raises back the question whether computer thinks, and whether we are getting closer to the "singularity" vision between human and machine intelligence.     While this game is a question answering one,  a complete machine intelligence will also require the ability to detect events and contexts, and react within a changing environment.  I suggested (as a joke) that the next game that a humanoid robot should strive to is the "survivor", which requires other capabilities, like social capabilities, but I think that this challenge is somewhat beyond even the current wildest dreams.   

DEBS 2010 -- first day of the main conference

The first day of the main conference yesterday ended in an organ recital in the King's College Chapel, seen in the picture, magic place - people keep talking about being in Hogwarts. The conference itself started with the first keynote by Mike Franklin who shared his experience both from his work in Berkeley and in his start-up Truviso. Mike's main motivation is to extend database technology to react in a continuous way in addition to the traditional batch way of analytics, and view batch as a special case of the continuous. The talk was a kind of "lite sell pitch", but had some good points, like the observation that a start-up can take a known technology and try to use it for new types of applications (which is what Turviso does), or devise a new technology and tries to attack existing applications better, but it is very difficult to do both at the same time (new technology and new applications). I think that the event processing area indeed tries to do both.

There were some other talks, mostly by graduate students. One of the interesting observations (not a new observation to me), that there is a lot of energy in this community to re-invent wheels, in different variations. I think that one of the good results in the database community of the relational model was that a large part of the research community took it as a basis and constructed the research on top of it, and people did not try to re-invent databases from scratch for every thesis. In event processing we are still not there, and IMHO the area will have more substantial results, if the research efforts will be more focused on advancing the state-of-the-art instead of re-inventing most and advancing a little bit (in the best case). Today we'll have follow-up meeting to the Dagstuhl seminar, and the research grand challenge we are discussing is aimed to that matter.

In the late afternoon there were a fast abstract session, where I gave a short (8 minutes, 4 slides) talk on the interactions between business rules and event processing (I'll write about this subject some other time), and then a session of demos and posters, with a couple of follow-ups for me.

More -later. The second day of the main conference is about to start.

The EU FET Flagship program

Back at home. In the previous two days I have attended an EU conference in Brussels on FET (Future Emerging Technologies) Flagship. The EU is recently acting to diversify its R&D investments, and launch some new programs in addition to the traditional programs that will keep running. One of this new ways is PPP (Public/Private Partnership) that is aimed at short term more engineering oriented projects, while the FET flagship goes to the opposite direction: launch two very ambitious large 10-years projects, with around 1B Euro investment in each.

They have brought some examples of projects in that scale such as: Large Hardon Collider, The Human Genome project and some others, with varying degrees of success. The rule of the game is that 5-6 ideas will be given a year (and a funding of around 1.5M Euro each) to develop the program (including partnership around the world, funding sources -- the EU will fund only part, expect that national funding agencies will support it as well as others), and out of them 2 will be selected.

In the last couple of days I have heard a lot of talks about potential proposals, many of them will probably merge, others will disappear. The commission people are encouraging people to merge and share, they don't think it is reasonable to have more than 10-15 proposals. One of the important tests for the proposals is the grandmother test: If you can explain your grandmother in one sentence what it does in terms that your grandmother will understand, then you'll be able to sell the idea to politicians. IMHO most of the presentations I have heard fail to satisfy this property, especially all the ones which has technology in their center. The promising directions relate to Biology -- human brain, breakthrough in robotics, and socio-technical systems.

What is the relationship to the "event processing grand challenge"? --- this may be one of the vehicles to advance the grand challenges, however, since the two flagship projects will start in 2013 -- we need to find other avenues to start earlier.

Event processing is relevant to everything, the human brain is very much event-driven, robots are in many cases event-driven, social evolution is event-driven and more.

One of the challenges is to sell the contribution of event processing to these projects, in discussing with one of them, they thought that they indeed need to process events, but there is no research here, just hard dirty work of programing. I told them that our research issue is how to eliminate the hard dirty work of programming in conventional languages and provide for them something much more high-level with "easy clean work" instead. They have not thought of it this way, since we have not reached out to these communities yet, I gave the analogy that 40 years ago everybody has created home-made database, until the relational database changed the picture, this is exactly what we are doing, raising the level of abstraction.

Next --- see where this can fit and how it can intersect with the EP grand challenge, and determine how to position it (in my talk on live ecology I have equated event processing to the nervous system of a human body). My presentation on live ecology is now on slideshare.

More - later.

The event processing grand challenge - live ecology

Still travelling in Europe, I have spent one day in Luxembourg, serving as a reviewer in the Pronto project that deals with "event recognition for resource management", the highlight was showing us a nice demo of a van that was covered with sensors that track noise, density and location of the car, and can provide diagnostics about the comfort of passengers (too dense, too noisy) and the quality of driving (unsafe driving, sharp turns). I also discovered that one can miss train connections and not only flight connections, and arrived to the hotel in Luxembourg very late at night. From Luxembourg I continued in another 3 hours train ride to Brussels, from which I am writing now.

In Brussels I am attending an EU conference about the FET Flagship program. The European Commission is trying to diversify its programs in order to achieve more impact, and invest more in high risk - high impact long term research; there are several big consortia already established, and I will try to see whether the "event processing grand challenge" ideas, can fit into one or more of these, I am planned to deliver a short presentation today on "live ecology", taking the theme of making Asimov, my favorite writer, come to live with some of his amazing ideas, starting from Gaia ("The Foundation Edge"), a galaxy behaving like a single organism, and moving through the fantastic voyage, the blend between human and robot, and predicting the society's course (Psychohistory), and then hopefully return home later tonight.

The event processing grand challenge - take one

In the Dagstuhl seminar on event processing that ended 10 days ago we launched the "event processing grand challenge activity". You may be familiar with the DARPA grand challenge of driving driver-less car through the desert, you can see the two winner cars in the pictures above. We are not DARPA, and don't have money to distribute, but we would like to take the opposite direction, first define the grand challenge, and then convince funding authorities that they want to support it.

Why do we need this grand challenge? The research community has incubated the "event processing" area as we know today --- some research projects in the 1990-ies, such as: David Luckham's Rapide in Stanford, Mani Chandy's Infospheres in Caltech, John Bates' Apama in Cambridge, and our own Amit project in IBM Haifa Research Lab, followed by the various stream processing projects, like Jennifer Widom's stream project in Stanford, and later the Aurora project, to name a few (there are many more, of course).

The state of the practice is now in the hands of the software vendors, and event processing is becoming part of the main-stream of enterprise computing. However, software vendors are by nature advancing technology in an incremental fashion. On the other hand, there is a strong feeling that "event processing" has barely scratched the surface of its potential to impact society, this goes beyond current applications, and even beyond enterprise computing as we know it.

It is now the role of the research community to jump-start and incubate the step-function required to be achieved in order to get this kind of impact. We would like to have a call to the research community in event processing to focus on such a grand challenge, and as an incentive (and enabler) to get funding agencies worldwide to adopt it.

A substantial brain power have been invested in Dagstuhl, and in a follow-up to work on it, there have also been some people outside the community that deal with socio-technical systems and bio-informatics.

As a metaphor we can view many type of IT systems, social systems and biological system as a "live ecology", similar to a single organism with a lot of brains, eyes, ears, hands and feet.
Event processing serves as the nervous system, and events are what flow between the different players. This subsumes the "Internet of things" vision, where many sensors are connected, and also robots of various kinds as actuators, serving as hands. Such an infrastructure will enable changing the life as we know them.

We are working now on various scenarios that will be enabled by such an infrastructure; you can hear some thoughts about it in my talk next week in the OMG Event Processing community of practice that holds its first event processing symposium. Stay tuned for much more on that topic.

On event processing grand challenges

This is the logo of the "Grand Challenges SIG" of the UK Electronic design knowledge transfer network. In the 4th Event Processing Symposium last year, Arkady Godin from MITRE, proposed in the business meeting to launch a work group that will deal with grand challenges in event processing. The EPTS steering committee decided that such a workgroup may be premature, and it is better to complete some work in the other workgroups -- languages, architecture, use cases, interoprability, in order to have a better understanding of the grand challenges, however, this issue has neither forgotten nor forsaken. We determined that the best forum to discuss the grand challenges is the (second) Dagstuhl seminar on event processing, which I'll co-organized with Rainer von Ammon and Mani Chandy, we also asked to invite all people that we saw as the right set of people to participate in such an event (if you have not been invited and think you would like to contribute, please let me know, there is a waiting list for available slots if any, no promises though).

In the coming 5th event processing symposium in Trento, we shall hold the first brainstorming about grand challenges. Pending technical feasibility (still requires confirmation from local organizers) we plan to do this as a public session that all EPTS members will be able to use through audio conference call, EPTS members will get full details with call-in numbers when it will be finalized.

Grand challenges can be in multiple areas and also refer to multiple addressees. There are grand challenges that will require a community effort, and these challenges should be picked up by EPTS, some challenges are to advance the state-of-the-art, and this will be addressed to the research community, with some creative incentives. Some will be referred to the product vendors, and may be some to other adjacent communities.

Idea for such grand challenges are solicited from various sources:

• EPTS workgroups that already are active, each workgroup will be asked to contribute a challenge in its area
• The reach-out sessions for other communities in the symposium (the BPM community and the the IT management community) might also lead to some challenges
• The research community is always a source for such challenges
• Last but not least -- all EPTS members (and other interested people that wish to contribute, and thus are hopefully future EPTS members), specifically customers who has long-term vision about their systems, like our MITRE colleagues who started this discussion, and our analysts colleagues who have cross-vendor and cross-customer perspective.
  
  Anybody who wishes to contribute ideas to this session, please let me know by the end of this week.

I'll wrap-up this posting, by pointing out a Blog entry I have seen just now, written by Niels, who runs a consultancy company called "SQL Develop" as its name testifies deals with DB related issues. The upcoming Microsoft product "StreamInsight" has raised the interest in the database community, as Microsoft seems to take an SQL oriented approach to its product, and Niels is blogging about "CEP resources" he covers Blogs in this area, on my Blog he writes: THE blog about CEP, if you were to read only one blog about CEP, this is it! First, thanks to Niels for the endorsement, I don't view the Blogging area as a competition, as people who read Blog tend to read multiple Blogs, and it is also kind of a network of Blogs that sometimes react to each other. To be fair, also most Blogs in this area are marketing oriented Blogs, that they have somewhat different motivation from my Blog, and one cannot compare. Anyway, a good way to start the day.