On cooperative writing, crowdsourcing and event processing

Earlier this week another Amazon shipment has arrived, now I need a month vacation to read all of that, and also thinking about where to put additional book closet or two, since may house is looking like a book warehouse, with piles of books everywhere.  One of the books that I have started reading is the book whose cover you see above, written by Lisa Lutz (one of my recent favorite writer) and David Hayward, and the cooperation is that Lisa is writing the odd chapters and David is writing the even chapters.  Since it is a kind of crime book, the plot  is developing according to two different mind, the book also includes the exchange of messages that the authors send to one another.   Events have been a long time part of cooperative work.  In computerized cooperative systems, events is means of communication, as well as a means to get to cooperative decisions (such as: cooperative transaction commitments protocols). Nowadays, cooperative systems are not just among computerized systems, but among people.   One of the consequences of  the Web 2.0  is the crowdsourcing.  Currently event processing is not mentioned as a key technology in crowdsourcing.  However, I can see the connection - first in the dissemination:  bringing the information to those who can do the next step.  Also, since different parts of the task are done by different people,  there is a need to see a combination of activities and notify about consequences.  As an example:  a vehicle design is done by different people, some design the mechanic components, and some the electronic components.  Change in one of them that impacts the other can be done using analysis of the various design events.    More on this topic - later

On outside-in vs. inside-out innovation

Today I participated in presenting to some young people from the Israeli Defense Forces, that are going around in research institutes in academia and industry to understand what is innovation, and what is research.   Recently, there has been an article in one of the Israeli newspapers that talked about inside-out vs.outside-in innovation, or where are great ideas coming from?    Many companies are trying to do outside-in innovation, doing market surveys to identify gaps and requirements in order to determine what next to do;  this article claimed that some of the bigger innovations (e.g. Facebook, iPhone and more) are inside-out, somebody comes with great idea out of the blue,  and it is sometimes being used for purposed that the inventor did not imagine.   The outside-in is used to refine, and shape the next generations.   I believe that this observation is true,  one cannot obtain great ideas from customers' requirements, as they typically don't think that they need it until it is there.   

On big events and big data

The "Big Data" phenomenon gains a lot of traction, interest, and related work in recent years.   The Internet and making everything in digital form has resulted in amounts of data beyond past imagination, and the rate of growth is amazing.   Mark Palmer in his Blog posting made the analog of data as sand, 

saying that  "If every grain of sand in the bucket was 1 byte of data, then:

• The entire work of Shakespeare fills just one bucket of sand (about 5MB)
• A fast financial market data feed (OPRA) fills a beach of sand in 24 hours (about 5TB) 
• Google processes all the sand in the world every week (about 100PB)
• We generate 60% more sand every year" 

Using this analogy - if all data in the world is a sand,  much of the sand is talking about facts, BTW - the fact that a fact appears as a data in the big data universe, does not say that this fact is in fact true.  

Events issue some of this data, but in many cases an event is the fact that a fact becomes true or false, and this fact is not really kept in the data.   

The "Dagstuhl grand challenge",  which is part of the event processing manifesto, is talking about an "event fabric", which will be the Internet equivalent of events instead of data, I guess that the quantities will be on the same cardinality, thus   it will have the same scalability challenge. The main difference is the type of processing -   event processing instead of queries/information retrieval.    Getting to an "event fabric" has indeed many challenges.  In DEBS 2011 there will be a tutorial about this grand challenge.   I'll write more about this challenge in the future. 

(and this is of course Schloss Dagstuhl) 

Watson meets DEBS 2011

DEBS 2011 will take place in the Yorktown Heights auditorium, the same site in which the famous recording to the Jeopardy! programs in which the computer has won against two human champions.  The Watson hardware resides very close to this auditorium.  With some effort we succeeded to bring Eddie Epstein, one of the senior persons behind Watson has agreed to be an invited speaker in DEBS 2011,  Eddie will also provide a demo of the system.    


The full program of DEBS 2011 will be advertised around this weekend, the list of accepted papers can be found on this Blog.   


It should be note that if you plan to be in DEBS -- plan to stay for the entire conference, besides the 33 presentations, the attractions will be spread among the days  - no day can be skipped!


Monday, July 11 will be the  tutorial  day, and in the late afternoon there will be the EPTS award granting event. 
Tuesday, July 12 will have the two industrial keynote speakers: Chris Bird and Don Ferguson (see here for more about the keynote speakers),  and in the later afternoon a reception, and after getting some alcohol, the "gong show",  in which the participants can express outrageous ideas about future features and utilization of event processing technology.   
Wednesday, July 13 will have the keynote talk of Johnnes Gehrke,  the demo and poster session including the grand challenge,  and the conference banquet with artistic performance, and the DEBS awards (best paper, best idea in the gong show, best demo and maybe more). 
Thursday, July 14  will have the keynote talk of Calton Pu, and the invited talk of Eddie Epstein on Watson.


None of these day should be missed -- plan accordingly!


The fifth day - July 15 will be the PhD workshop adjacent to the conference.    


See you all in Yorktown.  

1000 contacts in LinkedIn

Today, two more connections were added to my LinkedIn contacts list, and the number of contacts went up from 998 to 1000,  the 1000th contact is my IBM Haifa Research Lab colleague, Ran Ettinger.    LinkedIn is the social network in which I have most contacts, I also have twitter account followed by 236, and Facebook account with 96 friends, there are also some more social networks I am registered in somehow, but I am not active in any of them.


Among the 1000 members of the contact list 331 are (or have been at some point) IBM employees,   the two other high-tech companies with most contacts are: Google (16) and Microsoft (10),    There are also some contacts from my second universe, the academic world:  Technion (29) followed by Tel-Aviv University (8).
Some are friends and classmates from high-school,    Out of the 1000,  423 live in Israel,  131 in the Greater NY area,  69 from the SFO Bay area, and 46 from the UK, and then from many other countries in all continents.    I am also registered in about 40 LinkedIn groups -- from Temple University alumni, to the Haifa high-tech community,  and of course all event processing related groups, and many others.   


I have never asked anybody to recommend me on LinkedIn,  one person did in response to the fact that I've recommended him.  I have recommended six persons at their request, but never knew whether it is effective.


That's enough statistics for today.  

More on black swan events

Today I had a procedure of tooth extraction and bone transplant.  so I am semi-functioning with stitches in my mouth for the next week.    The reasons for the fuel pollution that paralyzed all flights from Israel yesterday is still in investigation, so the event was not classified yet to black or white swan.  If it will be classified to a black swan event this will be a second instance of the same event within 5 years, so it is interesting if a second instance qualifies as a black swan.  

Some comments about my previous posting claimed that a black swan event cannot be predicted by definition, however event processing can act to decrease the damage and mitigate its outcomes,  maybe the consequences of the black swan events can be predicted and some action need to be taken.  I guess that one of the interesting applications of proactive computing is to deal with consequences of black swan events, and then monitor white swan events to track the progress towards or away from undesired consequences.    More on this topic - later.

Black swan event?: Fuel pollution in Ben-Gurion Airport

Israel is a one-airport country  (there are several small ones for internal flights, but with the exception of some charter flights, all international flights go through Ben-Gurion Airport, near Tel-Aviv, given this fact, anytime that this airport does not function (e.g.due to strike) it becomes a major news item.  Today all aircrafts were grounded due to the fact that the fuel that were inserted to the aircrafts was polluted, due to a pollution that has been discovered in the fuel facility in the airport, many flights were cancelled, and other delayed were filled with a small amount of fuel that will enable them to fly to one of the nearest airport (mostly in Cyprus, some in Jordan) and refuel, the picture above taken from the YNET news site show passengers that are stranded in the airport,     Is such event a black swan event ?   seems so, but surprisingly this is the second instance of this event, such event happened before at the same airport a few years ago, and took a day to recover.     


From event processing point of view the interesting questions about such events are:

• Can black swan events be predicted?  first time that they happen?  second time that they happen? - this actually similar to the question - can we instantiate Merphy's laws -- infer everything that can go wrong? 
• How can rules and actions about  treating black swan events can be set, if we cannot even imagine the event's nature? 

The earthquake in Japan made thinking about black swan events more popular,  the challenge is to advance the thinking in that area.