Top ten reasons why large companies fail to keep their best talent

I have written earlier this year about the Plato vs. Aristotle conflict - in the organizational aspect, putting the organization in the middle, and the employee serves the organization goals vs. putting the employee in the middle, and having the organization as a platform to achieve the employee's goals.   When talking about top talents, they tend to be in the Aristotle side of the spectrum.  A recent article by Forbes discusses ten reasons why large companies are not good at retaining their top talents.  I still work at the big blue (IBM Haifa Research Lab), so I probably not qualified as top talent, but I heard all of these arguments before from people who left big companies.  The ten reasons relate to large companies' red tape; failing to find a project that matches the top talent's passion; annual reviews and career developments issues; lack of patience from the company's part to the top talents' initiative which can be longer term; lack of other top talents around, and mediocre management that don't know how to manage top talent -- very interesting! 

New tutorial on event processing

Roland Stuhmher, whom you can see in the right-hand side of the picture, on the Jeopardy! set, in DEBS'11, has recently recorded a video piece giving a tutorial on event processing.  Some of the slides looks familiar to me (well, he mentioned my name is his slide about "attribution of the slides").   In the slides he mentions a term iCEP, which either means that Apple has a new CEP gadget in its i series.  BTW - according to Steve Jobs, the apple "i" (started with iMAC) stands for: Internet, Individual, Instruct, Inform, Inspire -- all start with I...  So does any of them apply to iCEP?  Another possibility is, of course, intelligent (I have used the term IEP in the past). Anyway -  good tutorial, with some glance of the ETALIS project developed in FZI.  Enjoy!

On event processing for real-time and non real-time applications

Chris Carlson has written in the Informatica Blog about the fact that there is a growing segment of applications that are using event processing and are not really real-time, and stating the fact that their share in the event processing market is growing.   I have written four years ago in this Blog about the term real-time and the abuse in its daily usage.  When some people (and marketing messages of vendors) are talking about real-time, they mean "very fast", while real-time really means "within time constraints", the time constraint can be micro-second, second, 5 minutes, or 2 hours.    Indeed, the early adopters in event processing, trading applications in capital markets, are based on low latency and fast reaction.   Many other types of applications use event processing for the functionality of filtering-transformation-pattern matching (or continuous queries in the stream oriented programming style), and the non-functional aspects are secondary.   The area which I am working on these days , proactive computing, has some applications in which there are real-time constraints, but typically not in magnitude of micro-second, but in seconds to minutes.  This is the case where there is a forecast for a future problem (e.g. a traffic jam will occur in 5 minutes), there is a time constraint on activating an action (e.g. within 30 second there is a need to change the traffic lights policies to mitigate the traffic jam).  This is a real-time application, but it has to react within 30 second, to impact in 5 minutes.    The interesting thing is that low latency applications may be "best effort" and not have real-time constraints.  Thus - there are low latency applications, real-time applications, those who have both, and those who have none.  Interestingly, event processing applications can be found in all four groups.  

five forecasts from IBM - innovations that alter the tech landscape within five years

The annual IBM predictions on "five in five" has been published.  You can read more about it in the IBM Smarter Planet Blog.  


 This year’s predictions are:

·    People power will come to life 
·    You will never need a password again
·    Mind reading is no longer science fiction 
·    The digital divide will cease to exist 
·    Junk mail will become priority mail

Some EP related news

Today I had some time to browse through some new staff on the web, related to event processing.  Here are some of the things I've found:

I noticed that Sidhhi has posted a presentation on Slideshare.   I have written before about this new event processing open source from Sri Lanka.   While comparison to Esper, as an open source is fine,  the presentation makes the impression that the state-of-the-art today is Esper and some academic prototypes.   There are of course many other things in the market -- my previous posting makes a reference to Paul Vincent's list.

Fujistsu announced that it has developed complex event processing technology designed for use with cloud technology that employs distributed and parallel processing. The picture about is taken from their announcement.  A new player? 

A new course to teach Microsoft StreamInsight is now available on the web.  The course is described in Richard Seroter's Blog, with a link to the actual course.  The course lasts for 4.5 hours,  I have looked at the beginning, seems to be a good one. 

More updates - later. 

The genealogy of event processing players - December 2011 edition

                                 (Click to enlarge)

Paul Vincent, who took upon himself to be the record keeper,  posted a revised version of his famous market players with some additions. Over the last year I have encountered several more in this area, so if you are a player and want to be on the map,  notify Paul and get into the picture.

Cloud Computing Journal on Big data meets Complex Event Processing

Cloud Computing Journal published some citations from a podcast in which the analyst Dana Gardner, interviews Mahesh Kumar from AccelOps, about correlating streaming events with transient data to get real-time analysis of data in the context of big data, and cloud implementation.  The applications that AccelOps targets are mostly availability and performance management and security.   The idea is not new,it seems that there are various approaches and realization that data needs to be analyzed in real-time in order to get decisions, before being written to a disk.   

Reactive and proactive as relative terms

This is one of the most famous visual paradoxes, in this picture one can see a young girl or and old woman, some people see only one of them, with some concentration one see both.   This is a kind of relative view, in event processing there are some relative terms, I have written a while ago about the fact that the terms raw and derived events are relative,  a derived event in one sub-system can be raw event of another sub-system.. 
There are other cases of relative views (an entity may be both consumer and producer, for instance).  


I have reminded on relativism while reading an article about the keynote talk of Jeff Shulman (Gartner's manager of the application integration and web services analysts team).  Shulman is talking about cloud, mobile and CEP as the leading trends for application development and integration.   
As a remark to Shulman's keynote, the article bring an interesting response of  Chris Dressler, VP Technology in Cablevision,  he sees a  CEP can be used to find and correct issues before the end user has the need to make a complaining call.
This is an interesting example,  from the service provider's point of view (cable TV company in this case), this is a reactive application, tracking events that already happened and react to them, from the home consumer point of view this might be proactive, since the consumer may not yet felt the impact of the problem, so from the consumer point of view, this is elimination of problem that has not really happened.    More on this distinction - later. 

Book review: Event processing for business by David Luckham

I am holding in my hand a copy of the brand new book by David Luckham, entitled: EVENT PROCESSING FOR BUSINESS. His first book "Power of Events" is the first book that opened the current era of event processing, which made which made David Luckham the prophet and elder statesman of the event processing community. My first meeting with David in early 2004 inspired me to think about the future of this area, and gave me some label and framework for what I was doing at that time.  Some of David's ideas like event processing networks and event patterns found themselves as part of the area foundations, not exactly in the form that was defined in the "Power of Events" though.


The new book is aimed at being non-technical book, aimed at people in business and IT departments that want to understand what is event processing, and what are its uses.  It is serves a similar target  to the book by Chandy and Schulte. In comparison our EPIA book is aimed at more technical audience that would like to understand the building blocks of constructing event processing applications. 


The book starts with chapter 1, which has the ambitious title "event processing and the survival of the modern enterprise" - explaining what most event processing is - and gives six principles of how it should be used by enterprises.  Then it moves in chapter 2 to a history lesson - surveying all ancestors of event processing  simulation, networks, active data bases and more, getting to event driven architectures.
Chapter 3 surveys the concepts that Luckham used in his first book, with definitions and some modifications to the original concepts.   Chapter 4 is back to a history lesson - this time from the point of view of the commercial world.  Here Luckham repeats his evolution classification that he has talked about in the past: simple event processing, creeping CEP, CEP as recognized technology and unseen CEP.  According to Luckham we have just moved recently to the third phase (CEP became a recognized IT).  The fourth and last phase is unseen (CEP goes behind the scene since it is ubiquitous and exists everywhere), it also becomes holistic, and in fact part of the infrastructure of every system from household automation to national cyber security.   Chapter 5 views the markets - existing and emerging - and talking about industries and applications, with 13 examples (seems that the author is not superstitious!).  Chapter 6 explain the notion of event patterns, here is goes more to technical, but stays mainly at the example level, neither trying to define pattern language nor talks about the implementation of patterns in current languages, Chapters 7 and 8 are entitled: "making sense of chaos in real time: part 1 and 2", with some examples and methodological insights.  Chapter 9 is the last chapter entitled "the future of event processing" talking about the phases of evolution to the next phase, and some futuristic applications like:solving gridlock in the metropolis.   The EPTS glossary is reproduced as appendix.


Overall -- good source of material and insights about event processing especially for the non-technical reader and good summary of the various talks that David has presented in the last decade.  A must read for anybody interested in event processing. 

Orson Scott Card's empire

I have been sick and stayed home for a few days,  which gave me an opportunity to disconnect from the professional work and read "empire" and its sequel "hidden empire", both by Orson Scott Card, one of the my favorite writers, whose books are always thought provoking.  The story is around the 2nd USA civil war, between right and left,  where an history professor from Stanford, who dreams about American empire, following the step of the Roman empire, and concludes that like the Roman republic, the American democracy should be eliminated to achieve the empire, thus he orchestrates chaos, civil war, and some other actions, and takes over as an agreed upon president that both democrats and republicans nominate to restore order, later in the second book he takes advantage of a crisis to reshape some of the world and take another step in the empire vision.    There are some questions behind the plot - whether the goal justified the means, since many of the book's hero characters, lose their life as a result of some of the actions,   whether democracy is indeed a value (and if it is - whether today's political system is really democratic).  I think that the clash between the ancient Plato's outlook that the individual exists to serve the society  (Orson Scott Card's president is modeled after Plato's philosopher king as he writes in his epilogue to the books), or Aristotle's outlook that puts the individual in the middle and make society as means to serve the individual.
My own observation is that we gradually move from Plato's universe to Aristotle's universe, and that the current young generation puts the individual in the center.    Societies of all types (including high-tech companies) will require to adjust to this new world.    I have already written about the distinction between these two Greek philosophers, and will revisit this topic again. 

More on big data and event processing

Philip Howard, one of the analysts who follows the event processing areas for many years, recently wrote about "CEP and big data".    emphasizing the synergy of data mining techniques on big data as a basis to create real-time scoring based on predictive model created by data mining techniques, his inspiration for writing this piece was reviewing the Red Lambda product.   It is certainly true that creation of event processing patterns off-line using mining techniques and then tracking this event patterns on-line using event processing is a valid combination, although the transfer from the data mining part to the event processing part typically requires some more work (in most cases also involves some manual work).    In general getting a model built in one technology to be used by another technology is not smooth, and require more work.  
The synergy between big data and event processing has more patterns of use -- as  big data in many cases is manifested in streaming data that has to be analyzed in real-time,  Philip mentions Infosphere Streams, which is the IBM platform to manage high throughput streaming data.   The data mining on transient data as a source for on-line event processing, and the real-time processing of high throughput streaming data, are orthogonal topics that relate to two different dimensions of big data, my posting about the four Vs summarizes those dimensions.   

On the two-second advantage

The last shipment from Amazon has brought me (together with the usual collection of science fiction and fantasy books) the book co-authors by Vivek Ranadive, the founder and CEO of TIBCO.  I purchased this book to try and have some glimpse into what's going on in TIBCO, being one of the major players in the event processing market.  However, this is not a book about TIBCO products, the book is of a type that I would call popular science, making the thesis that people who have been successful in several areas (the first example was Wayne Gretzky, claimed to be the greatest hockey players of all times) recipe of success was their ability to predict something that is about to happen before it actually happens and behave accordingly.  Part I of the book analyzes several such cases in the human case.  Part II of the book talks about the use of the same ideas in computerized systems and its utilization in several areas: making better wine, ending traffic jams, and explaining why nothing should ever break.   In part III the authors talk about the concept of "the two-second advantage" and connecting it to event-driven technologies and claim that it will both make the world better and the human brain better.  The book states a vision, I think that two-seconds is more a metaphor and not real time-interval, because for different scenarios more time ahead is needed, it also does not talk much about how to utilize the two-second advantage, since just knowing about things are just part of the picture.    The idea somehow reminds of the movie NEXT, which actually talks about seeing two minutes into the future. 

On the whole I found the vision in the book quite consistent with our own vision of proactive world, which is a major task I am trying to cope with recently, including the understanding of the cultural change aspect 
In any event - interesting reading; it is refreshing to see that CEO of IT vendor can mentally release himself from daily life to write visionary books, maybe this is an implementation of Steve Jobs' seventh rule - sell dreams not products,

On Steve Jobs' seven rules of success

I have written before about rules and commandments for intrapreneuring.   
This time I would like to bring my an1notated version of Steve Jobs' seven rules of success.  


1.Do what you love  This is probably the most important advice, I see too many people continue with the inertia,  it is always easier to continue then to change, one needs sometimes a courage to change, or to quote somebody else:  "if you get up in the morning and don't go to work with enthusiasm - go somewhere else".  I followed this rule several times in my life, will I have courage to follow it again if needed? 


2. Put a dent in the universe  When I was a beginning programmer in the Israeli Air-Force, and came to my commander with some radical ideas, his amused response was -- "our mission is not to change the world".   Most people think this way, however those who have ambition to change the world are those who have any chance to do it.   There are many pressures, especially today when Wall Street metrics governs the behavior of corporations - I am not sure how much I have succeeded so far to change the world, but it will always be the driving force behind what I am doing, and I try never to lose sight of it.


3. Make connections  The over-specialization of modern life make people narrowly focused. This is good for achieving expertise of a certain area, but making impact on the world requires making connections between seemingly not related things from various disciplines. To make connections one has to do two things:  be broad minded, and explore other areas, and connect with people whose expertise are in other areas and inspire them share the same goal.  None of these are easy.


4, Say no to 1,000 things  This is the art of focus on the right things, as one has finite amount of time and energy.  In Apple it was reduction in amount of products.   In the case of an individual -- try to focus the energy on a single goal, and fight temptations to do something else.  This is a rule that I am well aware, but must admit that I have not completely followed.


5, Create insanely different experiences  Any interaction with other persons is an experience.  Apple had its Apple stores providing different experience,  in the individual level -- the way things are presented and communicated is not less important than the content.  Always tend to (positively) surprise.


6. Master the message  I often see people who try to sell or deliver some message that they don't master, it is easy to confuse them with trivial questions, and their entire message seems confused.  Mastering the message is important, starting with the message to yourself,  if you cannot convince yourself you'll not convince others.


7. Sell dreams, not products  In the IT area, both the buyers and sellers are more interested  in how something fits the architecture, interfacing with other products, and other technical properties, then in what it really does and what is the benefit.      Selling dreams in form of -- let's imagine how you can improve the quality of your life or your business -- is better approach - which makes it necessary to understand the needs and psychology of the customer.

AITE says that vendors need to be more aggressive in marketing event processing

I have not read the recent Aite report entitled: "complex event processing - beyond capital markets",  have to check if IBM subscribes to these reports.  But from the promo page linked here I copied the figure above. It is interesting that the numbers are somewhat different among analysts, but all shows growth.   One insight given in the report is that vendors should be more aggressive about their message, since many customers still don't understand what event processing is -- a challenge both to the vendors and to the community.   One of the challenges we'll work in the EPTS level is to disseminate the event processing manifesto and ideas  more aggressively in 2012 -- stay tuned for announcement in this area. 

The MSc exam of Ella Rabinovich

Yesterday I chaired the MSc exam of Ella Rabinovich, who finished her thesis in joint supervision of Avi Gal (who was my PhD student) and myself.  Looking at my own record, Ella is my 21st MSc graduate, in addition to 6 PhD graduates (one person, David Botzer, has done both MSc and PhD theses under my supervision - so the list comes up to 26 people).    I am doing graduate students supervision as a hobby (the Technion does not pay for it),  it enables to investigate idea that are typically one or two steps ahead from what we are doing in the IBM Haifa Research Lab (which by itself has to be some steps ahead of the markets), Ella works also in our Proton team in IBM,  the thesis deals with pattern rewriting as one of the means of performance improvement in event processing run-time,  I have already written about our DEBS'11 paper on this topic.   the thesis was well accepted by the examination team and got a relative high grade.   My guess is that we'll hear more about notable  research contributions of Ella in the future. 

Continuous event processing in Quartet FS

Continuing to survey additional product related to the event processing area, I came across Quartet,   this illustration is taken from Quartet FS' webpage.   Quartet FS advertises its product as "aggregation engine",  from the description it seems to be some incarnation of active database, where the OLAP cube is constantly updated, this variation is useful for some financial services applications.      I guess that we'll discover more event related products coming from different areas within different frameworks (in this case - OLAP/BI).

Proactive thinking as a cultural change

Yesterday I've returned home from my last trip in Europe (last for the next few months I hope, at least no trip is now planned in the horizon).    I spent Thursday in Zurich in a all day meeting with partners in a consortium we have created applying for EU project,  we have reviewed several use cases, and talked with domain experts in several areas who told us that in their domains, proactive solutions simply don't exist.    Reflecting on that I think proactive computing might be enabler for cultural change.   We are very much focused on technology -- what is the architecture, what is missing beyond the state-of-the-practice, what are all the moving parts -- this is of course important -- but let's look at the bigger picture;  the society is governed by policies, regulations, rules...  Likewise, enterprises like to have strict policies to handle each situations, current analytic software serve this culture -- it typically serves those who make the rules, trying to constantly improve the rules by analyzing the past; the result is recommendation to the policy makers to refine the policies.  
Proactive thinking is different,  here the aim is to determine when following the current rules will yield wrong result and change it, if possible, in autonomous way.   If we take a simple example,   in the Zurich meeting we have discussed traffic management systems -- there has been a lot of work in this area,   there are system who sense traffic congestion and alert about it; there are systems which learn traffic patterns according to hours a day, day in the week, and other relevant segments, and make optimization for the traffic-light policies.  In many cases, traffic light policies are determined by persons who make additional considerations (a powerful politician that lives in some street -  entering and exiting this street in certain hours get priority), anyway, people generally want the power to make these decisions, and prefer that the system will behave according to consistent rules.  Proactive thinking in this case is identifying anticipatory traffic congestion and change the traffic light policies accordingly, overriding the regular policies, trying to resolve a specific problem.


This is a cultural change, since the culture typically prefers that the measures will be statistical, and the behavior of systems will be consistent,  this preference has deep cultural roots in certain societies, and reflect the inability of people to get real-time decisions, or improvise.    Those who will adopt proactive thinking may go beyond the cultural handicap and achieve relative benefits....  more about it - later

Darko's PhD defense

On Wednesday I have been in Karlsruhe, Germany,  been on the exam committee of Darko Anicic in Karlsruhe Institute of Technology, one of the elite universities in Germany.   After the defense Darko's colleagues gave him this hat with a lot of pictures and some gadgets glued to it, and he had to guess for each of them what is the meaning and how it is related to him  (because of me, I guess, they have conducted this ceremony in English).    Darko's dissertation was on Etalis, a logic programming based event processing engine, Darko has investigated the implementation of event processing on top of a logic-based knowledge base that contains background knowledge and inference capabilities.  He also took the challenge and implemented the Fast Flower Delivery example from the EPIA book; Etalis is available as open source.  
I think that Darko has done very good work, both in the conceptual level and in proving the concept (he has done measurements that indicated that the performance is roughly equivalent and some times better than existing engines).  I hope that Darko will be able to continue and grow as a researcher. 

The Fear Index

This morning I have finished reading "THE FEAR INDEX" by Robert Harris, who is known for his ancient Rome novels.  This time he has a "science fiction" that happens in the present coupled with a thriller.  It concentrates around hedge fund that uses AI for algorithmic trading,  and succeeds to crash the market and make a  huge earning by trading in short.   It is inspired by the 2010 flash crash.  


As I am a big fan of autonomic computing and  it utilization in daily life and in the business world, I often encounter people who have concerns about computers making decisions, and computers taking over.  Stanley Kubrick and Arthur C. Clark wrote the famous plot for the  film of "2001 space odyssey" in which the spaceship computer took over, the computer name was HAL, which is backwardly-removed one letter from IBM.    Some science fiction writers like Asimov also have a vision that computers (in this case android robots) will take over the world, but Asimov considers it as a good property,  since he was not a big fan of the human race.


I am not sure we are close to computers taking over the world, but autonomic systems should be constructed with self-monitoring that can be visible.  This is something that we'll be exploring (among other things) within new proposed project we are doing with many partners.   

On video streams and events

I have not written in this Blog for a while, last week I have been in a business trip in Brussels and Luxembourg, had a few hours to play tourist in Brussels and chose to go to mini-Europe, in the picture we see the France section - in the background the atomium, Brussels' iconic site.  Travelling in the train between Brussels and Luxembourg I had two observations:  the train accumulates delay, so it is better to  take a big slack of time if you rely on trains in Belgium, also it was interesting to see in stations the amount of people getting off to take advantage of the 2-3 minutes of stopping in order to smoke,  the amount of smokers in Europe in general, and in Belgium in particular is probably much higher than those we are used to in Israel (and other countries like USA and Canada).   

Getting back to the events business - I have noticed Rainer's posting in the complexevents forum, about video camera on the bus in Ireland that he experienced, actually there are various cameras on buses for various reasons - example video cameras installed on school buses whose aim according to the site is: 

•    Deter bullying, vandalism, and other offenses
•    Reinforce driver accountability
•    Document chain of events
•    Substantiate or refute claims
•    External stop arm activated video of passing motorists.

Realizing these goals there is a need to identify the events that need to be processed/notified/acted upon from the video stream.  Some are required in real-time - for example stop fights or eliminate accidents.  There are various other uses of use of video streams as source of events,  I guess that within the "Internet of Things" infrastructure, cameras are important part - while some work on detecting events out of streams exist (e.g. identify car license plate numbers for automatic tolls in roads),  more work is needed to fully exploit the opportunities here

Evented API

I have written in the beginning of this year about the fact that standard APIs is now an emerging trend,  and about the benefit of having standard API for event processing.  No standard yet but I came across "evented API specification" developed by Kynetx.   While the terminology is not entirely consistent with the terminology I am using, it is interesting to review such works, and advance into standard APIs in this area. 

On use of event processing in routing

Eric Roch, chief technologist of Perficient posted a question about the use of event processing for message routing rather than ESB, the router pattern is taken from the enterprise integration patterns by Gregor Hohpe and Bobby Woolf that we cite several times in the EPIA book.  Roch mentions several reasons to use event processing, some of it  relates to non-functional characteristics such as low latency,  and in-memory management.   


I think that there is some continuum that relates to the routing requirements -  some of the routing decisions are quite simple, they are based on a predicate, typically on a value of a single attribute;  some are more complex, and the routing is determined according to a pattern that involves multiple messages, in other cases the routing decision might involved a structure like decision-tree or decision-table, yet in other cases the routing is determined according to probabilistic reasons.    


The better solution is not to have ESB and EP systems in isolation, and chose in each case what should be used, instead it is better to use a single programming model that includes all types of routing, and autonomic translation to implementation in various building blocks,  some of them are taken from event processing and some from ESB or traditional messaging systems,  this is the approach I prefer for event processing in general,   not stand-alone,  but synergistic with other technologies.    In 2008 I had a posting about event processing and ESB,  arguing that ESB is a natural carrier of event processing within a SOA architecture,  and indeed I think that we'll see more and more ESB products that are enriched by event processing capabilities. 

Pura vida

Pura vida  - means "pure life", is an expression that is being used in Costa Rica as: good morning, hello, response to the question "what's up?", general statement that life is good, thank you - and some others.  It is the motto behind life in Costa Rica.   We spent in Costa Rica 10 days (1 less than we planned),  and here are some impressions:

1.  Costa Rica is an excellent place for a trip in the nature - it has nice rain forests, rivers and volcanoes, as well beaches of the two oceans.
2. Animals -- you see them everywhere in nature.   There are various tours in foot, car and boat to see animals.   
3. The Parador hotel we stayed in the last 3 days of the trip is the best hotel I've ever been to (and I traveled all over the world). The picture above is the view from  the hotel room, it was elected several times as best hotel in central america (including Mexico), and best spa (I got there the best massage since my visit in Thailand in 2007), t it is located within a forest on a mountain overlooking the pacific  ocean in Manual Antonio,  it has various animals on site - a hawler monkey provided wake-up call in one of the days in 5am, with extremely loud voice, on the tree in front of the hotel room; a raccoon entered the daughters' room, when they did not close the balcony's door, and scared them a bit, other raccoons visited us at the dinner table,  and were scared away by the hotel staff.  An iguana caught some sun in the middle of the mini-golf course, and frogs have settled at night in the swimming pool. 
4. The local people in Costa Rica eat the same food - breakfast, lunch, dinner -- rice and beans and tropical fruits (in each they might have some supplements).   At the end we got tired from this food and escaped to the sushi menu at the Parador.
5. Rain -- Costa Rica has a lot of rain, especially in this season. It has a four day heavy rain storm at the pacific side, we had there in the last two of these in Monteverde, but we did most activities in the rain; luckily when we got to the beach the rain storm was over.
6. The road infrastructure cannot cope with the quantity of rain, thus many of the road have lot of holes, some of the main roads are not paved, and they call travelling in them as the "Costa Rican Massage".
7. Tourism has become the biggest industry in Costa Rica, over the last 25 years, which directed the education system towards this goal.   Most people can speak English.
8. The amount of Israelies we met was incredible given the distance and cost to get there,  it seems that it is now quite fashionable among Israelies who already explored Europe and USA to visit Costa Rica.
9. Iberia is the worst airline company I have ever used ---  in all aspects (including food),  the worst thing was that they did not let us get on the connection flight from Madrid to San Jose, Costa Rica, though we got to the gate 15 minutes before the scheduled flights, due to the fact that our flight from Israel was late 1 hour, and they decided they'll not have enough time to load the luggage, though we were 20 passengers coming on that flight, and the fact that they have 1 flight per day to Costa Rica, they did not agree to make any effort to get us on the flight.   They told us that since ELAL was responsible for the late flight - it is not their problem (we purchased the ticket from Iberia, and used Iberia ticket with code share).   We had to spend 24 hours in Madrid before getting on the next flight -- and lost the first segment of the trip (which is non-refundable as anything else in Costa-Rica).  I'll check if it is possible to sue Iberia for damages.   The attitude of their agents in Madrid was also very unfriendly - a big difference between the nice people of Costa Rica everywhere, and those of Spain). 
10. And on positive aspect --- Pura vida --- is a motto of positive thinking!

For more picture --- I am waiting that my daughters will put all the pictures taken on facebook. 

Now it is a lazy holiday -- back to work early next week, and already a business trip to Europe!

Touring the universe again; current destination: Costa Rica

In a few hours we are planned to start the long journey to Costa Rica, the chosen destination of our next Family vacation,  during the holiday period (here in Israel).     Will be out of touch for the next couple of weeks.

More on Internet of Things and event processing for personal systems

This illustration is taken from a recent Blog posting by Phil Windley, the CTO of Kynetx, whom I met in DEBS'11.   Phil writes about IoT as a source of personal event processing, and about "personal event bus", in which any consumer can subscribe to events and process them individually.


I have provided some background about IoT recently and about its potential to be game changer;  the personal systems is an interesting aspect.   Event processing models and systems of the first generation have been developed in mind of business oriented systems. However, I have stated before that I see the big potential of event processing outside the traditional corporate IT, and provided robotics and biological computing as some areas of interest.  Consumer applications is definitely an area with a lot of potential, however the challenges to do it is beyond collecting and subscribing to events, it is the ability to compose event processing application by consumers, which requires progress in the abstraction level of composing such applications.  This is definitely and avenue to pursue. 

On context and punctuation

This illustration is taken from the EPIA book and shows the notion of context as a first class construct that groups together events that need to be processed together, where each of its instances is processed as a single unit,  context may have temporal aspects, segmentation aspects, spatial aspects, and state-oriented aspects, and the actual context instance may be combined of one or more aspects.  

Recently I had some discussion with somebody from the data stream community who was in the opinion that the stream oriented way to do it is more simple since they are based on standard thinking (i.e. SQL).

In the stream research literature, the temporal context is divided between windows and punctuation, all the other aspects are somehow expressed by the group-by construct.    I guess that windows and group-by constructs are familiar to many people, punctuation is less familiar, thus I'll explain the idea briefly. 

A good source to learn about punctuation is Peter Tucker's site, from where the illustration below is copied:

Peter Tucker has done his PhD (under the supervision of Dave Maier) on data stream punctuation;  the idea of punctuation is to enable creating sub-streams, since a stream by definition is infinite.   The notion of windows in classical data stream models consists of variations of sliding windows, either by event count, or time, however, in reality there is a need to explicitly end stream, i.e. when the bid ends.  The definition of 
punctuation in the database encyclopedia (the value was written by Maier and Tucker) says:

In other words,  this is a "dummy event" put into the stream to denote the end of a sub-stream.  


Now the question is whether using windows + punctuation + group-by is indeed simple.    The relational model's claim to fame is its simplicity, however, from early days of the relational model, it has been realized that there are semantic anomalies, and all the normalization rules came to resolve those anomalies, yet, the overall model has been simple.  However, this is not really true for the extension of the relational model for streams,  while the relational model has a single entity: relation,  the extension has multiple entities: relation, stream, window.  Punctuation is a kind of trying to add semantics as a kind of logical patch.  


IMHO, the context model is semantically cleaner.  It has a single semantic notion for all the constructs that determine how to group event together (in the stream terminology, create sub-streams).   In the context model  there are no dummy event; the notion of temporal context (window) can also support event intervals - time windows that start by event and end by event; the events that start and end the window are real events, and can be used for other purposes.  Moreover, looking at the definition of punctuation above, it defines the end of a sub-stream, but assumes that the next bid starts when a new bid event arrives.  However, if we look at a situation where bids have not only "end bid" events, but also "start bid" events, then we would like to ignore bids that arrive when no bid is open,  and this has no natural representation, and has to be implemented with tricks, so we also need the equivalent of  "start sub-stream punctuation".   So, we view this as kind of context based on , and not as kind of event within the stream with a special semantics. 


The model we presented in EPIA (which we keep evolve)  to be a semantic model that provides abstractions above the current implementations of both event and stream processing systems;  several product owners have already told us that they use concepts from the book as inspiration to next versions of their products, and I guess that full implementation that is based on this model as native is yet to come.   

On the four Vs of big data

In my briefing to the EU guys about the "data challenge", I have talked about IBM's view on "big data", recently Arvind Krishna, the IBM General Manager of the Information Management division, talked in the Almaden centennial colloquium about the 4Vs of big data.  The first 3 Vs have been discussed before:

• Volume
• Velocity
• Variety  

 While the 4th V has just been added recently -  Veracity -- defined as "data in doubt".


The regular slides are talking about volume (for data in rest) and velocity (for data in motion),  but I think that we need velocity to process sometimes also data in rest (e.g. Watson),  and we need sometimes also to process high volume of moving data; the variety stands for poly-structured data (structured, semi-structured, unstructured).

The veracity --- deals with uncertain/imprecise data.  In the past there was an assumption that this is not an issue, since it would be possible to cleanse the data before using it,  however, this is not always the case.   In some cases, due to the need of velocity in moving data, it is not possible to get rid of the uncertainty, and there is a need to process data with uncertainty.    This is of course true when talking about events, uncertainty  in event processing is a major issue still need to be conquered.  Indeed among the four Vs, the veracity is the one which is least investigated so far.    This is one of the areas we investigate, and I'll write more about it in later posts.   

CICS event processing improved version

IBM CICS is an example for smart producer of event processing system, it does not do event processing inline, but instruments CICS transactions to emit events, and works in a loosely coupled mode with any event processing engine that can read its emitted events.  CICS TS 4.2  released recently has several improvements in the CICS event producing capabilities.  Among these improvement are:

• Including the event emission to be part of the transaction, by doing the event emission as part of the commit process.  Note that since it is loosely coupled with the event processing itself, this does not becomes atomic unit with the event processing itself, I have recently written about the relationships between transactions and events, and identified this area as one that need to be investigated more. 
• Change management inside the event instrumentation in CICS with appropriate tools
• Inclusion of system events inside the CICS instrumentation (e.g. connection/disconnection to databases, transactions aborts etc..). 

Since the strength of a chain s typically equivalent to the strength of the weakest link,  in many cases the producer is the weakest link, and the amount of work required to emit the right events and the right time is often much larger than the rest of the system.   Smart event producers like CICS making this weakest link much stronger.

On Actian and action applications

Ingres, one of the oldest DBMS companies which produces open source DBMS, and the first of the sequence of companies that Mike Stonebraker founded and sold, has recently changed its name to Actian, and positioned itself as focused on "action applications in big data".  The stated rationale about "action applications" is that current BI create reports and then it is left to the human to read the reports (or screens) and decide what to do, in "action applications", the application trigger actions automatically in response to data events and thresholds.   It seems that people from the BI community re-discovers/re-invents the Event-Condition-Action model?   so they'll probably get to more advanced event processing at some point.


It is interesting to note that the motivation they state on the Actian website (you'll have to press on "action apps" to see it) is - "BI is not working, more than $10B are spent every year on a pile of reports with no actions".    I guess that this is somewhat consistent with my previous posting citing a study that indicate that human decision makers don't succeed to get fast decisions based on BI.   Maybe BI is getting in the hype cycle to the phase of disillusionment, and maybe people in this community like SAS CEO who said last year that event processing has limited appeal  to BI (along with BI in the cloud), would have second thoughts.

On proton-based chips

Asimov talked about positronic brains for robots, maybe one step was the construction of proton-based chips, instead of the regular electronic-based computing (reference made by Rainer von Ammon on the complexevents forum).  The proton-based chips can have better potential to communicate with biological processes and serve as an actuator within the human body, even the human brain.    When we called our proactive computing project in the name "proton", we did not know about such a chip,  but it can have natural connection:  The proton system makes decisions that are carried out using the proton chip that serves as an actuator for the proton system.  Cool idea -- somewhat futuristic, but we are getting there.   While the scenarios we are working on now are still within the familiar domains,  I believe that the big potential of proactive systems are in robotics and computerized-control of biological processes.