On Event Reduction

The new year is going to arrive in less than three hours (local time), we don't really celebrate the new year, as our new year has already started in September, we celebrate our holidays according to the "Hebrew Calendar", but the daily life is handled according to the Gregorian Calendar, so I am mentioning that date. The mood here is not a celebration mood, various locations in the universe have their own natural disasters: The east coast of the USA and the Caribbean Islands have Hurricanes, parts of Asia have Tsunamis, the west coast of the USA and some other places have earthquakes, and we in Israel have periods of fightingwith one of our neighbors. Some are viewing it as the natural disaster of the middle east, like hurricanes; however, unlike hurricanes, I strongly believe that human violence can be avoided, but will not get in this Blog into the very complex situation. For those who were asking me about my personal safety -- this time we are (so far) quite far from the combat area, and in general -- I have better feeling of personal safety here than in many other places in the universe.

An interesting comment about my previous posting made by Richard Veryard
Richard referred to the comment I've made about being more productive in a Cafe, and asked whether this is a result of getting less interrupts, and what we can project to event processing in the enterprise level.

I think that this is a good point, actually, one the more marketed benefit of event processing systems are their ability to help not missing any event that requires reaction (I think that the term identifying "threats" and "opportunities" is much over-statement of most detected situations, but will write about it another time), in some cases, the business benefit of event processing system is actually reducing the number of events, and focus the decision makers on the important events.

Some examples:

• In Network Management there is a phenomenon knows as "event storm", e.g. when some network segment is out, many devices send "time-out" alerts, which are just the symptoms of the real problem. What we want is to reduce this event storm to a single event that need to be reacted upon.
• I would like to get alert when my investment portfolio is up by 5% within a single day (as you can see, I am still optimistic). Here I don't care about any of the many raw events about any of my investments, but about the situation defined above.
  

The conclusion (not very surprising) is that sometimes less is more --- the event processing system can eliminate events in various ways:

• Filtering out unnecessary events
• Aggregating multiple events to one
  
• Report on derived event when some pattern is detected.

I'll blog more about this issue, but remember -- some times less is more and more is less...

On single application vs. general event processing software - the network and system management case

There are people who like working in coffee shops, in my academic days I had a colleague in U.C. Berkeley which liked to work in one of the coffee shops that surround the beautiful campus. I am usually work at home or in my office (where I sometime spends 13 hours per day), but today I spent the morning in a coffee shop called "Grand Cafe", well - not quite the famous one in Paris seen in the picture, but a much smaller one bearing the same name in Haifa. I am not sure that it was cost-effective for the coffee shop, since I ordered one mug of "upside down" coffee, which is the Hebrew name for a coffee in which the milk is put into the mug first and the coffee later, anyway - I spent the time in writing some paper, and found that the coffee shop setting actually makes me more productive then my office... Will try again to see if this is consistent...

I have also talked recently with somebody who works for one of the NSM (network and system management) vendors; the discussion was about my favorite topic -- event processing, and why the NSM companies who are dealing with events as primary occupation did not really try to look at "event processing" as a more general discipline and extend it to more types of applications. The answers I got from him reminded me of two things --- one from the far past, and one from the near past.

From the far past I recall that when I was assigned as a programmer to the Israeli Air-Force IT unit, when I was in the age of 18.5, I got my first assignment, and very enthusiastically wrote that program (in those days -- in PL/1..), and then got the second assignment, when I got my second assignment I read it carefully, and then went to my division commander and told him that the second program is quite similar to the first program, the change are in some details -- somewhat different input, somewhat different validity checks, somewhat different calculations, but the same type of thinking, so instead of writing this program I suggest to think bigger, look at this class of programs and try to write "general parametric program" which takes all the individual programs as a set of arguments to the general program. My division commander heard me with a lot of patient and then said: great idea, however, our mission is to write certain kind of applications for the air-force, and not to invent general programs. You may talk with the guys in the "chief programmer" department (what today we would call CTO), they are in charge of generic software. I was just a beginner and he was a Captain, and the commander, so I deserted this ideas, but pursued them later in life, as I always was under the impression that people program again and again the same things, and the level of abstraction in programming should be higher.

So - as you can guess from that story, the NSM guy just told me: Our aim has been to build network and system management and not generic software.

I also remembered that there was some discussion on the Web on the same topic, and found in David Luckham's site an article
entitled "an answer from the engine room of the industry", the question that David phrases is:
I have often asked why the network monitoring applications that were developed in the late 1980’s and early 1990’s didn’t get extended to apply to business level events at about the same time.

The question is answered by Tom Bishop, the CTO of BMC, who is seen here in the picture.

I've met Tom Bishop in the late 1990-ies, after IBM has acquired Tivoli (and NSM vendor), and we in IBM Haifa had a project with Tivoli; Tom made an impression of an impressive big guy with Texan accent. Now he is BMC CTO. In his answer to David Luckham he makes roughly the same answer, in three parts (quoting Tom Bishop) :

1. When the architects for these products were building them, they weren't actually thinking of the broadest applications for the types of systems they were trying to build, but were really focused on solving a very specific problem;
   
2. As we know all too well, often the correct way to solve a problem is to find the most general description of the problem and then assume that, if you've done your job correctly, the specific solution can be described as an instance or subset of the more general problem. But this only works if you know to set your sights high enough. In the environment you note above, this didn't happen.
3. The people who buy IT management solutions don't care if the solutions they buy might also be used to solve a business activity monitoring solution, and the people who buy business activity monitoring solutions don't care if the solutions they buy might also be used to solve an IT management solution. In fact, these two groups of people almost never talk to each other!
   

This is all revolving around the same phenomenon --- there is a big difference between hard-coding an application doing X, and building a generic program that the application doing X is an instance of it. Furthermore, the fact there are various hard-coded applications doing variations of X, may help in requirements, but does not mean that it gets us closer to a generic application - since the level of abstraction is probably wrong.


I guess that if event processing generic software existed when the NSM software has been built, the NSM vendors would have used it, instead of re-inventing the wheel, the same as they used existing databases, existing communication network etc..

Event processing as a discipline is about creating generic software, my personal prediction: NSM vendors will gradually merge into the event processing community.

More - Later