On some recent news

Today I'll write on some (relatively) recent news that I would like to comment on.

First -- in the event processing area, the news of the week has definitely been the acquisition of Aleri (which last year acquired Coral8) by Sybase. Now most big and medium software vendors have either acquired or developed event processing capabilities. This is in-line with the trend of having event processing as part of a bigger game, we see some companies who are taking event processing as part of BI/data management game, and some as part of enterprise middleware game. The event processing industry has started as an event processing stand-alone game, since it was started by start-ups and not be big companies, but when the bigger vendors get into the game, they treat it as part of a certain portfolio. Streambase (with its expected amnesty program) now carries the flag of growing as a pure play event processing platform that can work with various platforms. I guess that there are segments in the market for both approaches, and we'll have to see where the wind will blow. Getting to standardization can help this approach.

Not really new news -- but I've noticed recently that my favorite social network, LinkedIn has come with new interface when one checks the "my contacts" option, which enables you to browse your connections in easier way, and view them by tags, names, companies, location, and recent activity. In LinkedIn when one crosses the 500 members, the number of members appears to the outside world as 500+, I guess that with the growth of the network they might consider the number, since I think there are many people with 500+, in my case, the current count is 737, and is typically growing every week. I have been introduced to LinkedIN when getting introduction from Mark Palmer, and since then the network keeps growing. The companies tell me that 237 are from IBM (and various of its sub-organizations), 17 are from Technion (my second home), and then 10 from Google, and other companies that have between 5-10 connections are: Amdocs, Microsoft, Progress, CA, Intel and Oracle. As for location, 331 connections are from Israel, 101 from the greater NY area, 49 from the SF Bay area, the list of countries contain also countries I've never visited like - Taiwan, Japan, Russia, India, Singapore, Brazil and New-Zealand. One of my main uses on this is to find Emails, but it also provides me information of who moves where.

Last but not least -- news about the Event Processing in Action book -- we have finishing the cleaning up, and getting it to production, the idea is to launch the book officially in IMPACT 2010, the IBM Websphere big conference, will write more about it. It seems that for the week of January 31, the book has been in the list of the early access best sellers of Manning.

More about it later.

On Research in Industry

I spent 12 years in various roles in the various roles in the IT industry, 10 years in academia, and 12 years ago I moved to the IBM Research Lab in Haifa, and since that time I still wonder what research in industry is all about. Recently, one of the IBM Research colleagues, the director of the IBM Zurich Lab (which is actually located a little bit outside the city of Zurich) has blogged recently about how research in industry in measured,

Research in industry may take several roles, and each of the roles require different kind of organization and type of leaders.

Role one: Research as the "smart arm" of the enterprise: In this case, the research activities are part of the daily (tactical and operational) business of the organization. Research people are going to pre-sale meetings, doing PoCs for customers, providing services to customers, and help product organizations in getting new features to products, typically in cases where knowledge and skills that they possess are needed. The research organization in this case is geared towards the short-term impact on the organization, and is measured by its tangible impact. In this case the research leadership should be geared towards good knowledge of the enterprise business and also well connected internally, to trace opportunities for getting more traction for business; the research is being funded by the business units according to its contribution, and the research leaders should be able to smoothly be able to switch jobs with any other leader in the organization, as organizational business and networking is the main qualification.

Role two: Research as the "scientific arm" of the enterprise: In this case, the research activities are geared to advance science and to research areas that are geared towards impact on the knowledge in the universe, and on long term directions of the enterprise. One of the questions is what is the tangible benefit of the enterprise from this type of research has been investigated recently to view what is the $value on the brand image, as a function of the reputation of research done there, this is also reported in the IBM Zurich Lab Director Blog. In this case the evaluation can be according to research quality metrics common in the academic world, as well as the impact of the research on corporate. In IBM Research the relational database is often mentioned as the most influential research projects both on the science and on IBM business. Funding can come from the enterprise as well as from external and government oriented research grants, similar to academic research funding. In this case the research leaders should be able to switch jobs with Deans of relevant faculties in top universities, or head of independent research institutes, as good familiarity with the research community and ability to lead scientific work, and good acquaintance with the research grant business.

Role three: Research as the "incubation arm" of the enterprise; In this case, the research activities should be constructed as a collection of start-ups inside the enterprise. Each of them gets some seed money and gets to its way, having the ability to sell and gain customers, in some cases even get external funding as partners. As in any incubator, some of the start-ups may fail, some of them may exit, and develop spin-offs out of the enterprise, and some of them grow inside the enterprise and become business units, in a similar model to the model of acquisition of other start-ups; this is an interesting model that typically enterprises don't really know how to apply, but there were examples of technologies, companies, and business units within the same enterprises that were originated in such start-ups. This also distribute the leadership, since each such start-up requires a leader with intrapreneurship abilities. The funding model is of start-ups where the enterprise is acting as VC, and the overall leaders should be able to switch jobs with VC executives, and they also need to create an atmosphere that leverages the enterprise resources where it helps, and masks the enterprise when it interrupts; The leader of each individual start-up leaders should be able to switch jobs with external start-up leaders (though inside start-up also requires some internal political skills).

Current research organizations try to have a mix of all these, the main issue is the diversity of leadership skills and how work should be organized which is different in each of them vs. the tendency of organizations to have homogeneity both in the organization type and in type of leaders.

On reasoning about events and states

Yesterday, we watched in our local theater, the play "Amadeus". I have seen the movie when it was first launched, and another version of this play in the nineties. Still very impressive, and an opportunity to hear some of Mozart's good music.

Thanks to Paul Vincent's Blog, I clicked the link and got to Paul Haley's posting entitled "time of the next generation of knowledge automation". Paul Haley starts his postings by classifying four types of reasoning:

1. Reasoning at a point within a [business] process
2. Reasoning about events that occur over time.
3. Reasoning about a [business] process (as in deciding what comes next)
4. Reasoning about and across different states (as in planning)

The claim is that the first kind is being supported by business rules (called EDM in the original), the second kind is being supported by event processing, while the third and fourth kinds are not really supported by the state of the art. The third type requires getting the notion of context into reasoning while the fourth type requires getting cross state reasoning (which is different from cross event reasoning). I agree with the classification, and will write more in the future about the third and fourth types in depth. Actually, we also need reasoning that will combine the different types of reasoning as well.

More on taxonomy and classification in event processing

Back to reviewing some recent Blogs on event processing, this time I am looking at Hans Gilde's Blog talking about the two types of event processing applications: detection oriented event processing and operations oriented event processing.

Detection Oriented Event Processing applications want to locate interesting information in a flow of data. Operations Oriented Event Processing applications want to take an action (which involves making decisions) based on incoming events.

This classification does not seem to be mutually exclusive, there are actually systems that locate interesting information in a flow of data, and after finding this interesting information are taking actions, which involves making decisions, based on that detection. Hans is right that these are two different things, one deals with the detection, the other deals with the action, and decision around the action.

Getting this classification further, let's look at two different issues: first -- what does the event processing system do (the "detection oriented") , and what are the results of the event processing system are used for (the "operation oriented").

Looking at the first aspect -- an event processing system can just filter and route event, it can also transform events (translate, enrich, aggregate etc...), and create derived events, it can detect predefined patterns, and it can detect patterns on the fly based on some reasoning, it can do part of these things, or all of them together, so an event processing platform is actually a tool box of all the above.

The event is then consumed by some consumer -- person, dashboard, system, and is (hopefully) used for something -- and this is the second aspect -- it may be used just for notification to a person, and what the person does or does not do is beyond the scope of the system, it can observe something and update a dashboard, it can diagnose something, it can do some real-time action based on decision, either in reactive mode or proactive mode (to mitigate some predicted state). This classification is based on a work we have done in IBM a few years ago by looking at applications in many industries, I have described this work in the past.

These two parts are typically decoupled. The event processing system gets events from some event producers, do some processing with it, and produce some events to the event consumers. There are certain set of techniques to do all these types of processing. The second type is done on what is called in the event processing architecture the event consumer. It gets event from an event processing system (or directly from an event producer if no processing is needed) and decides what do with it, the decision itself may apply business rules, and/or some quantitative analytic tools -- optimizers, simulators etc..., the decision can be totally automated ("autonomic"), semi-automated ("decision support system") or completely manual.

In essence, there are more and more systems that have complexity in both parts, thus the tool box to build this entire system is more extended, and while the execution is decoupled, the logic may be viewed in integrative way to build "decision processing network" in which "event processing network" is a subset, but it also includes business rules and quantitative analytics (and maybe some decision visualization tools).

I'll write more in the future abut the notion of "decision agents", introduced by James Taylor. This term reflect the atom of processing in both of the types that Hans is mentioning, however, in each of these type the agents are doing different things. I have mainly concentrated so far on the "event processing agents" type of processing in this Blog, but needs also to pay some attention to the other types of decision agents as well...

More - Later.

Review of Mark Palmer's event processing predictions for 2010

As an skeptic person, I have never really believed in astrology, but reading it from time to time, and somehow there is some correlation between years in which I had major changed in life, to years in which the astrology has predicted it, but this is not a scientific evidence, of course.

Anyway, the current prediction about this year came from Mark Palmer, the dynamic CEO of Streambase, who made nine predictions. Of course, his predictions were made from the viewpoint of his current role, but I thought that there is some value to review it. I'll say a few comments on them, one by one.

Prediction #1: 100+ Event Processing Applications Per Firm Milestone will be Passed.

This is something that I would like to see, and if will happen it will show that event processing is really part of the main stream computing. Personally I don't know any organization that is even close to 100+ applications, but I guess that Mark has somebody in mind. It will be even more interesting, if the 100+ applications will not be just thousand islands, but will also be connected, use heterogeneous event processing platforms...

Prediction #2: The "Deluge of Real-Time Data" Will Not Drive Event Processing Adoption

A refreshing view when it comes from Streambase which in the past over-hyped the mythical "event per second",

I totally agree, the main value of event processing seems to be the agility and reduction of total cost of ownership.

Prediction #3: Cognitive Physics Will Be as Important as Computing Physics

This is a challenging one, I was not familiar with the term cognitive physics, but with the help of all mighty Google, I found something about it on the Web. Anyway, Mark means that the way we communicate with computerized systems should align better with the way we think, getting to higher level abstractions, and consequently again to agility to implement new stuff. I totally agree that higher level languages is one of the most important directions for event processing, as a matter of fact, this is one of my major interest areas.

Prediction #4: Quantitative Thinking will Trump Traditional Thinking

This again has to do with higher level way to express quantitative thinking. Event processing is just one of various techniques to express quantitative thinking, the rest are - business rules and various analytics. Decision platforms will be combined of all. Event processing is certainly a player here, but not the only one.

Prediction #5: Software Stacks will Continue to Miss the Mark

There are systems in which event processing can be stand-alone island, but in many cases the producers and consumers for the events are coming from applications that have already been developed using the software stacks, thus, there will be a benefit to have good integration of event processing with these software stacks. As mark said customers are interested in solution to their business needs and not in technology, in some of the event processing systems I've seen, the main investment was not in the event processing system itself, but in its connectivity with the various producers and consumers. I agree that it is not cost-effective to buy an entire stack only to use event processing, but the same organizations that will have 100+ applications of event processing (prediction #1) are also those who already have BPM/messaging/application servers and other type of middleware and they are looking for good integration story, so this is not a black-or-white situation.

Prediction #6: A New Event-Based Software Stack will Emerge

The main claim here is that alternative stacks will emerge as combination of innovative technologies that comes from small companies. I've heard this idea before, in order to make it happen, more interoperability standards are needed, since the different parts of the stack need to talk to one another. I guess these standards will come, but from my limited experience, standards is a slow world.

Prediction #7: Stream-Based Platforms will Continue to Lead a Siege Against the $15B Data-Base Market

I am not sure what is the meaning of stream-based platforms, since the first predictions were made about event processing, and the body of this prediction also talks about event processing. Anyway, I agree that event processing is positioned as complementary to the business intelligence and general DBMS market. Analysts also claim that BI tools will have event processing capabilities (another software stack!), but still have some way to go until becoming main-stream there.

Prediction #8: Open Source CEP Won't Impact the Market, But Open Source Will

This is probably true for 2010, but might not be true for the longer future. When the event processing area will mature more and have its own standards, the event processing instance of MySQL may emerge.

Prediction #9: Event processing will Yield a Great New Software Powerhouse

Sometimes the emergence of new market can grow a software vendor from start-up to a medium or large software vendor, this has happened before, it might happen again - if indeed event processing will become a multi-billion business, I guess that part of it will be done by the current super powers in the software world, but some can go to growing companies. This will probably not happen in 2010, but I wish Mark success in his aspirations. Thinking big is a good practice, and a necessary but not sufficient condition for success.

On mobile phones and standards

Today I received a new mobile phone, since one of our family's mobile phone was broken, we did some shifting around, and it was my turn to get a new one. Starting to work with it was somewhat frustrating, I found myself in unknown territory, although it was produced by the same vendor of the old phone, they did not succeed to move the ringtones from my previous phone (although I purchased them in their store), since there is no standard for ringtones and each instrument represents it differently so they gave me a credit to get free ringtones from their store. They succeeded to move the list of my contact persons (there is a standard for that), but not the different groups defined for the contact person (no standard), the user interface is totally different, and requires some learning and more.

I have changed laptops several times in the last few years, and the transition was smooth, everything just worked -- no need to study anything new.

In our world of event processing we are not in better shape then in mobile phones, actually we are worse. In the event processing course that I have been teaching this semester, the students had a task to study six languages (each team has its own languages) and on Monday they will present the result of their experiments. I thought about it when struggling with the phone, if one of them was in the team that studied language X in the course, and tomorrow they find themselves working with another language, this might be somewhat confusing for them. In the course I am teaching our building block approach, which is valid for all languages. Working with standard oriented software is easier, we still have a challenge here.

Requiem to a small yellow cat

The small yellow cat adopted by my daughter a year ago passed away today. Eight days ago we realized he is sick and took him to the veterinary clinic, where they diagnosed that he was attacked by a violent virus, which his immune system could not handle. We were told that he'll probably last for 2 months or so, but it seems that this has been over-estimate. He had short life, only a little more than a year, but had a good life relative to his peers that live in the street. I have never been a cat fan, but over the last year he became part of the family, and was very attached to Hadas, my third daughter, he was very sad every time she has been away from home for a few days. For some strange reason the battery in my watch went dead at the same time he died, and my watch stopped. I have already replaced the battery, replacing a living creature will be more difficult, but we might adopt a new kitten after my daughter will get over...