On event processing components exchange

I haven't blogged for a while, hectic week is ending (our working week is Sunday - Thursday, so this is the evening of the last working day!). Meanwhile I have noticed that Streambase has declared "component exchange program", looking closer, this includes a collection of stuff, starting from Twitter adapter, and even including the Streambase implementation to our "Fast Flower Delivery" example. Very good idea. Now, the challenge is to have a global components exchange that is not specific to a single vendor. In my previous posting I have written about event processing building blocks, and having a building block exchange is the ultimate way to implement this idea, everybody will be able to mix and match building blocks. The inhibitor is the lack of standard. Let's say that I want to use two of the exchange components, and that one should send events to the other, then of course they need to agree on how event is being represented, writing adapters for each pair of components is not very appealing. Furthermore, if one develops an application out of various programmable components, one would want them to be speak the same language. So -- having an exchange around a single product is nice, having an exchange in the industry level will be even nicer. More -later.

On event processing building blocks

As response to my posting about stand alone or part of a bigger picture, Marco made a comment that in the event-driven world there is an opportunity to provide building blocks that may be used in various platforms and part of multiple bigger pictures. This is a valid point, as event processing is indeed a collection of capabilities of various types -- transformation, aggregations, pattern detection and more, each of them can also be of various types --- e.g. supporting variety of aggregating functions, variety of transformation, multiple patterns etc, coupled with the fact that event-driven computing is decoupled, thus the interfaces to these components can be quite simple --- receive events and send events, can provide an opportunity to provide variety of such components, kind of plug-and-play event processing. The key, and currently main obstacle in letting it happen is standardization and current lack of standards.

• Interoperability standards are needed for standard interfaces
• Event meta-data standards are needed so that the events exchanged between components
• Languages and meta-modeling standards so people can model and program in a seamless model.

I believe that this is the right direction going forward, and it is a direction we in Haifa Research Lab are investigating; this might be the key to make event processing a pervasive technology - more about it, later.

On EPTS Glossary 2.0

Version 2.0 of the EPTS Glossary has been posted for public review, this is an opportunity to everybody in the community to review and comment, within a few weeks, the final version will be reviewed for approval by EPTS.

Event processing - stand-alone, part of a bigger game, or both?

Following my previous posting, somebody told me that I was vague about whether I think that event processing as stand-alone technology is good or bad. Well -- when I was a student I took "image processing" course, and we worked on a graphical screens with 256 gray levels, since then I realized that even in "black and white" picture there are a lot of gray area Thus, I cannot classify it as "good" or "bad". But I'll provide some observations:

1. The event processing area started as "stand alone" engines, this has been obvious, since it started by start-ups and not by big companies as part of other frameworks
2. There is a gradual shift in the market from stand alone event processing solutions to event processing capabilities embedded in larger frameworks, when bigger companies got into the picture, and this trend has intensified.
3. "Stand alone" products may have to implement functions that "embedded" products can use existing functions in the original framework, such as: routing, transformation, filtering...
4. Unlike some software components that may need tight integration, event processing work in loose coupling relative to other components -- sending and receiving events --- thus this supports the possibility for having stand-alone EP.
5. However, there are no interoperability standards which requires to provide adapters for each producer and consumer, which makes stand-alone EP more difficult, relative to a single framework -- the level of difficulty is a functions of the quantity and diversity of producers and consumers. Enterprise integration framework may already include variety of adapters that the embedded EP can get "for free".
6. Event processing is in many cases part of a bigger application, and in this case, there is a benefit of having a single programming model for the bigger application, and not using different programming models/languages/user interfaces for the various part of the system, this also goes against stand-alone EP; In cases where the system is pure EP, this consideration may not be valid.
7. Stand-alone EP may support heterogeneous components -- e.g. work with DBMS from one vendor, messaging system from another vendor, and connect to BPM system from third vendor, while embedded EP is typically homogeneous, since it all comes from one vendor. This may be true, though today there are a lot of cross-adapters among various components that enable framework to support other components (say DBMS) from other vendors, especially where there are standard interfaces.

Is there a bottom line here? --- I guess that the gray area is that there is some segment of the market in which stand-alone EP can live, but I also think that the trend of moving from stand-alone to embedded will continue to exist.

On some recent news

Today I'll write on some (relatively) recent news that I would like to comment on.

First -- in the event processing area, the news of the week has definitely been the acquisition of Aleri (which last year acquired Coral8) by Sybase. Now most big and medium software vendors have either acquired or developed event processing capabilities. This is in-line with the trend of having event processing as part of a bigger game, we see some companies who are taking event processing as part of BI/data management game, and some as part of enterprise middleware game. The event processing industry has started as an event processing stand-alone game, since it was started by start-ups and not be big companies, but when the bigger vendors get into the game, they treat it as part of a certain portfolio. Streambase (with its expected amnesty program) now carries the flag of growing as a pure play event processing platform that can work with various platforms. I guess that there are segments in the market for both approaches, and we'll have to see where the wind will blow. Getting to standardization can help this approach.

Not really new news -- but I've noticed recently that my favorite social network, LinkedIn has come with new interface when one checks the "my contacts" option, which enables you to browse your connections in easier way, and view them by tags, names, companies, location, and recent activity. In LinkedIn when one crosses the 500 members, the number of members appears to the outside world as 500+, I guess that with the growth of the network they might consider the number, since I think there are many people with 500+, in my case, the current count is 737, and is typically growing every week. I have been introduced to LinkedIN when getting introduction from Mark Palmer, and since then the network keeps growing. The companies tell me that 237 are from IBM (and various of its sub-organizations), 17 are from Technion (my second home), and then 10 from Google, and other companies that have between 5-10 connections are: Amdocs, Microsoft, Progress, CA, Intel and Oracle. As for location, 331 connections are from Israel, 101 from the greater NY area, 49 from the SF Bay area, the list of countries contain also countries I've never visited like - Taiwan, Japan, Russia, India, Singapore, Brazil and New-Zealand. One of my main uses on this is to find Emails, but it also provides me information of who moves where.

Last but not least -- news about the Event Processing in Action book -- we have finishing the cleaning up, and getting it to production, the idea is to launch the book officially in IMPACT 2010, the IBM Websphere big conference, will write more about it. It seems that for the week of January 31, the book has been in the list of the early access best sellers of Manning.

More about it later.

On Research in Industry

I spent 12 years in various roles in the various roles in the IT industry, 10 years in academia, and 12 years ago I moved to the IBM Research Lab in Haifa, and since that time I still wonder what research in industry is all about. Recently, one of the IBM Research colleagues, the director of the IBM Zurich Lab (which is actually located a little bit outside the city of Zurich) has blogged recently about how research in industry in measured,

Research in industry may take several roles, and each of the roles require different kind of organization and type of leaders.

Role one: Research as the "smart arm" of the enterprise: In this case, the research activities are part of the daily (tactical and operational) business of the organization. Research people are going to pre-sale meetings, doing PoCs for customers, providing services to customers, and help product organizations in getting new features to products, typically in cases where knowledge and skills that they possess are needed. The research organization in this case is geared towards the short-term impact on the organization, and is measured by its tangible impact. In this case the research leadership should be geared towards good knowledge of the enterprise business and also well connected internally, to trace opportunities for getting more traction for business; the research is being funded by the business units according to its contribution, and the research leaders should be able to smoothly be able to switch jobs with any other leader in the organization, as organizational business and networking is the main qualification.

Role two: Research as the "scientific arm" of the enterprise: In this case, the research activities are geared to advance science and to research areas that are geared towards impact on the knowledge in the universe, and on long term directions of the enterprise. One of the questions is what is the tangible benefit of the enterprise from this type of research has been investigated recently to view what is the $value on the brand image, as a function of the reputation of research done there, this is also reported in the IBM Zurich Lab Director Blog. In this case the evaluation can be according to research quality metrics common in the academic world, as well as the impact of the research on corporate. In IBM Research the relational database is often mentioned as the most influential research projects both on the science and on IBM business. Funding can come from the enterprise as well as from external and government oriented research grants, similar to academic research funding. In this case the research leaders should be able to switch jobs with Deans of relevant faculties in top universities, or head of independent research institutes, as good familiarity with the research community and ability to lead scientific work, and good acquaintance with the research grant business.

Role three: Research as the "incubation arm" of the enterprise; In this case, the research activities should be constructed as a collection of start-ups inside the enterprise. Each of them gets some seed money and gets to its way, having the ability to sell and gain customers, in some cases even get external funding as partners. As in any incubator, some of the start-ups may fail, some of them may exit, and develop spin-offs out of the enterprise, and some of them grow inside the enterprise and become business units, in a similar model to the model of acquisition of other start-ups; this is an interesting model that typically enterprises don't really know how to apply, but there were examples of technologies, companies, and business units within the same enterprises that were originated in such start-ups. This also distribute the leadership, since each such start-up requires a leader with intrapreneurship abilities. The funding model is of start-ups where the enterprise is acting as VC, and the overall leaders should be able to switch jobs with VC executives, and they also need to create an atmosphere that leverages the enterprise resources where it helps, and masks the enterprise when it interrupts; The leader of each individual start-up leaders should be able to switch jobs with external start-up leaders (though inside start-up also requires some internal political skills).

Current research organizations try to have a mix of all these, the main issue is the diversity of leadership skills and how work should be organized which is different in each of them vs. the tendency of organizations to have homogeneity both in the organization type and in type of leaders.

On reasoning about events and states

Yesterday, we watched in our local theater, the play "Amadeus". I have seen the movie when it was first launched, and another version of this play in the nineties. Still very impressive, and an opportunity to hear some of Mozart's good music.

Thanks to Paul Vincent's Blog, I clicked the link and got to Paul Haley's posting entitled "time of the next generation of knowledge automation". Paul Haley starts his postings by classifying four types of reasoning:

1. Reasoning at a point within a [business] process
2. Reasoning about events that occur over time.
3. Reasoning about a [business] process (as in deciding what comes next)
4. Reasoning about and across different states (as in planning)

The claim is that the first kind is being supported by business rules (called EDM in the original), the second kind is being supported by event processing, while the third and fourth kinds are not really supported by the state of the art. The third type requires getting the notion of context into reasoning while the fourth type requires getting cross state reasoning (which is different from cross event reasoning). I agree with the classification, and will write more in the future about the third and fourth types in depth. Actually, we also need reasoning that will combine the different types of reasoning as well.