Saturday, December 19, 2009

More on the ecosystem of event processing systems

The one week school break for children ends tomorrow, and today I went with my two younger daughters to see Avatar, a very ambitious movie with a lot of advanced 3D graphics. The downside is that the film spans over 3 hours without a break -- could cut 1 hour easily, but nevertheless the graphics is very impressive, the plot is kind of paraphrase on other movies.

In my previous posting I've written about event processing functions, one of the frequently asked questions is whether event processing has a value by its own right, or as part of a larger ecosystem. Let's look at another question: DBMS software deal with update, retrieve, store, organize, backup and restore data, it also supports concurrency control, transaction management and other stuff. DBMS itself does not have value, the value is in the use of data.

In a similar way, event processing software deals with transferring, filtering, transforming, pattern matching, and situation discovering, it also provides execution control and support of some non functional properties, but the aim is to receive event process them and derive more events. The value is not in derivation of events, but in the way it is used.

In order to complete the cycle we need two other types of software: event producers that produce the raw events, and event consumers that consume the derived events and execute the actions triggered by these events -- the consumers are those who are driving the value.

The producer can be any type of software or sensors. One example of event producer I've written about in the past is the "EDA extension of CICS" that I have written about before. It is an example in which a software is instrumented to produce events. We are seeing more software of this type in the sensor area, and other areas as well.

Consumers are the part of ecosystem that utilize the events flowing from the event processing systems. I've posted before about the consumer chapter in EPIA, There are many ways to consume events, they can also be connected with other types of enterprise software: Events can drive decisions, and can serve as input to business rules; events can drive KPI (Key Performance Indicators) metrics and can be input to BAM systems, it can have various actions related to BPM systems, events can drive real-time analytics and be input to BI systems, it can post on Twitter, and update Ambient Orbs (see the posting about the EPIA chapter), or if it is part of a "smart house" it can turn off lights or control the temperature using some actuators.

Like DBMS that requires somebody to feed the data, and somebody else to use it, event processing requires its ecosystem, we'll see more of the ecosystem software both in the producer side (like the CICS example) or in the consumer side (like BAM software). More on event processing ecosystem later.

Friday, December 18, 2009

On event processing fuctions

I have been in a short vacation, and went with (some of) my family to see the film 2012, it is based on an ancient prophecy that the world as we know it will come to an end in December 21st, 2012 -- three more years to see whether this prophecy will come true.

This time I would like to write about event processing functions, I have written about them before, just summarizing it in one place.

There are various functions under the roof of event processing, some applications need all of them, but many applications need only part of them, in various level of sophistications.

Here are the major functions that I have observed:

1. Event distribution: This is the most basic one, event consumers are disseminating events through some intermediate brokers (often called channels), the events may be filtered, but are transfered without change, where any processing occur within the consumer's premises and is not part of the event processing system. Pub/sub systems are of this type, and there is a lot of work about such systems in the distributed computing area.

2. Event transformation: This goes another step and send the consumers transformed events, where the transformation may be translation, aggregation, composition, enrichment, projection and split. Aggregation is probably the most notable use of transformation, and there are many applications whose main usage of event processing is transformation.

3. Event pattern matching: This function is to find whether any subset of the input events satisfy a predefined pattern.

Note that some systems require transformation only, some require pattern matching only, some require both, systems can also have different levels of sophistication in both. It may require very simple patterns only, or sophisticated patterns; likewise it may require very simple types of transformation or much more advanced ones.

4. Situation discovery / event pattern discovery: This function is to discover that some situation occurs without having a predefined patterns, using intelligent techniques. While the first three types of functions are more investigated (although I can't say that all issues are figured out), the fourth one is still a challenge, since there are some experiments, but generally it is not well established yet.

This also remind me of a different topic -- misconceptions around event processing, and I'll write about this topic soon.

Thursday, December 17, 2009

On ecosystem for event processing

Today I woke up early, a car alarm has been activated, and was heard by the entire neighborhood, I think, except for the person who owns the car, although it parked in front of his house. I think that it lasted for 3 hours, maybe ran out of battery...

I also have to reject a lot of spam comments on my Blog recently, some of the problem was solved when I used the standard way to eliminate automatically created comments, but some still remain. Notes like: "I really like your Blog, you may be interested in my design", linking to a commercial site that can sell everything from home appliances to escort service in London.

One of the sections we added to the book, as requested by the last reviewers deals with the ecosystem of event processing with related concepts. One of the areas that we already started investigating is the relationship between event processing and business process management (AKA EDBPM), there is a recent paper on that c0-authored by Rainer von Ammon and some of his CITT colleagues, some of this work was done by Alex Kofman, my M.Sc. student (already graduated) and one of my IBM Haifa Research Lab colleagues. The paper has been linked recently from David Luckham's complexevents website. Enjoy! More on ecosystems later (I was asked to prepare something about the relationships between event processing and business intelligence for some IBM internal forum, will share it when prepared).

Tuesday, December 15, 2009

On Zamehnof and event processing

Today, December 15th is Zamenhof's day. Zamenhof is the inventor of the Esperanto language, his idea was simple: create a universal language, with very simple grammar and spelling and no spelling and grammar exceptions. While there are several millions of Esperanto fans, it has not become the universal language, and we still speak in many languages and dialects. There are street named after Zamenhof in many of the cities in Israel, not that there are so many Esperanto speakers, but somehow famous Jewish people get priority in streets names (in Haifa Zamenhof street is near Einstein street - another famous Jewish person).

Like natural languages, in event processing we have many languages, and languages styles (when preparing the course I am teaching now I realized that there are more styles than I've realized, next class we'll discuss it in class, showing examples from all the different products participating in the EPIA book's website).

I share Zamenhof's dream to get to a universal language, but this may take time (maybe infinity), worth trying to invent the event processing Esperanto.

Sunday, December 13, 2009

On event processing as a service

While working on the Website of the EPIA book, we asked the language owners to provide downloadable version of the product implementing their language. I was asked by some of the language owners if instead they can provide instead a possibility to provide their software as a service and let the readers run it on their servers. My answer was positive, and we'll see couple of such examples (one already there, one is coming up).

The book's website is just a resource for readers who wish to study languages, but this brought me to a thought about event processing as a service in general.

Some of the reasons for doing it is to gain the benefits of cloud computing in terms of scalability,
I've recently came across some material about activeinsights which seems to be a new Israeli company developing open source "event stream processing" in the cloud (well - I have some terminology comments to them, but this is not the main point) that advocate the use of event processing in the cloud to cope with scalability issues. Using SAAS model for event processing can give rise to some interesting cost models, that are either related to the input (amount of input event processed) or the output (amount of situations detected by the event processing service, with some cost per situation, or amount of aggregated/transformed derived events) which ties the cost directly to the benefit. One of the barriers of using event processing as a service is lack of standards especially for interoperability, which does not enable just to connect and run, but requires substantial investment in writing adapters in a proprietary way. I assume that we'll see more of that, when the cost/benefit model will be clarified.

There are more interactions between cloud computing and event processing, such as the use of event processing as part of the cloud infrastructure, but this deserves a separate discussion.