2009 - event processing perspectives

2009 is going away soon, and it is time to summarize it from the event processing perspective, which is the focus of this Blog.

According to analysts this has been a good year for event processing, in tough economic climate, the accumulative market for event processing continued to grow, more or less according to the original predictions, and is expected to continue the substantial growth. Here are ten statements about event processing in 2009.

1. In the vendors world, Microsoft has announced a forthcoming product, Software AG notified that it is working on a product, and more start-ups have joined the area; the most notable acquisition this year is the acquisition of Coral8 by Aleri that was not an intuitive acquisition.
2. In my own company, IBM -- besides Websphere Business Events (WBE) that was launched in 2008 and is growing rapidly in 2009 in number of customers, IBM announced three more products in this area in 2009: Infosphere Streams, Websphere Sensor Events, and EDA extension for CICS, as IBM believes in having event processing capabilities pervasive throughout its software portfolio
3. The emergence of new book. The first book in this area that has made a big impact was David Luckham's "Power of events". Eight year have passed with Luckham's book as a single book in this area. In 2009 several more books have been published, most notable the book by Chandy and Schulte. Some more books are due in 2010
4. Some popular magazines ran articles about event processing, one of them is International Journal of Banking Systems, and the other is IEEE Computer.
5. All major analysts had special reports on event processing. Gartner has written about it before, but now made it explicit part of its "hype cycle"; Forrester made a thorough report with comparison among several products over multiple criteria.
6. The major scientific conference of event processing DEBS has been endorsed by ACM and became ACM DEBS conference. The conference made a shift over the last couple of years from "pub/sub" conference to a larger event processing conference. EPTS provided two tutorials: languages and use cases
7. Other event processing related workshops interacting event processing with other areas were: event-driven business process management, or event-based processing in robotics. These two topics have been discussed in the annual EPTS meeting that was held in Trento.
8. it is announced that Streambase will receive the "world economic foundation" award, an indication that event processing is considered as one of the influential technologies for the world economy.
9. Another winner of the same award is Twitter. This year different applications that processing Twitter events have emerged.
10. The quote of the year comes from Alex Buchmann, in his Keynote address in DEBS 2010: regular programming is like drinking with a straw, this is good when the data is standing, while the data is moving, like in event processing, using the same kind of thinking is similar to use a straw to drink from a waterfall.

Something about what's coming in 2010 -- later.

On common misconceptions about event processing - the complexity misconception

David Luckham has coined the term "complex event processing", this term has caught as the marketing term behind many of the vendors that provide event processing platforms (comment: IBM, and recently Progress/Apama moved to use the term "business event processing"). While this term succeeded to get traction, it also is a source of on of the common misconceptions, Luckham talked about complex events, and their processing, some people understand it as the complex processing of events, and some just view it as the intersection between event processing and "complex systems". Complex event is defined as "abstraction of one or more other events", which also leads to some interpretations about the nature of abstractions, so this interpretation is easier to understand. However, the misconception is that it is more intuitive to think about "complex event processing" in the second interpretation as "complex" processing of events, and this brings us to the question -- what is complexity? there can be different dimensions of complexity.

1. The complexity may stem from the fact that we don't know what exactly we are looking for, and generally looking for anomalies in the system (e.g. trying to find security violations), thus some AI techniques have to be applied here.
2. Another case is that we know what are the patterns or aggregations we are using, but they require complex computing, in term of functional capabilities.
3. Another case is that the complexity is in some non-functional requirement: such as scalability in several direction (scale-up or scale-down), strict real-time performance constraints, highly distributed system etc..
4. Another case of complexity is in interoperability, the need to obtain events from many producers, and use events in many consumers, which requires instrumentation/modification of a lot of legacy systems.
5. Yet another case of complexity may be unreliable event sources, handling false positives and false negatives.

There are probably more complexity cases, however the interesting question is whether the main goal of an event processing system is to solve a "complex" problem.

Om my scientist hat, it is definitely more exciting to solve complex problems, even better, problems of the type that have never seen before. However, from pragmatic point of view, event processing applications are measured on their business value, and there might be a lot of business value of using event processing to systems that have none of these complexity measures, from complexity point of view they can be quite simple, moreover, there may not be an exciting aspect about the implementation as it is similar to other implementations already done, but on the measurement of "business value" it brings a lot of value, thus the value metric is orthogonal to any complexity metric, and indeed many of the applications in which event processing technology is very useful to is quite simple (according to one of the analysts report the "simple" applications are 80-90% of the potential market for the event processing technology). While there is certainly a segment of application for each type of complexity, and more work is required in these direction, the "simple" application will be the bread and butter.

More misconceptions - later.

On common misconceptions about event processing - the single application misconception

We start the introduction chapter for the EPIA book, by stating: Some people say that event processing is the next big thing; some people say that event processing is old hat and there is nothing really new in it. Both groups may be right to a certain extent. As with any field that is relatively new there is some fog around it: some of the fog stems from misconceptions, some from confusing messages by vendors and analysts, and some arises because of a lack of standards, a lack of agreement on terms, and a lack of understanding about some of the basic issues.

In the book we don't really talk about the misconceptions, but I think it is a good topic towards the end of 2009 to dedicate some postings towards the major misconceptions.

I'll start with misconception number 1: Event processing is a single-industry (some even say single-application) technology, and event processing software cannot generalize beyond this single industry/application.

The industry is, of course, capital markets, and the application is algorithmic trading

The diagram below is taken from the ebizQ customers survey (two years ago) about what are the business problems that they expect to solve with event processing, and the result is 9% indicated algorithmic trading.

This misconception is originated from the fact that the capital market industry has indeed been the early adopter of event processing software, and served as a proof of concept for the rest of the industries, there are indeed some vendors that focus mainly around this type of application, however, this does not show the entire picture. From the IBM experience I know of customers in various industries, most are not in the capital market area. Getting to the material collected by the EPTS use case work group (the material is on the EPTS members internal site, available to EPTS members) I find quite a lot of examples of systems working in production or being developed from variety of domains, here are some samples:

• Border security radiation detection (Eventzero)
• Mobile asset geofence (Rulecore)
• Logistic and scheduling application (Starview)
• Unauthorized use of heavy machinery (Rulecore)
• Hospital patient and asset tracking (IBM)
• Activity monitoring for taxing and fraud detection (IBM)
• Intelligent CRM in banking (TIBCO)
• EDA and asynchronous BPM in retail (TIBCO)
• Situation awareness in energy utilities (TIBCO)
• Situation awareness in airlines (TIBCO)
• Reduce cost in injection therapy (IBM)
• Next generation navigation (CITT)
• Real-time management of hazardous materials (Oracle)
• Finding anomalies in point of sales in retail stores (CA)
• Elderly behavior monitoring (U. of Munich)

These are only samples, I am familiar with variety of examples in various industries: healthcare, utilities, chemical and petroleum, insurance, security, transportation and others. In the last event processing symposium in Trento, we had one keynote address on event processing in robotics, and there are other areas as well such as smart house to monitor energy consumption in the house. While we are now in the first generation, and the utilization of event processing will increase in time, the coverage in terms of industries and applications is growing has gone far beyond algorithmic trading.

More misconceptions in subsequent postings.

More on the ecosystem of event processing systems

The one week school break for children ends tomorrow, and today I went with my two younger daughters to see Avatar, a very ambitious movie with a lot of advanced 3D graphics. The downside is that the film spans over 3 hours without a break -- could cut 1 hour easily, but nevertheless the graphics is very impressive, the plot is kind of paraphrase on other movies.

In my previous posting I've written about event processing functions, one of the frequently asked questions is whether event processing has a value by its own right, or as part of a larger ecosystem. Let's look at another question: DBMS software deal with update, retrieve, store, organize, backup and restore data, it also supports concurrency control, transaction management and other stuff. DBMS itself does not have value, the value is in the use of data.

In a similar way, event processing software deals with transferring, filtering, transforming, pattern matching, and situation discovering, it also provides execution control and support of some non functional properties, but the aim is to receive event process them and derive more events. The value is not in derivation of events, but in the way it is used.

In order to complete the cycle we need two other types of software: event producers that produce the raw events, and event consumers that consume the derived events and execute the actions triggered by these events -- the consumers are those who are driving the value.

The producer can be any type of software or sensors. One example of event producer I've written about in the past is the "EDA extension of CICS" that I have written about before. It is an example in which a software is instrumented to produce events. We are seeing more software of this type in the sensor area, and other areas as well.

Consumers are the part of ecosystem that utilize the events flowing from the event processing systems. I've posted before about the consumer chapter in EPIA, There are many ways to consume events, they can also be connected with other types of enterprise software: Events can drive decisions, and can serve as input to business rules; events can drive KPI (Key Performance Indicators) metrics and can be input to BAM systems, it can have various actions related to BPM systems, events can drive real-time analytics and be input to BI systems, it can post on Twitter, and update Ambient Orbs (see the posting about the EPIA chapter), or if it is part of a "smart house" it can turn off lights or control the temperature using some actuators.

Like DBMS that requires somebody to feed the data, and somebody else to use it, event processing requires its ecosystem, we'll see more of the ecosystem software both in the producer side (like the CICS example) or in the consumer side (like BAM software). More on event processing ecosystem later.

On event processing fuctions

I have been in a short vacation, and went with (some of) my family to see the film 2012, it is based on an ancient prophecy that the world as we know it will come to an end in December 21st, 2012 -- three more years to see whether this prophecy will come true.

This time I would like to write about event processing functions, I have written about them before, just summarizing it in one place.

There are various functions under the roof of event processing, some applications need all of them, but many applications need only part of them, in various level of sophistications.

Here are the major functions that I have observed:

1. Event distribution: This is the most basic one, event consumers are disseminating events through some intermediate brokers (often called channels), the events may be filtered, but are transfered without change, where any processing occur within the consumer's premises and is not part of the event processing system. Pub/sub systems are of this type, and there is a lot of work about such systems in the distributed computing area.

2. Event transformation: This goes another step and send the consumers transformed events, where the transformation may be translation, aggregation, composition, enrichment, projection and split. Aggregation is probably the most notable use of transformation, and there are many applications whose main usage of event processing is transformation.

3. Event pattern matching: This function is to find whether any subset of the input events satisfy a predefined pattern.

Note that some systems require transformation only, some require pattern matching only, some require both, systems can also have different levels of sophistication in both. It may require very simple patterns only, or sophisticated patterns; likewise it may require very simple types of transformation or much more advanced ones.

4. Situation discovery / event pattern discovery: This function is to discover that some situation occurs without having a predefined patterns, using intelligent techniques. While the first three types of functions are more investigated (although I can't say that all issues are figured out), the fourth one is still a challenge, since there are some experiments, but generally it is not well established yet.

This also remind me of a different topic -- misconceptions around event processing, and I'll write about this topic soon.

On ecosystem for event processing

Today I woke up early, a car alarm has been activated, and was heard by the entire neighborhood, I think, except for the person who owns the car, although it parked in front of his house. I think that it lasted for 3 hours, maybe ran out of battery...

I also have to reject a lot of spam comments on my Blog recently, some of the problem was solved when I used the standard way to eliminate automatically created comments, but some still remain. Notes like: "I really like your Blog, you may be interested in my design", linking to a commercial site that can sell everything from home appliances to escort service in London.

One of the sections we added to the book, as requested by the last reviewers deals with the ecosystem of event processing with related concepts. One of the areas that we already started investigating is the relationship between event processing and business process management (AKA EDBPM), there is a recent paper on that c0-authored by Rainer von Ammon and some of his CITT colleagues, some of this work was done by Alex Kofman, my M.Sc. student (already graduated) and one of my IBM Haifa Research Lab colleagues. The paper has been linked recently from David Luckham's complexevents website. Enjoy! More on ecosystems later (I was asked to prepare something about the relationships between event processing and business intelligence for some IBM internal forum, will share it when prepared).

On Zamehnof and event processing

Today, December 15th is Zamenhof's day. Zamenhof is the inventor of the Esperanto language, his idea was simple: create a universal language, with very simple grammar and spelling and no spelling and grammar exceptions. While there are several millions of Esperanto fans, it has not become the universal language, and we still speak in many languages and dialects. There are street named after Zamenhof in many of the cities in Israel, not that there are so many Esperanto speakers, but somehow famous Jewish people get priority in streets names (in Haifa Zamenhof street is near Einstein street - another famous Jewish person).

Like natural languages, in event processing we have many languages, and languages styles (when preparing the course I am teaching now I realized that there are more styles than I've realized, next class we'll discuss it in class, showing examples from all the different products participating in the EPIA book's website).

I share Zamenhof's dream to get to a universal language, but this may take time (maybe infinity), worth trying to invent the event processing Esperanto.