On Historic Truth, Archeological Truth and hype cycle

In one of the past blogs I have referred to Agnon an Israeli novelist who received Nobel Prize in Literature - today I am using another known essayist, Ahad Ha'am (literally: one of the people), who had an excellent essay in which he makes distinction between two types of truth: archeological truth and historical truth.

The archeologic truth is more objective, and is determined according to the archeological fidings. Historic truth, on the other hand, relates to the perception, the mythos, sometimes the legends around it, but this what is perceived regardless of what happend.

Why am I writing all of a sudden about Ahad Ha'am ? -- the association came while reading the amusing Blog entry by Paul Vincent, entitled: CEP: hype, or the next best thing since sliced bread?

In our context, the archeologic truth looks at the reality of the EP market, while the historic truth looks at the perception. Let's look about the two sides of this equation.

First, as people have realized, it is not very easy to determine the reality of the market. There is a variance between the figures cited by analysts, and each of them probably is looking for a certain segment of the market. Pure CEP players are typically privately held and don't provide public account of their sales figures, while bigger companies who do, do not isolate the sales of their CEP sotfware relative to other software. On the other hand, if the CEP software has been critical to close a larger deal, then the entire larger deal should be attributed to the CEP sotware, since deals have a binary nature... In some cases the application which the CEP software is used for is a marginal application, and in some cases it is critical mission applications, and they also should not have the same weight. Tim Bass has proposed the "reference clients" as a yardstick, by collecting public announcements and press releases, as done last year. I have not been very impressed from the results and their ability to reflect the reality, for example, when IBM acquired Aptsoft, it has been published that Aptsoft at that point had 19 active customers, most of them from 2005-2007, yet in the reference customers table they have 4 - a big difference (I heard from other vendors also that these numbers are far from reflecting their cusotmer base as they see it). It seems that the majority of sales are not reported as public refernce customers -- there are various reasons, and may be I'll return to the issue of reference customers, but the bottom line of archeology truth is not easy to obtain.

Let's talk about historic truth now - perception is easier to measure ? -- we can measure it by the VC investment in a certain area, analysts reports, big vendors attitude and customer's perception. Let's look at all these parameters:

• VC: As the EPTS gatekeeper, I am still getting membership applications from startups whose name I have never heard before. I estimate that there are 20-30 companies that are financed by VCs, and the fact that new ones are emerging is an indication that the perception of the VC community is that there is a potential in EP software.
  
• Analysts: While Gartner has endorsed this area long time ago, other analysts have joined. A recent quote from Forrester says: “Forrester Research has seen an increasing level of interest in and adoption of event technologies in our recent data on software decision makers. Based on this interest we have significantly increased our ongoing research focus in this area". Again - the perception is that this is an area that the analysts should watch.
  
• Big Companies: Big software companies, in many areas, tend to wait and let the smaller company play the first generations. In the recent year we have seen two of the big software companies - IBM and Oracle increase their involvement into the EP area both in acquisitions and self-investment. The other big software companies - SAP and Microsoft are showing signs of interest, again - this is an indication of positive perception.

• Last but not least - the customers -- we can see interest in several indications - some surveys like the ebizQ market pulse which gave insight about the perception, and I have cited it before, there are some other indications -- like the amount of customer participants that the Gartner Event Processing Summit will succeed to attract (the fact that they succeeded to do it second time is an indication); OMG is constnatly doing CEP sessions in its meetings, like the one that Paul Vincent reports about, and general interest among customers. Note, that Gartner in one of its advertisments to their summit stated that "event processing is the future of software technology".
  

Thus - on the perecption front, it seems that the indications show acceleration and growth.
Alas, one may claim that these are only indications of perception, and not to real substence, and are results of over hype. There is some truth in event processing being a current hype, however, this is a natural phenomenon, as the hype cycle theory indicate - new technologies are moving through hype up and down cycles, and EP seems to be in its way up. Hype is also a good phenomenon, since it act as a catalyst and increase awareness, however, hype cannot replace substence, and my own opinion is that all the perception indications are not a result of hype only. Actually some of them are helping to create the hype.

The interesting question is how the the substence of value to the client and the derived market opportunity is being viewed from the point of view of the people who has to put money on EP software (writing in Blogs is much easier...).

To (try and) get some answers for this question we plan in the EPTS F2F meeting ("the 4th event processing symposium") two panels: the first one will consist of people from the business side of vendors (small and large) to provide their view, the other will be a panel of customers - two populations that spend money on this - and we'll pose them some interesting questions, and summarize it as a service to the entire community. Enough for today...

On relationships among: derived event, composite event, complex event and situation

This is the water park "Luna Gal" where I am planned to spend the late afternoon with my children (it is my wife's turn to have a business trip this week) as part of a fun day for employees, meanwhile I am again at my (air-conditioned) office with the morning coffee and the Blog. One of my somewhat neglected obligations that I am trying to catch up these days are writing some terms to the encyclopedia of database systems, since we are in the glossary era, I would like to answer a FAQ question about the relationship between four terms: derived event, composite event, complex event and situation. The first three are glossary terms, the last is not, but is being used a lot in the community jargon. Tim Bass has recently posted in his Blog a simple solution model which is originated in the experimental psychology research.


The following illustration clarifies the relationships:

Derived event is an event that is created as a result of a computational derivation process as a function of one or more events.

Complex event is an event that is an abstraction of other events.

Composite event consists of collection of events that satisfy some pattern.

These are not the glossary definitions but they are explanations of the definitions that will enable to determine the relationships. As we can see not every derived event is a complex event - a derived event can be just an enrichment of a single event, and as such not a complex event, on the other hand, complex event can be created by explicit grouping of events and not by computational derivation, thus a complex event may not be a derived event. The relationships between derived events and complex event are - partial overlapping.

On the other hand, composite event is a subset of both complex event (since it is a collection of event and as such an abstraction) and derived (since it is derived by a computational process). Interestingly composite event is not the intersection between derived event and complex event, since the intersection also include scalar derived events that aggregate the raw events but don't keep the collection of events. Thus composite event is a subset of the intersection.

How does situation fit in ? - well, situation is a different beast, it is not some syntactic operation on event, but it is a semantic abstraction that denotes -- something that requires reaction. An event of any type may designate or approximate a situation, for example: fraud suspicion is the situation and a certain pattern is an indication (either exact or approximate) to the occurence of this situation, when there is a determination that a situation has occurred - a derived event may be issued to designate this situation, and the pattern is the condition (again - exact or approximate) for the situation. Note that an event pattern may have some certainty measure (e.g. probability) for the fact whether it designates a situation , not all cases are deterministic.

Does every derive event represent a situation? --- not really. A derived event can be used as an interim event or part of dissemination system and not have semantic role of situation. The same is true for all other event types.

Last but not least --- Paul Vincent in his Blog reported on a tutorial he has given about the ETPS glossary (useful presentation, I'll ask permission to borrow it). He asks a question - whether event inheritence is derived event ? well - let's check the definition of derived -- creating an event as a result of computational process as a function of one or more events. Does it describe the inheritance process ? -- IMHO it does, thus I think that inherited event is a type of derived event, though the derivation process is somewhat different from the derivation process in other cases. more - later.

On the EPTS Glossary

Today, EPTS has announced on the first version of the EPTS event processing glossary. The glossary has been edited by Roy Schulte and David Luckham, with contribution of many people throughout the community. The editors had quite a big challenge, as different vendors have been using different terminology, and this confuses the customers and sometimes owrselves. As you can see in the press release, there are support quotes from the entire community -- vendors, analysts, academic people and customers; this is an indication that the community prefers to have a common terms that everybody will use, instead on constant debates on terms.

The glossary can be downloaded from the EPTS website.

This glossary as one of the editors noted is a "living document" and will be updated periodically, since the area of event processing is still evolving. It is a starting point only, however, since it has been available in various draft forms on David Luckham's CEP portal, the glossary only received traction, and I am meeting people in various places who say that they adopted the terminology. Would I personally define all terms the way it has been defined in the glossary -- not really, and I guess this is true for many people, but since the glossary is a consolidation of various opinions and proposals, the editors have done a pretty good job of making everybody relatively happy, and the EPTS steering committee that consist of people who had initially used different terms, has unanimously approved it - as good enough to serve as first version.

What next ? - if you wish to propose to add more terms, or to modify the definition of existing terms - please use the Wiki on the EPTS website to make comments.

Citing my Irish friend, Brian Connel : friends, romans, countrymen - this is your glossary, help shape its future ! (I am the last year in Israel that has learned Shakespeare's Julius Caeser - so I can identify the source).

On messages and events

The smiling face belongs to Gregor Hohpe who is known as the co-author of the enterprise information patterns book. Gregor has conducted a BoF session in the EuroPLoP conference that I have written about in my previous Blog. The original book that Gregor has written has been about messaging oriented middleware, and now he is trying in his spare time to write the second volume - in the BoF he presented one chapter of the book talking about - identifying that something has gone wrong and handling it. This, by itself, is an interesting topic that I've been involved in the past on some related stuff, but as the title indicates the title of today's Blog is about messages and events. Hans Glide has recently dealt in his Blog in the issue of event processing vs. message processing, and part of the DEBS 2008 conference has been dedicated to messaging related research. Thus, this is a good timing to write a few lines about event and message processing --- being back home from all these travels.

Message is a transport mechanism, and as such message processing deals with the delivery of messages, relation to communication protocols, handling "not delivered messages etc..., one of the branches of messaging is also handling subscriptions when the protocol is pub/sub. The evolution of pub/sub has some similarity to that of event processing - started with subscriptions to every message published on a channel, moving to content-based subscription, which seem to be the focus of much work in the pub/sub research community. Content based subscription follows actually the ECA paradigm, where each message is treated as an event, thus, we see influence from the event world on the messaging world.

So - is event processing just fancy name to message processing ? -- not really, message processing is a lower level implementation that can serve also as infrastructure to event processing. An event can be represented by message, but not every message represents event. An event denotes something that happens, while I can send an old picture as a message and it is not an event at all. Thus the difference is in the level of architecture. Having said that, there are some interesting intersections: much of the simple and mediated event processing are shared with message processing (and indeed some of the patterns I am using to describe it are taken from Gregor's book).

The simple and mediated event processing are not new -- they indeed exist 15-20 years, and issue part of the functionality of event processing. The "detected patterns on history of multiple events" is the newer part in event processing, and it seems that it also a step in the evolution of pub/sub thinking.

Note that the pub/sub community calls itself "distributed event-based systems", and I indeed believe that they are part of the event processing community...

On EuroPLoP and Event Processing Patterns

This is Kloster Irsee in Germany, where the EuroPLoP conference is held. The conference itself is held in kind of an opposite way to other conferences, in a regular conference, the author of a paper presents, in this conference, the author of a paper hears other criticizing his paper, and does not talk. It also have some other properties -- some of the sessions are held in room which the chairs have been taken out, so people are sitting on the floor, or standing, there are also games embedded in the program --- my daughter would have said if she was here that there is a bunch of nerd children trying to behave in a cool fashion - well... anyway, we came here to combine work on creating a consortium for EU project and having a three hours "focused group" inside the conference, organized and moderated by Adrian Paschke. The audience was mixed - some were the EP gang, those who came to work on the EU project, and some where regular participants in the conference. The various interpretations of patterns related to EP have been discussed, and the need to advance in all of them. This is directly related to my tutorial in DEBS 2008, there is now work in progress on a pattern meta-language that we are doing in IBM, in a few weeks I'll tell you more about it - stay tuned.
More about my tutorial in DEBS 2008 - Paul Vincent from TIBCO has taken a picture of a slide that I've presented with different computer (due to some unclear technical problem) with probably different character set (Paul also blogged about this slide) - in footnote 1. Paul has been kind enough to send me the picture - so here it is, you may see some unknown characters...

More about patterns - soon.

On the multiple types of patterns in event processing

The red arrow in the map shows where I am now - in Kaufbeuren, a town in the south-west corner on Bavaria. I have arrived last night by train, and it has been a challenge to find a place to eat in 9PM, but with a few minutes of search I have found it. I am on the way to the patterns conference, in which we have a working group to establish a partnership towards EU project proposal. Thus, I would like to write a few lines about patterns. In my DEBS 2008 tutorial I have used the following slide:

In this slide there are four possible meanings for patterns in event processing:

(1). Patterns in the sense of functions that event processing application may perform (e.g. enrich, route, transform, filter, detect pattern).

(2). Patterns that are detected on the history of events (note that "detect pattern" is just one of the patterns of the first type, but the support of this pattern is what makes it CEP).

(3). Patterns in the user's domain - the way things are presented to the user (which may not be translated 1-1 to an implementation pattern)

(4). Patterns in the software engineering sense - best practices of how to use some language / product / technology.

All of the four are interesting and also have some dependencies, my tutorial has concentrated mainly on patterns of type (2), I have written about type (1) a lot in the past -- types (3) and (4) are frontiers that need to be conquered. More - Later.

On formal models and agents

This is an aerial picture of Karlsruhe, in Germany. It is morning, I have the habbit to get up early, so have some time for Email and Blogging (this hotel does have internet connection in the room!). Here I am a guest of the FZI team that is trying to apply semantic models to event processing, which is one of my recent occupations. One of the property of researchers everywhere (and IBM is not different) that they are constantly looking for research funding - since one of the available sources in this region is EU projects, researchers from the relevant countries (and Israel is part of the EU, for the sake of the R&D programs), so there is a need to join forces. The FZI group is looking at applying transaction logic as a formal model for both validation and run-time control of event processing applications. I am in favor of having formal model behind event processing to define precisely the semantics. A model that will succeed should also be relatively simple. I was asked whether I see a possibility to have a single formal model that inclues all variations of event processsing - and my answer is that my assumption is that this is possible.

Dealing with formal models - I have been asked by several people about the choice I've made in my DEBS 20008 tutorial to define "event processing agent" as an atomic unit of processing.

First - I agree that the term agent is overloaded, like many other terms -- event processing agent is not (necessarily) an intellignet agent, it is more a software agents. The glossary defines agent as a software module that processes events. I have making a distinction between conceptual agent and run-tine agent - the run-time agent may consist of multiple conceptual agents, and the decision is up to self-optimization process (will write about optimization issues in one of the next blogs). It is easier to reason (and optimize) when the abstraction deals with atomic elements. Whether should be called - "atomic agent" - may be. I prefer shorter names, and since this does not contradict the original definition. more - later.