A prelude to the 4th event processing symposium

Last night the tentative agenda of the 4th event processing symposium was sent to the EPTS members. I have issued a call for contributions in various forums, including this Blog.

I would like to thank to all those who contributed ideas and volunteered to give a talk or participate in a panel, while I am subjective, it seems to me that we'll have very interesting program this time.

What are the goals of this meeting?

I believe that F2F meeting are vital for any community, even in the era of everything virtual, the best communications means is still F2F meeting, communities consist of people, and their success is largely dependent upon ability of people to work together, chemistry among people and communication among people. The informal atmosphere of these meetings, and the restricted number of participants, enable good interactions both in the conference room and the corridors / restaurant etc..

Besides the social aspect, the main goal of this meeting is to determine the business agenda of EPTS for the next year, mainly in thinking about work groups, their scope and intended impact.

While the business meeting will be conducted in the third day, the first two (long) days will serve as background.

Some highlights:

four invited talks:

• A VC investor who has recently invested in an EP company will tell us his perspective about the EP market (hype vs. reality)
• A customer who has experience in use EP product will tell us his perspective and view of the future.
• A standards expert will tell us about the impact of standards on communities and areas.

Each invited talk will have follow-up panel; the first panel with vendors business managers and analysts, the second panel with customers in various industries (not just finance), the third with CTOs and senior researchers and the fourth with various people involved inssion of the two existing work groups: the glossary, and the use cases. The use cases workgroup will present its template and some use cases described using this template. We'll also have a research session to vieAt the end of the second day we'll hear some topics for thought including: event processing as a service, event distribution in heterogeneous environment and academic programs/ courses relate to EP.w contributions from the research community.

At the end of the second day we'll hear some topics for thought including: event processing as a service, event distribution in heterogeneous environment and academic programs/ courses relate to EP.

Participation is by invitation only, and is intended primarily for EPTS members, however there will be some slots available to non members. If you are interested in participation please send an Email message to:<events@ep-ts.com> .

On Patagonia Dinosaurs and Disruptive Technologies

Saturday night, at home in Haifa. Today I went along with (most of) my family to visit the fossils of some Patagonia Dinosaurs . The exhibition took place in the local technology museum, and since the last time I've been there, I noticed that they have eliminated the museum's parking lot, asking the visitors to find parking in the center city -- which is not easy even in regular days, and becoming more difficult when many people somehow have the same idea that they wish to spend Saturday noon in looking at old lizards... Anyway, after two rounds I have found a good parking place, making a note that there are soon elections for the mayor of Haifa.

Anyway, looking at the fossils, among the many people who were there, I also looked at a poster explaining the various assumptions why these animals became extinct.

This has drawn me to thinking in two directions -- when will humanity become extinct, and getting back into technology -- when does disruptive technology makes previous technologies obsolete? in our case -- will we have a second (or third) generation of event processing technologies which will be disruptive for everything that exists today ? --- well, we can only speculate at this point, but it is a good topic to think about... This can be one of the topics that we'll ask the senior technologist panel in the EPTS F2F meeting. More - Later.

On optimization criteria for EP applications

This picture shows optimization of sitting on chairs, I actually know a person who sits on a big ball when he works, claiming it is good to his back. I have read with interest Paul Vincent's report on the OMG Real-Time workshop (since I cannot be everywhere, it is good that other people are reporting on what's happening, and Paul is especially good on reporting on conferences), in this meeting there has been a discussion about metrics for metrics for how to measure event processing applications. We don't have a standard benchmark yet, and I don't believe in a single benchmark fits all - but on a collection of benchmarks based on classification of applications. I would like to go deeper into the issue of "runtime performance" mentions there -- interestingly "runtime performance" means different things to different people, and indeed different application have different requirements -- if we just look at the metrics of -- latency and throughput, then we have the following variations of goal functions (this is probably not a complete list):

• min (average e2e latency)
• min (max e2e latency)
• min (variance e2e latency)
• min (deviation from time constraints)
• max (input throughput)
• max (output throughput)

The metrics are not identical - in latency there is a difference if the metrics is to minimize average latency or minimize maximal latency. For example, in Java the maximal latency can suffer from garbage collection that will make it untypically high, while "real-time Java" implementations that smooth the garbage collection minimize the maximum latency, but the price is that the average latency may grow. Throughput can be measured by input or output events, which are not really identical. Each of these goal functions indicates different kind of optimization, and this is just by looking at two parameters of throughput and latency...

This poses two interesting questions: will there be partition of the market according to optimization capabilities, or will be able to generate adaptive software that will be able to be tuned to multiple optimization ? more about performance metrics - later.

On Historic Truth, Archeological Truth and hype cycle

In one of the past blogs I have referred to Agnon an Israeli novelist who received Nobel Prize in Literature - today I am using another known essayist, Ahad Ha'am (literally: one of the people), who had an excellent essay in which he makes distinction between two types of truth: archeological truth and historical truth.

The archeologic truth is more objective, and is determined according to the archeological fidings. Historic truth, on the other hand, relates to the perception, the mythos, sometimes the legends around it, but this what is perceived regardless of what happend.

Why am I writing all of a sudden about Ahad Ha'am ? -- the association came while reading the amusing Blog entry by Paul Vincent, entitled: CEP: hype, or the next best thing since sliced bread?

In our context, the archeologic truth looks at the reality of the EP market, while the historic truth looks at the perception. Let's look about the two sides of this equation.

First, as people have realized, it is not very easy to determine the reality of the market. There is a variance between the figures cited by analysts, and each of them probably is looking for a certain segment of the market. Pure CEP players are typically privately held and don't provide public account of their sales figures, while bigger companies who do, do not isolate the sales of their CEP sotfware relative to other software. On the other hand, if the CEP software has been critical to close a larger deal, then the entire larger deal should be attributed to the CEP sotware, since deals have a binary nature... In some cases the application which the CEP software is used for is a marginal application, and in some cases it is critical mission applications, and they also should not have the same weight. Tim Bass has proposed the "reference clients" as a yardstick, by collecting public announcements and press releases, as done last year. I have not been very impressed from the results and their ability to reflect the reality, for example, when IBM acquired Aptsoft, it has been published that Aptsoft at that point had 19 active customers, most of them from 2005-2007, yet in the reference customers table they have 4 - a big difference (I heard from other vendors also that these numbers are far from reflecting their cusotmer base as they see it). It seems that the majority of sales are not reported as public refernce customers -- there are various reasons, and may be I'll return to the issue of reference customers, but the bottom line of archeology truth is not easy to obtain.

Let's talk about historic truth now - perception is easier to measure ? -- we can measure it by the VC investment in a certain area, analysts reports, big vendors attitude and customer's perception. Let's look at all these parameters:

• VC: As the EPTS gatekeeper, I am still getting membership applications from startups whose name I have never heard before. I estimate that there are 20-30 companies that are financed by VCs, and the fact that new ones are emerging is an indication that the perception of the VC community is that there is a potential in EP software.
  
• Analysts: While Gartner has endorsed this area long time ago, other analysts have joined. A recent quote from Forrester says: “Forrester Research has seen an increasing level of interest in and adoption of event technologies in our recent data on software decision makers. Based on this interest we have significantly increased our ongoing research focus in this area". Again - the perception is that this is an area that the analysts should watch.
  
• Big Companies: Big software companies, in many areas, tend to wait and let the smaller company play the first generations. In the recent year we have seen two of the big software companies - IBM and Oracle increase their involvement into the EP area both in acquisitions and self-investment. The other big software companies - SAP and Microsoft are showing signs of interest, again - this is an indication of positive perception.

• Last but not least - the customers -- we can see interest in several indications - some surveys like the ebizQ market pulse which gave insight about the perception, and I have cited it before, there are some other indications -- like the amount of customer participants that the Gartner Event Processing Summit will succeed to attract (the fact that they succeeded to do it second time is an indication); OMG is constnatly doing CEP sessions in its meetings, like the one that Paul Vincent reports about, and general interest among customers. Note, that Gartner in one of its advertisments to their summit stated that "event processing is the future of software technology".
  

Thus - on the perecption front, it seems that the indications show acceleration and growth.
Alas, one may claim that these are only indications of perception, and not to real substence, and are results of over hype. There is some truth in event processing being a current hype, however, this is a natural phenomenon, as the hype cycle theory indicate - new technologies are moving through hype up and down cycles, and EP seems to be in its way up. Hype is also a good phenomenon, since it act as a catalyst and increase awareness, however, hype cannot replace substence, and my own opinion is that all the perception indications are not a result of hype only. Actually some of them are helping to create the hype.

The interesting question is how the the substence of value to the client and the derived market opportunity is being viewed from the point of view of the people who has to put money on EP software (writing in Blogs is much easier...).

To (try and) get some answers for this question we plan in the EPTS F2F meeting ("the 4th event processing symposium") two panels: the first one will consist of people from the business side of vendors (small and large) to provide their view, the other will be a panel of customers - two populations that spend money on this - and we'll pose them some interesting questions, and summarize it as a service to the entire community. Enough for today...

On relationships among: derived event, composite event, complex event and situation

This is the water park "Luna Gal" where I am planned to spend the late afternoon with my children (it is my wife's turn to have a business trip this week) as part of a fun day for employees, meanwhile I am again at my (air-conditioned) office with the morning coffee and the Blog. One of my somewhat neglected obligations that I am trying to catch up these days are writing some terms to the encyclopedia of database systems, since we are in the glossary era, I would like to answer a FAQ question about the relationship between four terms: derived event, composite event, complex event and situation. The first three are glossary terms, the last is not, but is being used a lot in the community jargon. Tim Bass has recently posted in his Blog a simple solution model which is originated in the experimental psychology research.


The following illustration clarifies the relationships:

Derived event is an event that is created as a result of a computational derivation process as a function of one or more events.

Complex event is an event that is an abstraction of other events.

Composite event consists of collection of events that satisfy some pattern.

These are not the glossary definitions but they are explanations of the definitions that will enable to determine the relationships. As we can see not every derived event is a complex event - a derived event can be just an enrichment of a single event, and as such not a complex event, on the other hand, complex event can be created by explicit grouping of events and not by computational derivation, thus a complex event may not be a derived event. The relationships between derived events and complex event are - partial overlapping.

On the other hand, composite event is a subset of both complex event (since it is a collection of event and as such an abstraction) and derived (since it is derived by a computational process). Interestingly composite event is not the intersection between derived event and complex event, since the intersection also include scalar derived events that aggregate the raw events but don't keep the collection of events. Thus composite event is a subset of the intersection.

How does situation fit in ? - well, situation is a different beast, it is not some syntactic operation on event, but it is a semantic abstraction that denotes -- something that requires reaction. An event of any type may designate or approximate a situation, for example: fraud suspicion is the situation and a certain pattern is an indication (either exact or approximate) to the occurence of this situation, when there is a determination that a situation has occurred - a derived event may be issued to designate this situation, and the pattern is the condition (again - exact or approximate) for the situation. Note that an event pattern may have some certainty measure (e.g. probability) for the fact whether it designates a situation , not all cases are deterministic.

Does every derive event represent a situation? --- not really. A derived event can be used as an interim event or part of dissemination system and not have semantic role of situation. The same is true for all other event types.

Last but not least --- Paul Vincent in his Blog reported on a tutorial he has given about the ETPS glossary (useful presentation, I'll ask permission to borrow it). He asks a question - whether event inheritence is derived event ? well - let's check the definition of derived -- creating an event as a result of computational process as a function of one or more events. Does it describe the inheritance process ? -- IMHO it does, thus I think that inherited event is a type of derived event, though the derivation process is somewhat different from the derivation process in other cases. more - later.

On the EPTS Glossary

Today, EPTS has announced on the first version of the EPTS event processing glossary. The glossary has been edited by Roy Schulte and David Luckham, with contribution of many people throughout the community. The editors had quite a big challenge, as different vendors have been using different terminology, and this confuses the customers and sometimes owrselves. As you can see in the press release, there are support quotes from the entire community -- vendors, analysts, academic people and customers; this is an indication that the community prefers to have a common terms that everybody will use, instead on constant debates on terms.

The glossary can be downloaded from the EPTS website.

This glossary as one of the editors noted is a "living document" and will be updated periodically, since the area of event processing is still evolving. It is a starting point only, however, since it has been available in various draft forms on David Luckham's CEP portal, the glossary only received traction, and I am meeting people in various places who say that they adopted the terminology. Would I personally define all terms the way it has been defined in the glossary -- not really, and I guess this is true for many people, but since the glossary is a consolidation of various opinions and proposals, the editors have done a pretty good job of making everybody relatively happy, and the EPTS steering committee that consist of people who had initially used different terms, has unanimously approved it - as good enough to serve as first version.

What next ? - if you wish to propose to add more terms, or to modify the definition of existing terms - please use the Wiki on the EPTS website to make comments.

Citing my Irish friend, Brian Connel : friends, romans, countrymen - this is your glossary, help shape its future ! (I am the last year in Israel that has learned Shakespeare's Julius Caeser - so I can identify the source).

On messages and events

The smiling face belongs to Gregor Hohpe who is known as the co-author of the enterprise information patterns book. Gregor has conducted a BoF session in the EuroPLoP conference that I have written about in my previous Blog. The original book that Gregor has written has been about messaging oriented middleware, and now he is trying in his spare time to write the second volume - in the BoF he presented one chapter of the book talking about - identifying that something has gone wrong and handling it. This, by itself, is an interesting topic that I've been involved in the past on some related stuff, but as the title indicates the title of today's Blog is about messages and events. Hans Glide has recently dealt in his Blog in the issue of event processing vs. message processing, and part of the DEBS 2008 conference has been dedicated to messaging related research. Thus, this is a good timing to write a few lines about event and message processing --- being back home from all these travels.

Message is a transport mechanism, and as such message processing deals with the delivery of messages, relation to communication protocols, handling "not delivered messages etc..., one of the branches of messaging is also handling subscriptions when the protocol is pub/sub. The evolution of pub/sub has some similarity to that of event processing - started with subscriptions to every message published on a channel, moving to content-based subscription, which seem to be the focus of much work in the pub/sub research community. Content based subscription follows actually the ECA paradigm, where each message is treated as an event, thus, we see influence from the event world on the messaging world.

So - is event processing just fancy name to message processing ? -- not really, message processing is a lower level implementation that can serve also as infrastructure to event processing. An event can be represented by message, but not every message represents event. An event denotes something that happens, while I can send an old picture as a message and it is not an event at all. Thus the difference is in the level of architecture. Having said that, there are some interesting intersections: much of the simple and mediated event processing are shared with message processing (and indeed some of the patterns I am using to describe it are taken from Gregor's book).

The simple and mediated event processing are not new -- they indeed exist 15-20 years, and issue part of the functionality of event processing. The "detected patterns on history of multiple events" is the newer part in event processing, and it seems that it also a step in the evolution of pub/sub thinking.

Note that the pub/sub community calls itself "distributed event-based systems", and I indeed believe that they are part of the event processing community...