On Event Pattern Semantics

Today is Passover, while I am far from being religious, there are several traditions we keep, one of them is to have a family dinner in Passover-eve, and reading (at least part of) the Haggadah, so I've looked at the internet to find some fancy Haggadah in English, and here is the result.

The call for EPTS founding members
is also progressing - by now more than 20 compnies either signed or indicated that they are in internal approval process, and intend to sign as EPTS members, in addition to about 20 individual members. We excpect this number to grow towards the deadline, and call anybody who has not joined and wish to contribute to the emerging EP community to join.

Moving to today's topic: Tom Puzak has posted on the CEP interest group a message about nine features the CEP engine should have. This discussion is useful, since there is no agreed upon "CEP manifesto", a definition what are the functions that should be supported by "CEP engines", and we are going to need one, sooner or later.

Since I am working on a tutorial for the DEBS conference which will talk about event pattern semantics as a major theme, here is a sneak preview about the type of semantic decisions that are needed, this is in addition to the semantics of the specific pattern (conjunction, disjunction, absernce, sequence...).

1. In which context this particular pattern is relevant. Context can be temporal (within working hours, 1 hour from the power break), spatial (within the headquarter building), semantic (only for platinum customers or state-oriented ( while it is rainining) - or combinations of all the various dimensions (I have written before about the notion of context).

2. Is an event participate in the same pattern in a single context or in multiple contexts ? this can happen when there there is overlap among contexts.

3. Is the action / notification about the fact that the pattern has been detected should execute immediately or in a deferred mode (example: at the end of the temporal context).

4. Within a context - is the pattern existential (i.e. we are looking for a single pattern per context) or can there be multiple instances >

5. Using quantifiers on synonims - Taking the example from Tom Puzak's message: we are looking for a message of A, B within 60 secondes (temporal context), and the actual flowing events are: A1 A2 B1 A3 B2 B3 - we may want the cartesian product, but typically this is not what we really wish - thus, we can use quantifiers to select among the A and B events. Quantifiers can be according to order - firts, last, each or according to content of attributes (or both).

6. Can a single event particpate in more than one pattern within the same context ?

7. Should newer synonim kill older sysnonims ?

This are just titles - and in the DEBS tutorial I'll explain each with examples and show how they impact the pattern detection behavior.

Bottom line -- tune up the semantics of a pattern consists of several decisions, if these decisions are not supported in the language, and the application does not conform with the default, results in hacking around... more - later.

On Event Processing Agents

There are different type of agents -- double agents, as seen above which is a series of sweets, insuracne agents, travel agents, and some computerized agents - in my past I have dealt with mobile agents, and there are the intelligent agents in AI, and our own event processing agents (EPA).

David Luckham has written an amusing piece that follows Paul Vincent's "CEP and Agents" in TIBCO's Blog. The amusing part is that Paul has written about AI agents, which uses somewhat different terminology then the event processing terminology, and putting it in a "CEP Blog" is somewhat confusing. I am a "product" of the databases community, and have done some work that was on the AI border in the past, alas, the AI folks are using different terminology to talk about the same thing, and I thought at that time that they are doing it on purpose to confuse me. So, while there are many types of agents, I'll concentrate on the concept of "event processing agent" that has been coined by David Luckham. I like this term and adopted it in the following way: EPA (Event Processing Agents) is a software artifact that receives an event cloud or stream or collection of events or a single event (depends on the agent type and capabilities), does some computation on these event, and produce one or more events as an output. That's it. EPA is also a node in the EPN (Event Processing Network). There are different types of EPAs :

• "simple event processing" EPAs - filter and routing,
• "mediated event processing" EPAs - enrichment, transformation, validation
• "Complex event processing" EPAs - pattern detection
• "intelligent event processing" EPAs - prediction, decisions...

The common denominator: each of them receives events as input, emits events as output and does a single type of function.

I find this type of abstraction both very easy to explain people how EP systems work, and also basis for architecture. The EPN routing can be done by standard middleware, or in a stand-alone mode. Other terminology issues raised by David Luckham is the relationships to the "actor model" and to "engines".

The actor model is a model that helps reasoning about concurrency, while agents in AI are autonomous goal-driven artifacts. These are orthogonal terms, of course. In the context of EPA - when looking at EPAs as an executable network, we can look at each EPA as an actor and apply actor models.

Last but not least -- relationships of EPAs to engines -- an EPA is a software artifcat, it can be an instance of an engine, it can be some software that contains an engine, and it can be hard-coded program, as long as it complies with the EPA definition. In a future world, with inter-operability (and perhaps also language) standards, we'll be able to run (and maybe to self-select) multiple engines for the same EPN, residing in different EPAs.

More about EPA types -- later.

On the spectrum of event processing applications

Back in my office and reading some of the EP Blogs. In the picture above, somebody has tried to draw the spectrum of Blogs (you may want to link to the original in order to see better). One of the last Blogs has dealt again with "simple and complex event processing" claiming that everything done so far in this area is indeed "simple event processing", while real "complex event processing" should support uncertainty and backward chaining. Several posts on this area has been posted by Greg Reemler. We don't have "CEP manifesto" that makes an official definition what is CEP and what is not, and I am not sure that this will be very useful, as it will confuse the customers even more. There is a spectrum of applications that have spectrum of functional and non-functional requirements. On my scientist hat, I am partner to a research work about "uncertainty in event processing" together with Avi Gal and our co-supervised Ph.D. student Segev Wasserkrug However, while there there are applications that require uncertainty reasoning in event processing, there are many others that don't. As I have written several times before, I am not a big fan of the term "complex event processing", due to its ambiguity - some people mean complex processing of events and some mean processing of complex (derived from more than 1) events, some people actually mean complex processing of complex events. I think that we should continue to classify applicatiosn and match the right functional and non-functional requirements to the right applications, but we'll never get to a single functional or non-functional benchmark that will cover all applications in this area. It is better to attract the energy to areas that can help most customers to deal with the problems for which they would like to apply event processing. See my previous posting on : killer applications of EP

More - later.

On Semantic Event Processing

This nice castle seems European, but it is really located in Tarrytown, NY. I am not living in castles, but in a hotel in the small town of Tarrytown that has many hotels. After a busy week - starting in Pasadena, visiting Mani Chandy in Cal Tech, spending three days in IBM's Impact 2008 and visit the IBM research center in Hawthorne yesterday, and before going home this evening, I have some time for Blogging. Catching up with the recent Blogs I have found several ones that has gone semantic: Jack Rusher from Aleri, Marco from Rulecore, Paul Vincent from TIBCO all have written about "semantic CEP", with the idea to use domain dependent onthologies. The idea to use semantic relations between entities as part of event processing is something we played with a few years ago (during my previous tenure in IBM research) and I believe it has a big future, certainly when the trend is to move parts of the development to the business user, this can certainly help. So - this is certainly in our road map. Will write more on this topic later.

On Impact 2008

This is the MGM Grand Hotel and Casino in Las Vegas, the place that hosts this week the IMPACT 2008, a very big customer conference of IBM Websphere (more than 6000 persons).

The hotel is huge, the logistics is very good, and the program is very diversified. James Taylor described the two first days in his Blog . One of the main themes in this conference was the issue of business event processing, IBM's product Websphere Business Events, was announced last week. Today I have given a talk on the concept of "business event processing" together with my colleague Steve Lyons (a former Aptsoft guy), thus I have done my share. This was well covered by the press. The main message behind it is enabling the business user to control the behavior (e.g. define patterns) of EP applications, this concept won a lot of interest from the IBM clients that were present.

On another matter - congratulation to Mark Palmer for his new position in Streambase, Mark is definitly one of the notable figrue in the EP area.

More on actions as part of event processing

The recent Blog posting by Matt Rothera from Apama has argued that actions are fundemental parts of CEP, and added two reasons: avoiding latecny and providing seamless view by the user. Both of these claims are valid, the question is that makes the "action" part - an integral part of event ptoceding or not. What I stated in a past posting is that there are two types of actions - those which are done internally inside the "event processing network", and those that are done in the outside universe, and those, architectually are part of consumers and not of the event processing network. This is still valid, if the action is a "buy" or "sell" it is executed inside some trading system, and not in the event processing system, the event processing system can, at most, make a decision (or recommendation, depending on the type of application). This leads to an observation on packaging - if the event processing product is a "stand-alone" then it probably needs additional capabilities that are not pure event processing in nature, however, then the integration effort of these products may be high in some cases. I also stated in a a past posting that event processing is part of a bigger picture, thus, it needs to be nicely integrated inside a bigger whole - e.g. an application integration middleware from the run-time aspects, and then the latency issues are solved, since everything is using the same infrastructure, and QoS properties can apply to the entire application. It should also be nicely integrated from modeling, tooling, and interface aspects, thus a customer will be able to express E2E application.

My main claim is that in many cases - event processing capture part but not whole of an applicaton, and thus we need to look anyway on the bigger picture anyway, however, it is not realistic to develop event processing products into an entire middleware...

Call for EPTS founding members

After some delays (getting everything through legal procedures in multiple big companies...) we are starting to prepare for the official launch of EPTS that has existed informally in the last couple of years. The founding steering committee consists of representatives of -

Coral8, Gartner, IBM, Oracle, Progress, Streambase, TIBCO and Professor David Luckham as an individual member, in the future members of the steering committee will be elected, so both company members and individual members can be elected; We are now issuing a "call for founding members", which means that any company, or individual person, that will sign the membership agreement by May 9, will be included in the list of founding members that will be mentioned in the press release notifying the launch. Here is the call - you may have already received it through other channel.

Call for EPTS Founding Members

EPTS - the Event Processing Technical Society - is going to be officially launched in late May or early June (exact date – TBD)
The EPTS goals are to:
Increase awareness (promote mindshare) about event processing, including understanding what its value is, producing a common glossary, and highlighting the current state of best practices,
Collaborate with various standards organizations making sure all standard efforts are in sync (EPTS will not become a standards organization)
Help establish Event Processing as an academic discipline
EPTS has existed as an informal group for the last two years, holding three Event Processing Symposiums to date; for details see: meeting 1, meeting 2, meeting 3

We invite participation of all vendors, individual participants (academic people, independent consultants, and employees of non-member organizations), and customers who would like to contribute to the evolution of the event processing area. There is no participation fee, but all members (organizational and individual members) must sign the EPTS Members Agreement.
EPTS activities will be performed through workgroups, and periodic general meetings. Examples of current workgroup tasks include glossary creation, use cases analysis, event processing meta-modeling, the study of event formats, and event processing academic education.
Benefits to members include the ability to identify new requirements or topics for discussion, lead or participate in workgroups and other community activities, being listed and potentially quoted in official announcements and press releases, and the ability to be elected to the EPTS Steering Committee,
Please respond immediately by email to Opher Etzion

If you are interested in becoming a founding member.
If you are able to sign the membership agreement by May 9th, 2008, you'll have the opportunity to appear in the list of founding members in the launch press release.
In order to join, please take the following actions:
1. Print the EPTS Member Agreement document

2. Sign (for individual member) or have an authorized representative of your company sign (for a company member) two hardcopies of the agreement. Keep one for your files. You will need to produce this copy if the EPTS Steering Committee ever makes a request for it at some point in the future.
3. Fax one copy to:
Attention: Mike Kaiser
Fax #: 1-845-489-9958
4. Courier the second hard copy signature to the following address for EPTS archival purposes:
IBM Corporation
Attn: Mike Kaiser (JGWA/062/M313)
3039 Cornwallis Rd.
Research Triangle Park, NC 27709
Phone: 919-254-7605
Note: Along with this hardcopy, please include the name, e-mail and phone contact information of the PR person that EPTS should coordinate with for the launch press release.