More on the layered approach of the event-driven world

I have been asked by several people to write in more detail about what I meant  by the layered approach, so I thought it should be better illustrated in an architecture diagram (converting it to picture did not result in high quality, somehow).  The rationale behind it that event processing became pervasive in use, with both event processing products, and "build your own solution".    In the same way that application servers based on standards like J2EE contributed to building of certain type of web-based enterprise applications,  an application server for event-based applications is required to provide services from various types: from context service, adapters to sensor and mobile platforms, dashboard, management services, meta-data service and more.   The second layer is the agent layer which provide both directory of agents and tools to build your own agent.  The application is the third layer.   This architecture can provide independence, and ability to get best of breed in both services and functionality components from different sources, and combinations of "build" and "buy".

Standards are of course the key to make it happen.  

On virtual vs. real in event processing

As a follow on to my last posting on virtualization of event processing, one of the frequently asked questions, and maybe somewhat open question can we get really to platform independent models that can be directly compiled to implementation of our choice in fully automatic way. This is one of the major topics I am investigating recently. Two issues are involved: Is it possible to automatically compile to a specific platform that may be semantically incompatible with the platform independent model, the second issue is whether we can create an efficient implementation without taking more platform specific considerations. The answer lies within the compilation process from the platform independent model into the specific platform. This was not really done in event processing yet, but problems of the same magnitude have been handled in other areas, and we intend to leverage the experience of people who have done it in order to see how far this can go. As a related issue, Jim Odell continued his series on event processing and agent technology, hosted in the TIBCO CEP BLOG .

Since the platform independent model we are using has "event processing agents" as its major building block, agent oriented implementation is the most natural one, and the compilation to such a model will be straightforward, however, the platform independent model should also be compilable to centralized implementations, where the agents are all hidden inside a single run-time artifact. My guess is that the shift to agent oriented implementations will happen gradually, I'll talk about it when I'll present my view on "event processing - seven years from now" in the OMG event processing consortium meeting next month. I'll write more in the future on the compilation from model to implementation as we'll get progress in that project,

More on event processing agents

There are various agents types of agents in reality, like the one shown above, likewise there are various agents types in computing, as I've discussed long time ago in this Blog, but since that time the thinking was somewhat evolved. Recently, Jim Odell, a long time agents advocate, has been hosted by the TIBCO CEP Blog, and advocated the use of agent technology in event processing, providing scalability as main motivation. In the Event Processing in Action book we are making EPA (Event Processing Agent), a term coined by David Luckham, as the most notable building block in our model. The term agent is used as software agent, and not necessarily agent in the AI sense. We are using the term agent as a meta level, where in the run-time level there are agent instances that can be implemented in various ways. Event Processing Agents in event processing are event-driven in the sense that they are taking one or more events as input, perform some processing on these events, and derive one or more events.

EPA can filter events, transform events, detect event patterns or do any combination of the above. Event processing agents are typically (but not always) associated with context, thus context related operations determine when an EPA instance is open or close, assume that the context is a temporal sliding window of non overlapping periods of 2 hours, and there is an EPA associated with this context, then every 2 hours, an instance of this EPA terminates and another instance is initiated. In essence EPA interacts with events in various ways:

• An EPA receive events as input
• An EPA processes events
• An EPA may query historical events
• An EPA derives events as output
• Through context -- an EPA instance may be initiated or terminated by events

Each EPA instance is autonomous in the sense that it does not communicate with any other EPA instance, and thus can be implemented by different run-time artifact, which indeed can enable scalability. However, there can be various grouping of EPA instances to run-time artifacts, where the two extremes are: run-time artifact for every EPA instance, and a single run-time artifact that contains all the EPA instances within the application.

Benefits of using EPA are -- simplicity of the model, modularity, and as said flexibility in implementation that may support various scalability and performance objective functions. I'll write more about EPA - later.

More on the nodes in the event processing network

This is the illustration of the event processing network for the Fast Flower Delivery example that accompanies the EPIA book we are writing, don't spoil your eyes reading it. We'll provide the readers with an open source editor to the EPN meta-language we have developed (developed by Ronen Vaisenberg, a Ph.D. student in UCI). I'll have posting about it with more instructions in one of the next postings.

In this posting I would like to go back to the nodes in EPN and discuss why do we need multiple nodes and what are these nodes. One version is that there is a single type of node called EPA -- event processing agent, but my preference is to have multiple nodes, since they have different roles. Here are the types of nodes in our EPN model:

• Event Processing Agent (EPA): a node that gets one or more events as an input and produces one or more event as an output.
• Event producer: An entry point to the event processing system that emits events to the EPN.
• Event consumer: An exit point from the event processing system that receives events from the event processing system
• Event channel: A node that make routing decisions within the EPN, I have recently discussed event channels in this Blog.
• Global states: This is a node that holds various type of state (reference data, global variables)
• EPN node: An EPA can represent an EPN, and thus make the EPN recursive.
  

One of the frequent questions is whether an event producer which is also an event consumer is considered an EPA. In my opinion, the answer is no. Event producers and consumers operate on the borderline of event processing network, and is actually have have two nodes one for each of its roles. The information that a consumer is also a producer is important for traceability and lineage, but we'll discuss this aspect another time.

Christoph Emmesberger, whom I met in Trento, has proposed to extend the notion of channel to provide aggregation and filtering, this idea is rooted in pub/sub systems where event processing seems as an extension of some event processing functionality. However, when we deal with event processing system, I prefer to make a clear distinction between event processing agents that is doing -- filtering, aggregation, transformation and event pattern detection , and leave channel as a routing mechanism. thus we'll ensure proper separation of concerns, and have a cleaner model. We are now finishing the chapter about pattern detection, and I'll write about patterns soon.

EPDL expression for the "on off windows"

Yesterday, I drove my 11 year old (youngest) duaghter, Daphna, with four of her friends (all boys), 65 kilometers to the nearest branch of Max Brenner (see pictures above) a chocolate store which also has a restaurant, most of the menu is the chocolate oriented menu, but for parents there is also some real food. They still did not get to Haifa, but I think that they have brnaches in New York. Anyway, we found the nearest branch and got there...

I have posted an explanation about the approach to the "on off window" that Pedro posted; somebody asked me beyond the explanation -- how this scenario can be expressed in the EPDL meta-language that we are working on. Actually this is a very simple scenario, it is expressed like this:

Let E be the event type; with two attributes param1 and param2 (as given); we also need to route the event e to the agent a, and the output event to a producer - lets define two output pipes, p1 and p2, both of them carry events of type e.

We define:

• Context, name = c, type = temporal, initiated-by = event e with param1 = 2, terminated-by = event e with param2 = 0.

• Agent, name = a, type = filter, within-context = c, filter-condition = all, input = p1, output = p2
  

Explanation: Context determines which subset of the "event cloud" are applicable, there may be multiple agents within a context, but each agent has a single context. In this case - we can use the simplest type of agent - filter - with a trivial filter-condition - "all". Agent a will not be active before cotnext c will be active. With the event that satisfies the condition, the context is opened, thus agent a receives input, does nothing with it, and post it as output.

This is of course very simple case -- we can also concatenate all selected events to one composite event; we can also use more sophisticated agent e.g. pattern detection.

More educational material about languages -- later.

On formal models and agents

This is an aerial picture of Karlsruhe, in Germany. It is morning, I have the habbit to get up early, so have some time for Email and Blogging (this hotel does have internet connection in the room!). Here I am a guest of the FZI team that is trying to apply semantic models to event processing, which is one of my recent occupations. One of the property of researchers everywhere (and IBM is not different) that they are constantly looking for research funding - since one of the available sources in this region is EU projects, researchers from the relevant countries (and Israel is part of the EU, for the sake of the R&D programs), so there is a need to join forces. The FZI group is looking at applying transaction logic as a formal model for both validation and run-time control of event processing applications. I am in favor of having formal model behind event processing to define precisely the semantics. A model that will succeed should also be relatively simple. I was asked whether I see a possibility to have a single formal model that inclues all variations of event processsing - and my answer is that my assumption is that this is possible.

Dealing with formal models - I have been asked by several people about the choice I've made in my DEBS 20008 tutorial to define "event processing agent" as an atomic unit of processing.

First - I agree that the term agent is overloaded, like many other terms -- event processing agent is not (necessarily) an intellignet agent, it is more a software agents. The glossary defines agent as a software module that processes events. I have making a distinction between conceptual agent and run-tine agent - the run-time agent may consist of multiple conceptual agents, and the decision is up to self-optimization process (will write about optimization issues in one of the next blogs). It is easier to reason (and optimize) when the abstraction deals with atomic elements. Whether should be called - "atomic agent" - may be. I prefer shorter names, and since this does not contradict the original definition. more - later.

On DEBS 2008

Still in Rome, still in the hotel's bar since my room does not get enough wireless signal. The DEBS conference site is just next door to the church in which the famous Moses status is located.

I have also finished tonight my participation in DEBS (it goes one more day, but I will be heading for home tomorrow according to the plan). The DEBS conference (disregarding some organizational challenges) have gone well, while it has been a scientific conference, there has been a relatively big participation from industry - I have seen there people from vednors as well as customers, DEBS has been originally the home of the pub/sub and messaging community, however they have successfully extended it to include various other event processing "sub-communities" like the CEP guys, the streaming guys, people from modeling area etc..

I have contributed my part by providing a tutorial about EP patterns - which is a work in progress for building a meta-language which abstracts the semantics behind event processing functionality.

Since it is late tonight I'll defer discussions that I had in the conference to later posts, will just make a note that I owe everybody discussion on three points:

• Boundaries and territories among disciplines -- in the wake of discussion with Dieter Gawlick from Oracle.
• The term "event processing agents" - some discussions, and Blog comments from Paul Vincent
• The notion of pattern in event processing.

Overall - very good conference, an opportunity to meet a lot of friends, and (I am ashamed to admit) my first visit in Rome ever.

On CEP agent in EPN

This logo shows that we are not alone in the world of TLAs - there are other who use this acronyms, once I have made a list of acronyms of EDA which has shown that this is not a very good choice of acronym, but an acronym these days is context-sensitive. Anyway -- talking about "event processing networks", which we now try to clean its definitions - we have talked about SEP (simple event processing) agents, MEP (mediated event processing), CEP (complex event processing) and IEP (Intelligent event processing). The names can still change, I am not happy about any of them, but it seem to be relatively intuitive, and can be explained to non-technical people quite easily. Anyway, this time I'll concentrate on what is CEP agent. CEP agent have two basic phases:

1. Detect patterns over multiple events (these can be either multiple events of the same type, or multiple events from multiple event types - some are doing distinction here, but I don't see the rationale behind it) - this creates a collection of event-instances that satisfy the patterns.

2. Derive events as a function of the collection of events created in phase 1.

Some comments:

• There can be cases in which one of the two capabilities is used in a degenerated form. The pattern can be trivial (take all events without any pattern), but still a derivation (e.g. count, calculate the average of a certain attribute etc...) is required; on the other hand - the pattern can be complex, however the derivation is trivial (just a concatenation of all participating events).
• The term for the result of phase 1 - can be called "composite event" or "complex events", the two are not exactly synonyms, according to the glossary, as composite event is a syntactic term that is created by operators from primitive events, whereas complex event is a semantic term that includes aggregation of events that may not be obtained from patterns - however, the definition of phase 1 results is consistent with both.
• A CEP agent always lives within a context (temporal, spatial, semantic) -- most common type of context is a time-window, but the concept of context is much wider.
• The semantics of a CEP agent need to be fine-tuned in order to pick the instances of events participating in patterns relative to other possibilities which require to define policies - I'll talk more about semantics fine-tuning is a later postings.
• The result of a CEP agent is one or more derived events (consistent with other types of agents).

On Event Clouds

Marc Adler in a couple of his blog postings wondered about support of event clouds in the product he chose, and at the end has settled in the opinion of the vendor (Mark Tsimelzon from Coral8) who claims that "cloud" is an abstract term, and in reality we are facing multiple streams that may or may not be ordered. The response comes from Greg-the-architect who is in "everybody are confused" mode recently. Greg-the-architect claims that vendors have sinned in disinformation towards their customer to hide their inabilities to cope with hidden causal relations.

So - what can I contribute to that party ?

First - let's look again at the defintion of event cloud in the glossary:

Event cloud: a partially ordered set of events (poset), either bounded or unbounded, where the partial orderings are imposed by the causal, timing and other relationships between the events.

Clouds became a fashionable term, we hear a lot about cloud computing in the recent year, that we all feel like flying in various clouds.

What about the clouds/streams debate ? -- one of the differences that are stated is that a cloud is a poset (partially ordered set) while a stream is totally ordered. I agree that this terms come from two different origins, the question is if indeed a cloud can be supported by multiple streams, while people focus the discussion on whether streams are always totally ordered or can also support non-ordered set of events - this is not really an interesting distinction. I agree here with Mark Tsimelzon that a stream can also be un-ordered, this is up to implementation. If one wants to make a distinction between "streams" being ordered and other things that can be unordered, I propose the term "pipes" - where ordered pipe is a stream. But the ordered/unordered does not make the main difference. Reading the cloud definition again, it is the notion of cuasality that is important for having a cloud. The "partial ordering" in the cloud is a result of causality relations between events. I have discussed in a past posting the notion of causality, support in causality (including pre-determined causality that may be result of mining, or inference system) is the enabler for the support of clouds (i.e. the partial order vs. no order).

Cloud is indeed the collection of events that an enterprise is faced with, and this cloud may be implemented by a collection of pipes (or streams, if you wish) and support in causality relation.

We can also look at a (small) cloud, which is the collection of all events that a single EPA (Event Processing Agent) is facing as an input - and this is just a subset of the big "Cloud" - with its own pipes and causality relations.

Now - to the most important question - besides the game in terminology, is it important to make these distinctions?

As stated before, the world of event processing is not monolithic, there are some applications which need total order, while other applications need partial order, and other applications don't care about the notion of order at all. Causality relations are required by some applications, either if the pre-defined relations between the events play a role in the event processing, or if there is a need to trace back the lineage of a certain event / action. For other applications it may be just an unnecessary overhead. So my (2 cents worth of) advice to the people who are looking at CEP products - is to look at their requirements and determine if they need causality, and partial ordered set. It may be that the support of totally ordered stream is totally sufficient for their applications, if it is not - they should look for if and how causality is implemented. I hope that I have not confused you even more... More - later.

On Event Processing Agents

There are different type of agents -- double agents, as seen above which is a series of sweets, insuracne agents, travel agents, and some computerized agents - in my past I have dealt with mobile agents, and there are the intelligent agents in AI, and our own event processing agents (EPA).

David Luckham has written an amusing piece that follows Paul Vincent's "CEP and Agents" in TIBCO's Blog. The amusing part is that Paul has written about AI agents, which uses somewhat different terminology then the event processing terminology, and putting it in a "CEP Blog" is somewhat confusing. I am a "product" of the databases community, and have done some work that was on the AI border in the past, alas, the AI folks are using different terminology to talk about the same thing, and I thought at that time that they are doing it on purpose to confuse me. So, while there are many types of agents, I'll concentrate on the concept of "event processing agent" that has been coined by David Luckham. I like this term and adopted it in the following way: EPA (Event Processing Agents) is a software artifact that receives an event cloud or stream or collection of events or a single event (depends on the agent type and capabilities), does some computation on these event, and produce one or more events as an output. That's it. EPA is also a node in the EPN (Event Processing Network). There are different types of EPAs :

• "simple event processing" EPAs - filter and routing,
• "mediated event processing" EPAs - enrichment, transformation, validation
• "Complex event processing" EPAs - pattern detection
• "intelligent event processing" EPAs - prediction, decisions...

The common denominator: each of them receives events as input, emits events as output and does a single type of function.

I find this type of abstraction both very easy to explain people how EP systems work, and also basis for architecture. The EPN routing can be done by standard middleware, or in a stand-alone mode. Other terminology issues raised by David Luckham is the relationships to the "actor model" and to "engines".

The actor model is a model that helps reasoning about concurrency, while agents in AI are autonomous goal-driven artifacts. These are orthogonal terms, of course. In the context of EPA - when looking at EPAs as an executable network, we can look at each EPA as an actor and apply actor models.

Last but not least -- relationships of EPAs to engines -- an EPA is a software artifcat, it can be an instance of an engine, it can be some software that contains an engine, and it can be hard-coded program, as long as it complies with the EPA definition. In a future world, with inter-operability (and perhaps also language) standards, we'll be able to run (and maybe to self-select) multiple engines for the same EPN, residing in different EPAs.

More about EPA types -- later.

More on Standards and Event Processing

I have written about standards before a few months ago, and nothing seems to have happened since that time. As I was asked to talk in the OMG meeting in March about the role of standards in EP, I'll have to devise a more detailed view - but I still think that standards is one of the key issues we should invest in. In my opinion the most important standard is the language standard, the EPL - which should standard as a meta-language. There is an intention to start working on it later this year. Other areas - interoperability, modeling, various standards for event structure (header, payload). The importance of standards stem from various perspectives.

1. The customer's perspective -- the main competitor of EP COTS is hard-coding the functionality within a regular programming, and making EP as a non-issue. One of the claims of enterprises is that their developers know how to program in standard languages, and they will not invest in teaching them proprietary languages. While not everybody take this approach, the industry does not like proprietary. There is an effort to make Stream SQL language - but since it does not cover the entire market (even not most of it) - this is not enough.
2. The ability to draw incremental knowledge -- as an example, work in the academia on query optimization around SQL has helped the entire industry. Concentrating around a single language can serve as a focus and have the knowledge be incremental instead of disiributed.
3. When moving to the "platform" oriented EP, instead of the "engine" oriented of the first generation, a platform will be able to include various implementations - agents that are based on various technologies/engines... thus, various technologies need to inter-operate, but also be part of the same application, and we don't want the application be built in a mix of languages...

more -later.

On ECA and E*C*A*

Back on the ridge - Haifa is a collection of ridges on the Carmel mountain, my residence neigborhood Ramat Begin is seen here from bird eye's view. Anyway - still have some backlog from discussions last week in the CITT conference. One notion is about ECA (Event-Condition-Action) and its role in Event Processing. A few months ago in the VLDB conference I've met

Umesh Dayal, HP Fellow and a person with inspiring work, and we had a short discussion whether the area of "active databases" we have both involved in the past has failed, Umesh was in the opinion that while it has not become mainstrem in database products, as we hoped, but the notion of ECA (that has been coined by Umesh and his colleagues in the HiPAC project) has survived and had a substantial impact in various technologies (not necessarily in the database area). ECA stands for Event - Condition - Action. ECA still has role in event processing, however, this role is more similar to the role of rules in Event Processing - and mainly used in the edge of the EPN network, where the event is being consumed, ECA rules can control the action that is being trigerred at the consumer side. Interestingly enough we can charachterize the "event processing network" itself also using the same three letters - with stars - E*C*A* , where the interpretation is:

• E* - zero or more events that a single agent see (AKA "event cloud")
• C* - zero or more contexts that the agent is associated with
• A* - zero or more agents are activated.

The EPN routes events (both input event from a producer, and derived events from the EPN agents) according to context to activate agents.

On Enterprise Service Bus and Event Processing

This is one of the variations of Enterprise Service Bus (ESB) illustrations that I have taken from an article by one of my IBM colleagues. The topic of today is -- what are the relations between ESB and Event Processing ?

• An event processing functionality that runs Event Processing Agents in various sources, requires a that will take care of the routing and execution in a distributed environment. There are three alternatives here:

1. No native platform -- an engine that can run in multiple platforms (thus need to be integrated to each platform that it runs in by adapters etc...).
2. Dedicated event processing platform -- the event processing part has a dedicated platform that provides the infrastructure for the event processing functions.
3. Event Processing is built as part of an already existing platform.

All of these variations exist in the market today, and there are pros and cons for each of them, smaller vendors may prefer the first alternative as my friend Marco noted in his Blog.

When getting to the third alternative, if the environment is a SOA environment, then the ESB is a natural place in the SOA middleware to be the principle carrier of event processing functionality:

• It provides messaging infrastructure and routing capabilities
• It provides mediations like - validation, transformation and enrichment that can be reused for event processing (have a large intersection with the "mediated event processing" functions)
• It supports distributed environment.

While the principle usage of ESB in SOA has been to mediate between consumer and producer of services, being a carrier for event processing is now considered as a step in the evolution of ESBs.

This does not say that ESB is the ONLY place in which event processing functionality can run, which brings to a discussion about the Event Processing Conceptual Model- which I'll deal in a subsequent posting.

The ESB gets into the picture in alternative