Tuesday, April 15, 2008

On Event Processing Agents


There are different type of agents -- double agents, as seen above which is a series of sweets, insuracne agents, travel agents, and some computerized agents - in my past I have dealt with mobile agents, and there are the intelligent agents in AI, and our own event processing agents (EPA).
David Luckham has written an amusing piece that follows Paul Vincent's "CEP and Agents" in TIBCO's Blog. The amusing part is that Paul has written about AI agents, which uses somewhat different terminology then the event processing terminology, and putting it in a "CEP Blog" is somewhat confusing. I am a "product" of the databases community, and have done some work that was on the AI border in the past, alas, the AI folks are using different terminology to talk about the same thing, and I thought at that time that they are doing it on purpose to confuse me. So, while there are many types of agents, I'll concentrate on the concept of "event processing agent" that has been coined by David Luckham. I like this term and adopted it in the following way: EPA (Event Processing Agents) is a software artifact that receives an event cloud or stream or collection of events or a single event (depends on the agent type and capabilities), does some computation on these event, and produce one or more events as an output. That's it. EPA is also a node in the EPN (Event Processing Network). There are different types of EPAs :
  • "simple event processing" EPAs - filter and routing,
  • "mediated event processing" EPAs - enrichment, transformation, validation
  • "Complex event processing" EPAs - pattern detection
  • "intelligent event processing" EPAs - prediction, decisions...

The common denominator: each of them receives events as input, emits events as output and does a single type of function.

I find this type of abstraction both very easy to explain people how EP systems work, and also basis for architecture. The EPN routing can be done by standard middleware, or in a stand-alone mode. Other terminology issues raised by David Luckham is the relationships to the "actor model" and to "engines".

The actor model is a model that helps reasoning about concurrency, while agents in AI are autonomous goal-driven artifacts. These are orthogonal terms, of course. In the context of EPA - when looking at EPAs as an executable network, we can look at each EPA as an actor and apply actor models.

Last but not least -- relationships of EPAs to engines -- an EPA is a software artifcat, it can be an instance of an engine, it can be some software that contains an engine, and it can be hard-coded program, as long as it complies with the EPA definition. In a future world, with inter-operability (and perhaps also language) standards, we'll be able to run (and maybe to self-select) multiple engines for the same EPN, residing in different EPAs.

More about EPA types -- later.

3 comments:

PF said...

Hi,

I got to this post by you (Opher) after being interested in EPN and EPAs and searching for this topic on Google. Interesting is that just a few days ago we were talking about a similar subject by email. Our specific subject was recursion in event processing. Your point was that this is supported by loops in the event processing network (EPN). However, I was not convinced that recursion is not part of the event processing agent (EPA) (depends on its expressive power). Researching more on the subject I started to think of reasons for changing this architecture (better integration of EPA and EPN, declarative dynamic workflow instead of fixed structure, etc.) and I got to this post by you (Opher :), as the first link on Google when one searches for "event processing agent".

First of all, although I don't agree with fixed workflows, I agree with the overall idea of an Event Processing World/Web of agents (a specific case can be a fixed network, but it shouldn't be limited to that). There is definitely value in EPW by having separate processing entities in the form of agents communicating in a common environment (the world). This makes sense in the Semantic Web area where are distributed "namespaces" and agents in the entire Web.

Second, I see CEP agents as full agents with enough expressive power so they can compute any kind of complex events.

Regarding the hierarchy of EPAs above, I have a few remarks:
-"simple event processing" EPAs - filter and routing,

Filter means selection with some condition on the original events, while routing is part of the event processing network (EPN).

-"mediated event processing" EPAs - enrichment, transformation, validation

“Enritchment” means using the internal database to enrich the input events with some extra information; transformation is both selection on some condition on the arguments and projection of some arguments of the original events. Validation is selection on some condition and maybe a mechanism of exceptions and alarms.

I'm not sure I can see the distinction between "simple event processing" EPAs and "mediated event processing" EPAs.
(to continue)

PF said...

Hi,

I got to this post by you (Opher) after being interested in EPN and EPAs and searching for this topic on Google. Interesting is that just a few days ago we were talking about a similar subject by email. Our specific subject was recursion in event processing. Your point was that this is supported by loops in the event processing network (EPN). However, I was not convinced that recursion is not part of the event processing agent (EPA) (depends on its expressive power). Researching more on the subject I started to think of reasons for changing this architecture (better integration of EPA and EPN, declarative dynamic workflow instead of fixed structure, etc.) and I got to this post by you (Opher :), as the first link on Google when one searches for "event processing agent".

First of all, although I don't agree with fixed workflows, I agree with the overall idea of an Event Processing World/Web of agents (a specific case can be a fixed network, but it shouldn't be limited to that). There is definitely value in EPW by having separate processing entities in the form of agents communicating in a common environment (the world). This makes sense in the Semantic Web area where are distributed "namespaces" and agents in the entire Web.

Second, I see CEP agents as full agents with enough expressive power so they can compute any kind of complex events.

Regarding the hierarchy of EPAs above, I have a few remarks:
-"simple event processing" EPAs - filter and routing,

Filter means selection with some condition on the original events, while routing is part of the event processing network (EPN).

-"mediated event processing" EPAs - enrichment, transformation, validation

“Enritchment” means using the internal database to enrich the input events with some extra information; transformation is both selection on some condition on the arguments and projection of some arguments of the original events. Validation is selection on some condition and maybe a mechanism of exceptions and alarms.

I'm not sure I can see the distinction between "simple event processing" EPAs and "mediated event processing" EPAs.
(to continue - I found out that the blog doesn't allow me to post the whole comment after I wrote it)

PF said...

-"Complex event processing" EPAs - pattern detection

EPA pattern detection is detection of sequences of events and other patterns (based on time, filters with static conditions or conditions from the database, etc.)

Up to here EPAs were definitely "stateless". However, “recursion” presumes that agents are stateful, or, at least, have access to their outputs and compute some extra events or a fixpoint of its own derived events. That is, the relations that define new complex events are recursive. The example that we discussed in the email is:

reach_event(X,Y) <- edge_event(X,Y).
reach_event(X,Y) <- reach_event(X,Z) ’seq’ edge_event(Z,Y).

I think this still follows the definition above:
"Complex Event Processing Agents is a software artifact that receives an event cloud or stream or collection of events or a single event, does some computation on these event, and produce one or more events as an output."

However, this might also be a different category or EPAs:
-"Stateful Complex event processing" EPAs or
-"Recursive Complex event processing" EPAs

Personally, I prefer an expressivity hierarchy in the class of "Complex event processing" EPAs. In logic programming, recursive programs are a completely different category of programs then non-recursive programs. Moreover, when one has negations (classical and default negations) the hierarchy becomes much more complex, e.g., stratified programs, non stratified programs, locally-stratified programs and not locally-stratified programs, with diverse logic semantics: well-founded semantics, stable models, etc.

-"intelligent event processing" EPAs - prediction, decisions...

Just to finish the hierarchy, I think that the last category is just about integration of CEP with other semantic constructs: "prediction" IEP EPAs are CEP EPAs with underlying meaning of events (semantics like in the Semantic Web and knowledge representation areas), while "decision" refers to integration with event-actions rules (up to my understanding). The point is that this last category is regardless to the issues discussed above.

I also understand your statement that "this type of abstraction both very easy to explain people how EP systems work, and also basis for architecture. The EPN routing can be done by standard middleware...".

Finally, the point of this comment is to see if the issues mentioned above are valuable or not, and that I believe in a better overall integration of the whole architecture for complex event processing and maybe in a different hierarchy of EPAs.

I hope this post also helps other people interested in the same problems discussed above.

Paul Fodor.