event processing as a part of a bigger picture

The picture is the view seen from "Midreshet Sde Boker" , where I have driven yesterday (3.5 hours drive each direction !) to attend the opening ceremony of the activity which my second daughter (out of four), Adi, is going to participate during the coming year. This is an activity in which high-school graduates postpone their army duty for one year, and invest it in volunteering activities, and learn leadership skills, in the Negev dessert, "in the middle of nowhere". From the Greek philosophy we have two competing views of the relations between the individual and the society -- the individual-centered, where society is aimed to serve the individuals, and society-centered, where individuals are part of society. Our universe is very much individual-centered, so it is a refreshing to find a group of young people who are ready to contribute one year from their life (yes, without being paid), to do volunteering work that help the society.

In moving back to "event processing", it is clear that event processing is not a detached thing, as the customer really wants a manufacturing system, or trading system, or billing system, and technology is only a way to achieve it, it is also true that there are typically no pure event-driven applications, but parts of applications that do event processing, which makes the interoperability, the ability to embed event processing within existing paradigms, and the ability of systems and applications to sense and respond, is a critical success factor.

Consequently, we'll see more and more event processing systems embedded in application integration middleware, rather than a stand-alone engines. More on that - later.

Causality and lineage in event processing

[Today I am drilling into the micro (technical issue), with some macro level lesson at the end]


This is what I've found on the web looking for an image describing causality



David Luckham , in his book has made "causality" as one of the major abstractions in CEP. As you can see in the Wikipedia entry, this term has interpretations in multiple disciplines. Causality simply says that existence of event E2 is caused by event E1. Anybody familiar with discussions around the meaning of "causality" in logic, realizes that there are various approaches and terms in this area. In our context, event processing, we can look at different types of causality.

Type I: predetermined causality - Event E2 always (or conditionally) occurs as a result the occurrence of E1, thus we don't need to have any sensor to get event E2 we may assume it happened if E1 happened (and the condition is satisfied), some time offset or interval may be attached to this causality. Note that in this case E1 and E2 are both raw events.

Type II: The event E1 is an input to an EPA (Event Processing Agent) AG, and the event E2 is an output of AG. In this case E2 is a derived (virtual) event. This can be further refined to other types according to the issues - whether E2 is really a function of E1, this is known since the agent's specification is part of the system.

Type III: The event E1 is an event that is sent from an EPN to a consumer C. C applies (conditionally) some action AC, where the specification of AC is not known to us, but we observe that it emits the event E2. This is another type of causality (the event E2 would not have been emitted, if E2 would not have triggered AC), however, E2 may or may not have functional dependency with respect to E1 (i.e. the value of E2 is somehow function of E1) .

Some questions have been asked on my previous post on ECA rules, if EPA is not just an action, why is this distinction important - and the answer is - in the EPA case we have dependency of type II - which means both causal and functional dependency, while in a general action, it is not known whether there is functional dependency.

The question is -- why is this all causality discussion important ? is there just a theoretical notion, and the answer here is -- lineage tracing.

In some cases, it is important to be able to answer questions like:

• What have been the chain of events and transformations that caused a certain action (decision) to occur ?
• What are all the consequences of a certain events (of a certain type) ?
• What would have happened if a certain event that did not happen would have happened in a certain time-point ? (or the reverse - an event that happened would not have happened).

Applications ? auditing, decision analysis... the last type of question relates to some past work done in the temporal area -- temporal issues are very pervasive in event processing and deserve one (or more) postings at a later point.

More about event processing and business rules

Since my previous post on this topic has triggered some discussion, comments and questions - I am returning to the issue of "event processing and business rules". One observation I got today is that if we'll some of the "event processing typical applications", example: Paul Vincent's post about EDA and CEP in Insurance are considered by the business rule people also as typical applications for business rule engine (and the insurance area has strong presence of business rules). To answer that I'll cite another CEP blogger, whom I don't have the honor to know personally, who used the eternal-wisdom title: When all you have is a hammer everything looks like a nail, while the post talks about SQL, this is true also for business rules. In fact, if some programming tool has enough expressive power, one can, somehow, express everything using this tool, which does not say a lot, since everything can be expressed nicely by a Turing machine (this wikipedia link also points at some freeware implementation of Turing machine, so we have a single programming tool to do everything...). However, there are things that are naturally fit into a certain programming paradigm and things that look rather awkward. Of course, there are some cases in which one can equally implement it within multiple paradigms, this is typically one of two cases - it may be very simple, or there is not (yet) a specific paradigm to express it nicely, so multiple paradigms are awkward in the same way.
It is not surprising that the same phrase about hammer and nails is also going in the opposite directions- I have seen cases where people who had CEP engines, "force" them to do things that are not natural for this paradigm, by creating artificially events and patterns.

And one comment about ECA rules (Event-Condition-Action) --- the contribution of the glorious "active databases" community to the computing industry. Complex Event Processing deals with the E part - taking events, apply some function, and at the end deriving more events. Each derived event that arrive to the "edge of the event processing network", i.e. sent to a consumer, can then conditionally trigger an action. Thus, from architecture point of view, ECA rules are really part of the consumer logic, and not part of the "event processing network". More - later.

Who should define event processing applications ?

This picture is taken from the recent ebizQ's event processing survey and shows an interesting picture, the survey done shows that the customers prefer that rules will be defined in 84% of the cases by non-programmers. This is not the current situation, but this is the way that the customers prefer it. Indeed, it is not possible to have the entire event processing end-to-end story done by non-technical people in this phase (maybe with applying some autonomic computing techniques we can come closer to it, but this is a topic for another Blog), but let's concentrate now in the CEP (Complex Event Processing) part - detection of patterns. Given the right level of abstraction both of the "programming in the small" - the individual pattern, and the "programming in the large" - the event flow, pattern composition may be delegated to business people (well - with some understanding of basic terms like the distinction between "OR" and "AND", but with no need to know how to program in Java, C, SQL and other languages geared for programmers). Bear in mind that the bigger expense in the application's life-cycle is not the cost of development, but the cost of maintenance, and this is especially true when changes are frequent, or there is a need to create new patterns to detect unexpected events within a short time. In fact, the agility, ability to quickly adapt, and ease-of-use for non-technical people is a success factor for the entire concept of event processing COTS, and the ground for competition among vendors. Investments in tools for developers address only 16% of the market, according to this survey. This reminds me that once in my academic past, I attended a teaching workshop given by an external experts, and they started their session saying that - most teachers base their teaching on the students' sense of hearing, while for most people the visual sense is much stronger, so there is a mismatch.... More later.

event processing and business rules

New week is starting after all holidays (in Israel the working week is Sunday - Thursday), so let's start it with writing something in the Blog. I have stated that SQL is not the ideal way to express event processing functionality, so, people ask me what is the alternative - business rules ? the answer is - not really. The term business rules is somewhat overloaded, however, we can group the rule types into two types:

• knowledge-creation rules - rules that create more 'knowledge" - facts, data-elements etc... These rules can use various inference techniques - deduction, induction, abduction...
• behavioral rules - rules that cause something to be done, or prevent something from happen.

The EPL should include constructs with some similarity to these two types, but this does not say that existing rule languages can be taken as a basis, this is because the domain in event processing is events (not facts or data), and that event has a distinct semantics.

• The first type of construct in EPL is the "event derivation rules" which includes transformations (filtering, translation, aggregation, split, and enrichment) - these typically transform an event (aggregation is a special case, we'll dedicate a blog to it), and "pattern based derivation" which derives events, such that the instances of the derivers are selected as part of a detected pattern. We can see that all these types require specialized semantics, thus while there is similarity in ideas - it is not really rule semantics.
• The second is somewhat similar to the "orchestration" constructs of EPL - this is being performed in the leaf of the network - after all derivations occurred, the result event may (conditionally) perform some action related to a consumer (notify, orchestrate etc...), which is also known as ECA rules (event-condition-action). Note, that from architecture point of view, ECA rules are not done in the event processing network (i.e. they are not MEP or CEP), but they occur in the territory of the consumer - at the border of the EPN.

An interesting observation id that Complex Event Processing (CEP) as defined in the glossary, deals with derivation of new events, it is neither an instance nor an extension of ECA rules, since CEP deals with "knowledge creation" and ECA deals with behavior.

Some more discussion on this topics, and relation with "reaction rules" - later.