On Actian and action applications

Ingres, one of the oldest DBMS companies which produces open source DBMS, and the first of the sequence of companies that Mike Stonebraker founded and sold, has recently changed its name to Actian, and positioned itself as focused on "action applications in big data".  The stated rationale about "action applications" is that current BI create reports and then it is left to the human to read the reports (or screens) and decide what to do, in "action applications", the application trigger actions automatically in response to data events and thresholds.   It seems that people from the BI community re-discovers/re-invents the Event-Condition-Action model?   so they'll probably get to more advanced event processing at some point.


It is interesting to note that the motivation they state on the Actian website (you'll have to press on "action apps" to see it) is - "BI is not working, more than $10B are spent every year on a pile of reports with no actions".    I guess that this is somewhat consistent with my previous posting citing a study that indicate that human decision makers don't succeed to get fast decisions based on BI.   Maybe BI is getting in the hype cycle to the phase of disillusionment, and maybe people in this community like SAS CEO who said last year that event processing has limited appeal  to BI (along with BI in the cloud), would have second thoughts.

On ECA and E*C*A*

Back on the ridge - Haifa is a collection of ridges on the Carmel mountain, my residence neigborhood Ramat Begin is seen here from bird eye's view. Anyway - still have some backlog from discussions last week in the CITT conference. One notion is about ECA (Event-Condition-Action) and its role in Event Processing. A few months ago in the VLDB conference I've met

Umesh Dayal, HP Fellow and a person with inspiring work, and we had a short discussion whether the area of "active databases" we have both involved in the past has failed, Umesh was in the opinion that while it has not become mainstrem in database products, as we hoped, but the notion of ECA (that has been coined by Umesh and his colleagues in the HiPAC project) has survived and had a substantial impact in various technologies (not necessarily in the database area). ECA stands for Event - Condition - Action. ECA still has role in event processing, however, this role is more similar to the role of rules in Event Processing - and mainly used in the edge of the EPN network, where the event is being consumed, ECA rules can control the action that is being trigerred at the consumer side. Interestingly enough we can charachterize the "event processing network" itself also using the same three letters - with stars - E*C*A* , where the interpretation is:

• E* - zero or more events that a single agent see (AKA "event cloud")
• C* - zero or more contexts that the agent is associated with
• A* - zero or more agents are activated.

The EPN routes events (both input event from a producer, and derived events from the EPN agents) according to context to activate agents.

More thoughts on Rules in the context of Event Processing

I spent today (and tomorrow) in the IBM Hursley Lab - my second home in the last couple of years, this is a picture of the "Hursley House" (well - from the back side) - a countryside English manor that serves today as place for meetings and conferences, and the office of the Lab Director - besides this building there is a complex of connected building with multiple systems for room numbering that can provide in-door navigation challenges. I'll write today about some ideas that came out from discussions in the CITT meeting earlier this week about the role of rule technology. I still hold my opinion that although it is possible to take a technology that has been developed for one purpose and "hack" it to use it to other purpose, it may not be the most natural/effective/efficient way to do. This is true for SQL as well as rules when we are talking about pattern detection in complex event processing (I am working now on tutorial on the issue of patterns). However, this does not say that rule technology does not have a place in event processing in general. Here are some places it can be used:

(1). Decision-based routing in event processing networks.

(2). Transformation of events.

(3). Validation of events.

(4). Orchestration rules.

(5). Intelligent Event Processing.

Note that different type of rules are being used for the different cases -

for routing and transformation - it is typically - if-then rules/decision trees/decision tables.

for validation - constraint oriented rules.

for orchestration - ECA rules (note that orchestration rules are in the domain of the consumer that receives an event from the event processing network and has to decide what to do with it).

for intelligent event processing - all type of rules - deductive, inductive, abductive, rules with uncertainty - can play in different cases.

More about ECA rules and event processing - soon.

On the classification of events

I am returning to the area of "classification of events", although it has been discussed in this blog several times regarding certain aspects, the reason is - some discussion in which I realized that the confusion is still alive and kicking.

Events can be classified according to their structure, semantics, or pragmatics. This classification is orthogonal.

Based on Structure:

• Simple or atomic event - an event that reports on a single occurrence.

The event structure may be flat (like relational database), or semi-structured (XML) - which also can express non-flat structure (e.g. hierarchical). In some cases events are represented as objects. Typically events contain "header" - information about this event (event class, time stamp, source...), and "payload" - the content of the event.

• Complex event - an event that is composed of multiple events

There is some difference between this and the term "composite event" from active databases which mixes structure and semantic aspect, but I don't see it useful to use this term any more.

Based on source

• Raw event: Event that has been reported to the event processing system by an external event producer.

Note that a raw event can be simple or complex. The term "raw" is relative, since it can be result of some computation process in the event producer that is transparent to the event processing system.

• Derived event: Event that is created by the event processing system as a result of some processing.

Again, the derived event can be simple/atomic - where the attributes are computed as functions of raw and derived events, or complex - in which the derived event is a concatenation of events.

Based on ontology:

• Real event: An event that designates something that occurs in reality
• Virtual event: Hypothetical, simulated or uncertain event.

Again, this is a separate dimension - can be in any structure and by any source.

Based on pragmatics

• Reactionable event (AKA situation): An event that is consumed by an external consumer at the end of the event processing network (either notify or orchestrate).

We used the term "situation" for a phenomenon in the consumer's domain of discourse that the consumer wishes to react to. This can be raw event (and in this case the event processing system is reduced to "simple event processing" - i.e. filtering and routing) or derived event, it can also be real or virtual, and in any structure.

BTW - ECA - "Event-Condition-Action" rules are rules in which the events are situations, and they are executed within the consumer domain, but I'll continue to discuss the conceptual model in the next posting.

On simple events and simple event processing

This is a picture I have borrowed from Siemens, however, I'll use it talking about simple events and simple event processing. There is a constant confusion around the terms "simple" (and "complex") here due to the ambiguity of the phrase: simple (complex) event processing - does it mean: processing of simple (or complex) events? or does it mean simple (or complex) processing in events. In this posting, I am drilling down to the simplicity notion.

Let's start with simple event - I prefer to contrast simple event with composite event where the contrast is in the structure - composite event is a collection of simple events and simple event is an atomic event, nothing said about the processing yet.

Now - is there different processing for simple events and composite events ? - in principle no - there are some functions on collections that are not applicable for atomic events, but if we take a collection of simple events that has not been concatenated we can apply the same processing for them.

Thus - my preference is to attach the simple to the processing and not to the event type, and define simple event processing as simple type of processing, no matter what the event structure is. What are the characteristics of simple event processing ?

• processing is done on a single event - not looking on other events, but events are processed "one at a time".


• only types of processing possible are: filtering and routing


• filtering decides whether this event should be passed


• routing decides to whom this event should be passed

Basic pub/sub with filtering is simple event processing.

ECA (Event-Condition-Action) rule is also simple event processing -- this does not say that the event cannot be derived/complex/composite - but regardless of the history of the event, its structure, the reason why it is created, and its source the ECA rule still performs a simple event processing, and processes one event at a time, where the condition provides filtering, and the action is indeed routing to somewhere to do this action.

Furthermore, in many cases, the "simple event processing" is a preamble to the event processing network done by the event producer, or post-processing done by the consumer, however, it can still be part of the Event Processing Network.

More related concepts - in the next postings.

Causality and lineage in event processing

[Today I am drilling into the micro (technical issue), with some macro level lesson at the end]


This is what I've found on the web looking for an image describing causality



David Luckham , in his book has made "causality" as one of the major abstractions in CEP. As you can see in the Wikipedia entry, this term has interpretations in multiple disciplines. Causality simply says that existence of event E2 is caused by event E1. Anybody familiar with discussions around the meaning of "causality" in logic, realizes that there are various approaches and terms in this area. In our context, event processing, we can look at different types of causality.

Type I: predetermined causality - Event E2 always (or conditionally) occurs as a result the occurrence of E1, thus we don't need to have any sensor to get event E2 we may assume it happened if E1 happened (and the condition is satisfied), some time offset or interval may be attached to this causality. Note that in this case E1 and E2 are both raw events.

Type II: The event E1 is an input to an EPA (Event Processing Agent) AG, and the event E2 is an output of AG. In this case E2 is a derived (virtual) event. This can be further refined to other types according to the issues - whether E2 is really a function of E1, this is known since the agent's specification is part of the system.

Type III: The event E1 is an event that is sent from an EPN to a consumer C. C applies (conditionally) some action AC, where the specification of AC is not known to us, but we observe that it emits the event E2. This is another type of causality (the event E2 would not have been emitted, if E2 would not have triggered AC), however, E2 may or may not have functional dependency with respect to E1 (i.e. the value of E2 is somehow function of E1) .

Some questions have been asked on my previous post on ECA rules, if EPA is not just an action, why is this distinction important - and the answer is - in the EPA case we have dependency of type II - which means both causal and functional dependency, while in a general action, it is not known whether there is functional dependency.

The question is -- why is this all causality discussion important ? is there just a theoretical notion, and the answer here is -- lineage tracing.

In some cases, it is important to be able to answer questions like:

• What have been the chain of events and transformations that caused a certain action (decision) to occur ?
• What are all the consequences of a certain events (of a certain type) ?
• What would have happened if a certain event that did not happen would have happened in a certain time-point ? (or the reverse - an event that happened would not have happened).

Applications ? auditing, decision analysis... the last type of question relates to some past work done in the temporal area -- temporal issues are very pervasive in event processing and deserve one (or more) postings at a later point.

More about event processing and business rules

Since my previous post on this topic has triggered some discussion, comments and questions - I am returning to the issue of "event processing and business rules". One observation I got today is that if we'll some of the "event processing typical applications", example: Paul Vincent's post about EDA and CEP in Insurance are considered by the business rule people also as typical applications for business rule engine (and the insurance area has strong presence of business rules). To answer that I'll cite another CEP blogger, whom I don't have the honor to know personally, who used the eternal-wisdom title: When all you have is a hammer everything looks like a nail, while the post talks about SQL, this is true also for business rules. In fact, if some programming tool has enough expressive power, one can, somehow, express everything using this tool, which does not say a lot, since everything can be expressed nicely by a Turing machine (this wikipedia link also points at some freeware implementation of Turing machine, so we have a single programming tool to do everything...). However, there are things that are naturally fit into a certain programming paradigm and things that look rather awkward. Of course, there are some cases in which one can equally implement it within multiple paradigms, this is typically one of two cases - it may be very simple, or there is not (yet) a specific paradigm to express it nicely, so multiple paradigms are awkward in the same way.
It is not surprising that the same phrase about hammer and nails is also going in the opposite directions- I have seen cases where people who had CEP engines, "force" them to do things that are not natural for this paradigm, by creating artificially events and patterns.

And one comment about ECA rules (Event-Condition-Action) --- the contribution of the glorious "active databases" community to the computing industry. Complex Event Processing deals with the E part - taking events, apply some function, and at the end deriving more events. Each derived event that arrive to the "edge of the event processing network", i.e. sent to a consumer, can then conditionally trigger an action. Thus, from architecture point of view, ECA rules are really part of the consumer logic, and not part of the "event processing network". More - later.

event processing and business rules

New week is starting after all holidays (in Israel the working week is Sunday - Thursday), so let's start it with writing something in the Blog. I have stated that SQL is not the ideal way to express event processing functionality, so, people ask me what is the alternative - business rules ? the answer is - not really. The term business rules is somewhat overloaded, however, we can group the rule types into two types:

• knowledge-creation rules - rules that create more 'knowledge" - facts, data-elements etc... These rules can use various inference techniques - deduction, induction, abduction...
• behavioral rules - rules that cause something to be done, or prevent something from happen.

The EPL should include constructs with some similarity to these two types, but this does not say that existing rule languages can be taken as a basis, this is because the domain in event processing is events (not facts or data), and that event has a distinct semantics.

• The first type of construct in EPL is the "event derivation rules" which includes transformations (filtering, translation, aggregation, split, and enrichment) - these typically transform an event (aggregation is a special case, we'll dedicate a blog to it), and "pattern based derivation" which derives events, such that the instances of the derivers are selected as part of a detected pattern. We can see that all these types require specialized semantics, thus while there is similarity in ideas - it is not really rule semantics.
• The second is somewhat similar to the "orchestration" constructs of EPL - this is being performed in the leaf of the network - after all derivations occurred, the result event may (conditionally) perform some action related to a consumer (notify, orchestrate etc...), which is also known as ECA rules (event-condition-action). Note, that from architecture point of view, ECA rules are not done in the event processing network (i.e. they are not MEP or CEP), but they occur in the territory of the consumer - at the border of the EPN.

An interesting observation id that Complex Event Processing (CEP) as defined in the glossary, deals with derivation of new events, it is neither an instance nor an extension of ECA rules, since CEP deals with "knowledge creation" and ECA deals with behavior.

Some more discussion on this topics, and relation with "reaction rules" - later.