Causality vs. correlation - statistical reasoning is not enough - NY Times Interview with Dave Ferrucci

Dave Ferrucci, who was until several months ago an IBM Fellow  and was known as the father of Watson, was interviewed by the NY Times in his new working place at Bridgewater Associates.

In the interview Ferrruci somewhat continues the line of thought of Noam Chomsky,  saying that AI has concentrated around statistical reasoning based on correlations, but the drawback is that one cannot understand why the prediction made by the statistical reasoning is correct.  While Chomsky bluntly stated that statistical reasoning does not create a solid model of the universe, Ferruci claims that a complementary approach is required -  understanding causality.    This is a rather old issue, in symbolic logic, there is a distinction between "material implication"  which states that  IF A is true then B is true, and the meaning is that always when A is true then B is also true, which makes a sentence like  "If the week has seven days than  the capital city of France is Paris" - a valid statement in logic.    Entailment, on the other hand, said that "A ENTAILS B" if it is necessary and relevant, in other word, there is a causality among them.  Thus, Ferruci concentrates now on building causality models to model the world economy.      I concur with the assertion that understanding causalities give better abilities of reasoning and prediction.   As David Luckham already noted, causality among events is one of the major abstraction of event processing models.   Here is a rather old discussion about causality of events.  

On inter-system event causalities

A lot has been spoken and written about the Wall Street Plunge, David Luckham raised the question about the role of event processing systems in prevention and/or postmortem analysis of understanding how such a situation happened. I am not sure exactly how much event processing system have been part of the reason of causing this plunge in the first place, but here we have an interesting challenge, the challenge is that we need to understand causality in a real event cloud, where the events are part of multiple systems. The nature of the challenge is twofold:

First: Each of the automated trading systems logic is, of course, not transparent, since they are used as competitive edge by the traders, thus it is difficult to know how various systems would function, and there will be a need to learn their logic (which can be constantly changed) based on their actual behavior;

Second: The logic of the various systems is expressed in various ways, and there is no standard causality model that can be used across systems.

This can be one of the challenges that we can put into the challenges of the next generation of event processing systems, that we plan to discuss in Dagstuhl.

More on Revision

Long day today, I got to the office around 8AM and left around 9PM. Since we have holiday this week I am trying to condense the remaining days of the week and the result is long day with plenty of conference calls. The picture above is a glance (from below) on the IBM Haifa Lab (the pair of connected building on the right hand side of the picture), my office is in the back building (known as the "banana" due to its shape), and is not really in the nice part of the building -- the one with the view to the Haifa Bay -- well, one can have everything in life -).

I still need to complete the previous posting on revision. I gave some explanation about the concept of revision, and now I still need to discuss implementation of revision in event processing. To recall -- a revision in event processing is getting later knowledge that asserts that either a reported event did not really happen, or some information associated with the event was wrong.

Let's look at two separate cases, one in which the processing has not gone out of the event processing network, and second that the results of the processing have gone out to the "outside world".

In the first case, there may be an opportunity to revise the impact of the revised event by doing kind of undo-redo for all the event processing agents that it passes directly or indirectly. Direct ones are easy -- those that the revised event participate as an input in them, indirect is more tricky, since we need to trace the causality among events, in this case, an event that is an output of an event processing agent in which the revised event participate (relate to the same context) has a causality relation to the original event, thus, an event processing agent, in which this event participates as an input, also needs to do an undo/redo, and causality is a transitive relation, so it continues as far as the EPN arrived so far. It should be noted that the fact that there is a causality may not require a real undo/redo, take as an example that an event of type E1 designates a bid, and the event of type E2 designates the bid with maximal value arrived in a certain time interval. Let's assume that a certain bid has been revised, however, neither the revised bid, or the revising bid change the selection of E2.

The second case is that the revised event has consequences that have been sent to an external consumer, thus, it may have triggered an action, a collection of actions, or a workflow that has been carried out, and this may propagate further ("the butterfly effect"), in this case, either we can treat it as "too late" and do nothing, however, there may be a cases that it can be critical to undo/redo also the consequences, e.g. the revised event has some financial meaning. In this case we'll need to issue compensation for the triggered action, which may be impossible (the consumer does not support compensation) or difficult. I'll blog again about revising the history and its aspects at a later phase.

On the three meanings of CEP

There is an old Jewish story on two people who had some dispute, and decided to go to the Rabbi and ask his opinion. The Rabbi listened to the first person and told him: you are right, then he listened to the second person and also told him: you are right. The Rabbi's assistant who has been present asked him: Rabbi, how can they both be right ? and received the obvious answer: you are also right.

Recently, a dispute between two opinionated persons - Hans Glide and Tim Bass has stormed the network, and somehow got also to my Blog. Somehow both of them understood that what I've written is consistent with their view - which encourages me to change career and become a Rabbi, but I think that there are some skills I lack - so anyway, I'll stay on event processing thinking.

While the dispute started around POSETS, the last posting by Tim Bass in the CEP forum re-focuses the discussion around - what is CEP ? so I'll stay with this topic for a while, since I think that there are (at least) three different interpretations of what CEP is - and this is a source of a lot of confusion. Thus I'll try to explain the three interpretations.

Interpretation One: the glossary interpretation -- complex event is the processing of complex events, where complex event is an abstraction or aggregation of events. According to this interpretation, a software function can be defined as CEP if it involves the processing of multiple events - typically collecting or matching some pattern among multiple events; The test for CEP according to this interpretation is typically support of a state that accumulate the relevant events - since they typically arrive over time. This definition does not say anything about the number of events, number of event types, whether they are totally ordered or not, or whether causality relation is supported or not - these are all attributes of specific implementations.

Interpretation Two: Event Processing = Complex Event Processing. This is a common practice in the commercial work to label all event processing functions as CEP. This spans from support of CEP (according to the interpretation one) and other event processing functions (such as: routing, enrichment, transformation) that don't satisfy the CEP test (since they are stateless and deal with a single event at a time). It can get to an absurd that some product that does not support CEP at all according to interpretation one, calls itself CEP. This is the most confusing interpretation, IMHO.

Interpretation Three: Complex event processing is event processing that has some complexity associated with it. According to Tim Bass - hidden causality and Markov Processes are vital for something to be defined as CEP. This really says that it CEP must involve uncertain events, causality that need to be discovered (by mining and other techniques), and the general usage of CEP is to predict something according to analysis of recent (past) events. According to Interpretation three, indeed the products that current products that call themselves CEP, do not satisfy this criteria, and thus are not CEP.

My opinion: as stated in the past, the term CEP has some inherent ambiguity, therefore I always thought it is confusing term. As far as my own taste in terminology - I prefer Interpretation One, saying that CEP is a subset of EP functions that deal with "complex events", it also seems that this is the closest to glossary definition. Interpretation two is confusing, as it turns CEP from a well-defined term to a marketing buzzword, and thus there is no test for what it is. Interpretation three is interesting, there are certainly applications that require prediction and various usages of stochastic processing and use of AI techniques (machine learning and others) in event processing. Hidden causalities is an important term, and I'll refer to it in another posting, since it has some pragmatic difficulty to obtain. However, I prefer to stick with the concept that CEP is processing of complex events, and not complex processing of events, and for interpretation one, we don't really need (necessarily) to apply AI techniques, this is just one type of application, there are a variety of applications that does not require it, and that detect predefined patterns on events is sufficient.

So the terminology that I personally prefer is:

Interpretation One = Complex Event Processing

Interpretation Two = Event Processing

Interpretation Three = Intelligent Event Processing.

On Posets and Red Herrings

Hans Glide, one of the knowledgeable persons in the EP area, has made in his Blog couple of postings about Posets, one of them claimed that the term Poset in EP is a red herring and of interest to mathematicians only, the other one says that causality graph and causality inference are not the same.

Let's make some assertions in this area, in order to clarify (or, god forbid, confuse...):

1. The EP applications space is not monolithic, different applications may have different requirements.
2. The issue of total order vs. partial order has been blown out of proportion, since it is not a major differentiator among products. Products who lack some functionality (see below) can somehow hack it in practice.
3. The main importance of causality relation in general is in the ability to trace all causal descendents of a certain event, or conversely to trace reasons for taking some action, this is a key requirement in anything that requires auditing. This does not cover the entire space of applications, and if the application does not require tracing and auditing then the notion of causality has little value to the application.
4. On the other hand, there are applications that manage scenarios, like software verification, simulation, games etc - in which causality is an important abstraction, as indeed reasoning is done on partially ordered set (directed acyclic graph) of events.
5. There are several types of orders and several types of causality relations.
6. Order may be according to the time that the arrives "happens in reality" (as reported by the source), or according to the time it is discovered by the event processing system (by getting to a system's API). In some cases the order does not play any role, in other cases, such as time series that are far enough, the sequence in the system is assumed to be good enough, and in other cases, the "reality" order has to be preserved -- how orders are preserved, if it is at all posssible, is a topic for another discussion.
7. Causality can be explicit or implicit. Explicit causality is meta-data in the event processing system, it exists because somebody put it there (maybe using mining techniques) saying that if Event E1 happens, than event E2 also happens, and we can assume that E2 happened even if there is no explicit indication that E2 happened. Implicit (inferred) causality is there since E2 is an output to some computation that E1 is an input to. In the class level it can be modelled, in the instance level, it is dynamic and created at run-time.

Closing statement: there is no - "one size fits all" in event processing. For each type of applications there are types of functionality that are more or less important to be supported. Better understanding of the existing types and mapping functionality to these types, is a work we are trying to achieve, by analyzing a significant number of use cases, as one of the EPTS tasks, by a large team of volunteers from all sectors (vendors, customers, academic people). We'll have more news on this work later this year.

On Event Clouds

Marc Adler in a couple of his blog postings wondered about support of event clouds in the product he chose, and at the end has settled in the opinion of the vendor (Mark Tsimelzon from Coral8) who claims that "cloud" is an abstract term, and in reality we are facing multiple streams that may or may not be ordered. The response comes from Greg-the-architect who is in "everybody are confused" mode recently. Greg-the-architect claims that vendors have sinned in disinformation towards their customer to hide their inabilities to cope with hidden causal relations.

So - what can I contribute to that party ?

First - let's look again at the defintion of event cloud in the glossary:

Event cloud: a partially ordered set of events (poset), either bounded or unbounded, where the partial orderings are imposed by the causal, timing and other relationships between the events.

Clouds became a fashionable term, we hear a lot about cloud computing in the recent year, that we all feel like flying in various clouds.

What about the clouds/streams debate ? -- one of the differences that are stated is that a cloud is a poset (partially ordered set) while a stream is totally ordered. I agree that this terms come from two different origins, the question is if indeed a cloud can be supported by multiple streams, while people focus the discussion on whether streams are always totally ordered or can also support non-ordered set of events - this is not really an interesting distinction. I agree here with Mark Tsimelzon that a stream can also be un-ordered, this is up to implementation. If one wants to make a distinction between "streams" being ordered and other things that can be unordered, I propose the term "pipes" - where ordered pipe is a stream. But the ordered/unordered does not make the main difference. Reading the cloud definition again, it is the notion of cuasality that is important for having a cloud. The "partial ordering" in the cloud is a result of causality relations between events. I have discussed in a past posting the notion of causality, support in causality (including pre-determined causality that may be result of mining, or inference system) is the enabler for the support of clouds (i.e. the partial order vs. no order).

Cloud is indeed the collection of events that an enterprise is faced with, and this cloud may be implemented by a collection of pipes (or streams, if you wish) and support in causality relation.

We can also look at a (small) cloud, which is the collection of all events that a single EPA (Event Processing Agent) is facing as an input - and this is just a subset of the big "Cloud" - with its own pipes and causality relations.

Now - to the most important question - besides the game in terminology, is it important to make these distinctions?

As stated before, the world of event processing is not monolithic, there are some applications which need total order, while other applications need partial order, and other applications don't care about the notion of order at all. Causality relations are required by some applications, either if the pre-defined relations between the events play a role in the event processing, or if there is a need to trace back the lineage of a certain event / action. For other applications it may be just an unnecessary overhead. So my (2 cents worth of) advice to the people who are looking at CEP products - is to look at their requirements and determine if they need causality, and partial ordered set. It may be that the support of totally ordered stream is totally sufficient for their applications, if it is not - they should look for if and how causality is implemented. I hope that I have not confused you even more... More - later.

Causality and lineage in event processing

[Today I am drilling into the micro (technical issue), with some macro level lesson at the end]


This is what I've found on the web looking for an image describing causality



David Luckham , in his book has made "causality" as one of the major abstractions in CEP. As you can see in the Wikipedia entry, this term has interpretations in multiple disciplines. Causality simply says that existence of event E2 is caused by event E1. Anybody familiar with discussions around the meaning of "causality" in logic, realizes that there are various approaches and terms in this area. In our context, event processing, we can look at different types of causality.

Type I: predetermined causality - Event E2 always (or conditionally) occurs as a result the occurrence of E1, thus we don't need to have any sensor to get event E2 we may assume it happened if E1 happened (and the condition is satisfied), some time offset or interval may be attached to this causality. Note that in this case E1 and E2 are both raw events.

Type II: The event E1 is an input to an EPA (Event Processing Agent) AG, and the event E2 is an output of AG. In this case E2 is a derived (virtual) event. This can be further refined to other types according to the issues - whether E2 is really a function of E1, this is known since the agent's specification is part of the system.

Type III: The event E1 is an event that is sent from an EPN to a consumer C. C applies (conditionally) some action AC, where the specification of AC is not known to us, but we observe that it emits the event E2. This is another type of causality (the event E2 would not have been emitted, if E2 would not have triggered AC), however, E2 may or may not have functional dependency with respect to E1 (i.e. the value of E2 is somehow function of E1) .

Some questions have been asked on my previous post on ECA rules, if EPA is not just an action, why is this distinction important - and the answer is - in the EPA case we have dependency of type II - which means both causal and functional dependency, while in a general action, it is not known whether there is functional dependency.

The question is -- why is this all causality discussion important ? is there just a theoretical notion, and the answer here is -- lineage tracing.

In some cases, it is important to be able to answer questions like:

• What have been the chain of events and transformations that caused a certain action (decision) to occur ?
• What are all the consequences of a certain events (of a certain type) ?
• What would have happened if a certain event that did not happen would have happened in a certain time-point ? (or the reverse - an event that happened would not have happened).

Applications ? auditing, decision analysis... the last type of question relates to some past work done in the temporal area -- temporal issues are very pervasive in event processing and deserve one (or more) postings at a later point.