On reasoning about events and states

Yesterday, we watched in our local theater, the play "Amadeus". I have seen the movie when it was first launched, and another version of this play in the nineties. Still very impressive, and an opportunity to hear some of Mozart's good music.

Thanks to Paul Vincent's Blog, I clicked the link and got to Paul Haley's posting entitled "time of the next generation of knowledge automation". Paul Haley starts his postings by classifying four types of reasoning:

1. Reasoning at a point within a [business] process
2. Reasoning about events that occur over time.
3. Reasoning about a [business] process (as in deciding what comes next)
4. Reasoning about and across different states (as in planning)

The claim is that the first kind is being supported by business rules (called EDM in the original), the second kind is being supported by event processing, while the third and fourth kinds are not really supported by the state of the art. The third type requires getting the notion of context into reasoning while the fourth type requires getting cross state reasoning (which is different from cross event reasoning). I agree with the classification, and will write more in the future about the third and fourth types in depth. Actually, we also need reasoning that will combine the different types of reasoning as well.

On composite contexts

Today is a happy day to the Israeli scientific community it was announced that Professor Ada Yonath from the Weizmann Institute won a Nobel prize in chemistry, this adds to two Israeli scientists that won a Nobel prize in chemistry a few years ago, so Israel is a chemistry super-power. Computer science was not exist when Nobel decided on prizes, the equivalent is Turing Awards, that, if I am not mistaken, have been already awarded to three Israeli scientists, so we are not bad in this area either..

Back to event processing thinking --- getting progress on the EPIA book, yesterday we had two hours review with the editor on editorial stuff around the last three chapters, and upon revision, it will get to the 2/3 review. Our target date for publication is now April 2010, and this will probably be the final target.

Anyway, to continue my previous posting on contexts, I would like to discuss the notion of context composition. Recall that context is grouping of events based on one of the following: time, space, state and segment, for either grouping in order to apply operations on this group, or make distinct behavior for distinct groups. Composite context -- as its name suggests is just a multi-dimensional context, i.e. it contains cross-section of several contexts.
In the picture below this is a composition of segmentation ("per customer") and time (in this case fixed sliding temporal interval - per hour, each square is a context partition.


This is just example of combination, there are other useful compositions such as: spatio-temporal context: partition with one dimension is time and one is location oriented, or space and state contexts combination, example: spatial context == within the city of Trento, state of weather = {sunny, cloudy, rainy, snowing}, in each of these partitions other agents are applicable.

Somebody asked me -- what is the benefit of using the context abstraction anyway --- the answer is, like any other abstraction -- it saves work. The same application can be written with much less code and is much simpler to develop, maintain and understand --- the use of context is quite useful in this sense, see also some discussion by Marco. Next -- I'll write more about the next chapters on event processing patterns, stay tuned.

More on states and event processing

During my last visit in the USA I have purchased and now reading the book "the last oracle" (nothing to do with the software company that bears this name), one of the thrillers that are based on some history and span around the world, exposing some scheme that will bring disaster on the universe. It seems that this type of books became popular recently.

Anyway, I wanted to come back to the issue of states that I have recently blogged about. Actually the term state in event processing are somewhat overloaded. A state may refer to any of the following:

• The internal state of a single agent -- when trying to detect a pattern (say - E1 and E2) there is a need to keep the state, e.g. the fact that E1 already arrived but E2 did not. This can be kept either by keeping the input event as an object, keeping another object somehow derived from the original event, or keeping some internal data structure that helps in the detection.
  
• An event store where events are kept for further consumption by other agents -- this is a global (or bounded) state.
• Reference data that is being used by event processing agents for enrichment, this data is not maintained by the event processing system, but can be considered as part of the event processing state, since the event processing results may depend upon values of this data.
• Global variables (whether persisted or used on a shared memory) that are being used as a global state across event processing agents.
• Context -- on which I have blogged several times, which may have temporal, spatial, segmentation and external state (which is indeed a global variable) dimensions.
  

All of them are in a way part of the system state, they have different roles, though.

BTW -- we have revised the building blocks that we are using in the EPIA book, by replacing event derivation (which is actually part of the event processing agent) with global state.



Among the types of states we mentioned before -- context is a first class citizen in the model, internal state is hidden inside the implementation and is not explicitly modelled (although some languages allow also to model the internal state explicitly) and global state contains all the rest (reference data, event store, global variables). More - Later.

On state based event processing

Early morning in Nashville. I arrived here yesterday, and after some rest, went to dinner with some of the people in the community (this was initiated by Matthew Cooper from Eventzero who invited some people). In the dinner I was sitting next to Dieter Gawlick from Oracle, one of the most active people in the community. Dieter is recently advocating the concept of "state based event processing", based on some work he is doing. Today I'll hear more about it in the use cases tutorial, and then in a later meeting with Dieter to get feedback from the use cases workgroup to the language analysis and reference architecture work groups. My initial impression is that in the term "state" he more or less means what we call "context". I have just written the first draft of the chapter in EPIA book defining and describing the term context (will be available on the Web within a few weeks I hope). It will be interesting to see if how the notion of context will further evolve. Most event processing languages have very modest support in context these days (e.g. time window as a simple case of temporal context). I'll write again about contexts and states in one of the next postings. Today -- the DEBS conference start, and the tutorial day (last year it was the most interesting day of the conference, let's see if we'll keep the standard high). Now -- going over my part in the event processing language tutorial, so I'll not mix up the English words...

On state processing and event processing

Yesterday, I got visited by my (now- Ex) Master student Elad Margalit, about his thesis regarding dynamic setting of traffic light policies I have written before. For some strange reason he decided that I deserve a gift for his graduation so he brought me a "flip clock" that looks like this. Strangely enough it switches the labels to show the correct time, all people who somehow got to my office yesterday thought it is a cool gadget, and it is now located in front of my eyes.

Today's topic is some echo to the discussion started by my friend and ex-IBM colleague Claudi AKA patternstorm on the forum in the complexevents site. Claudi has defined state as a sequence of events, while several others answered that this is not really the definition.

Before getting to definition, there was also very concrete motivation that Claudi mentioned -- if we equate state to "sequence of transitions" than state processing becomes a kind of event processing. I think that it is important to discuss this statement.

While state is not exactly a sequence of transitions, it is true that the value represented by the state can be reconstructed if we apply the series of transitions on an initial state, and considering that the initial transition is null and the first transition creates the state, we can obtain all information as part of series of transitions.

Let's take a simple example. The state represents the value of my balance in the bank checking account. The transitions start from a one that opens this account, going through deposits, withdrawals, commissions of the bank etc.. I have opened my current checking account in 1984. Assuming that I would like to process this state, such as getting an alert everytime that my account balance becomes negative (unlike the USA, in Israel overdraft is a common practice). I can make it an event processing activity by taking all transition from 1984 and reconstruct the state with each new transition, however, this is not an efficient way to do it, first I'll need to keep all the historical transitions forever, second, it is much more cost-effective to maintain the balance as an entity, and process it.

State processing and event processing are complementary, in states processing we are processing the snapshot of the present time, while in event processing we process the history of transitions. If I want to get alert on overdraft -- this is state processing, If a compliance officer looking for money laundering suspect is seeking if three deposits with more than $10,000 each were done to my account within a single week, he is doing event processing. In reality we need both, but each of them has other techniques for its cost-effective processing.

More on this topic -- later.