More on states and event processing

During my last visit in the USA I have purchased and now reading the book "the last oracle" (nothing to do with the software company that bears this name), one of the thrillers that are based on some history and span around the world, exposing some scheme that will bring disaster on the universe. It seems that this type of books became popular recently.

Anyway, I wanted to come back to the issue of states that I have recently blogged about. Actually the term state in event processing are somewhat overloaded. A state may refer to any of the following:

• The internal state of a single agent -- when trying to detect a pattern (say - E1 and E2) there is a need to keep the state, e.g. the fact that E1 already arrived but E2 did not. This can be kept either by keeping the input event as an object, keeping another object somehow derived from the original event, or keeping some internal data structure that helps in the detection.
  
• An event store where events are kept for further consumption by other agents -- this is a global (or bounded) state.
• Reference data that is being used by event processing agents for enrichment, this data is not maintained by the event processing system, but can be considered as part of the event processing state, since the event processing results may depend upon values of this data.
• Global variables (whether persisted or used on a shared memory) that are being used as a global state across event processing agents.
• Context -- on which I have blogged several times, which may have temporal, spatial, segmentation and external state (which is indeed a global variable) dimensions.
  

All of them are in a way part of the system state, they have different roles, though.

BTW -- we have revised the building blocks that we are using in the EPIA book, by replacing event derivation (which is actually part of the event processing agent) with global state.



Among the types of states we mentioned before -- context is a first class citizen in the model, internal state is hidden inside the implementation and is not explicitly modelled (although some languages allow also to model the internal state explicitly) and global state contains all the rest (reference data, event store, global variables). More - Later.

On Event Stores and Temporal Databases

I am an old-fashioned guy who carries handkerchiefs, like this one, anywhere he goes, it is handy for multiple usages, anyway - while in the past, all department stores in Israel carried handkerchiefs and it was quite a popular product, for some reason, it went out of fashion, and I have hard time to renew the inventory of handkerchiefs, and in this sense, I wish I could step for a minute into the past, buy two dozens of handkerchiefs and return. In the past, I have been involved in work around temporal databases and even co-edited a book in this area. Temporal databases had two major goals:

(1). Keep historical data, and enable easy retrieval of this data

(2). Enable to issue queries "as of" any point in time, i.e. issue query that takes into account the information that was available at a certain point in time (not as seen from "now") - again, returning for the past.

One may wonder why am I writing about temporal databases today, well - the issue of temporal databases is coming back when thinking about "event stores", I know that some of my database colleagues don't like the term "event store" or "event repository", since it does not include explicitly the word "database", but for me, using DBMS is just a possible implementation, while others, such as grid cache are also possible - but this is a topic for another discussion.

Anyway - why do we need an "event store" - in some cases we need to maintain historical events and use them, in some cases even apply pattern detection on past events. For auditing purposes we may also want to issue "as of" queries. Note that temporal representation of events can be done according to multiple temporal dimensions (see discussion about temporal dimensions of events). One of the characteristics of temporal databases are that they are "append only" databases, meaning: database records can be added, but not modified or deleted; modification and deletions are logical operators that create other instances, keeping the old ones. This is linked to one of the properties of events - immutability, which is actually a controversial property that still needs discussion about - in what conditions it is needed. Temporal databases seem to be a proper way to represent historical events.

Some concluding comments:

(1). Current DBMS do not support temporal databases as primitive, although temporal databases have been built as a second layer above them.

(2). Not all events need to be persistent for historical processing, this is a property of event-type, and its retention policies. Different events need to be persisted for different purposes.

(3). The issue of what language should be used to process "event stores" is also a matter of opinion, some believe that SQL is the answer (however, for some patterns it is an awkward language), there is an attempt to extend the SQL language with pattern extensions, here I will quote a wise person, Paul Vincent, who wrote in a footnote to this posting : This will be especially good news for those who like their SQL statements to run to multiple pages… Another option is to use on-line pattern language that is used for on-line patterns, and translated it to SQL (or one of its variations).

There are several issues that still need deeper discussion - but enough for today.