On the structure of event representation

Today I took the day off, and spent the afternoon and early evening in the "Achziv Park" seen above. My third daughter Hadas is in a few days trip to the Western Galilee, and today was a day where the families can join for falafel in the park, so I have travelled there with my fourth daughter Daphna, and spent some quality time with the girls, even played ball.

Earlier in the day, I spent a couple of hours in my favorite coffee shops, starting chapter 5 of the "Event Processing in Action" book. Writing a book is a major commitment, and since I am doing it in my (imaginary) spare time, this is quite a burden on my time. I had a lunch last week with Roy Schulte from Gartner who is also writing a book (together with Mani Chandy), and he also complained that the book writing messes up all his spare time.
I was asked about the scope of the book, well - the book concentrates on pure event processing. There are some are complementary technologies like -- image processing, text analysis, speech recognition, sensor networks, statistical reasoning, machine learning that can be used to automatically generate either events or patterns (in relative small number of applications at this point), as they are really complementary technologies, they are mentioned briefly in the advanced topics chapter, the main stream of the book is about event processing .

After this long introduction , I'll turn to write about some portion of chapter four of the book which we submitted yesterday to the author. Chapter four deals with the representation of events (meta-data). There are currently no standards about representation of events, thus, we have taken ideas from different directions, to form such a model. We are looking at typed events, thus, each event type has some information particular to event type. We distinguish between three types of such information: header, payload, and event to event relations.

• Header attributes provide information about the event - type, time granularity for the event, and times associated with the event (occurrence time, detection time), event identity and some others.
• Payload attributes provides information about the event content -- references to entities and other attributes
• Event to event relation provides information about semantic relations among events.

In subsequent postings I'll discuss each of them in depth. It is late -time to rest.

More On Event Representation

CAC Japan is the only Japanese member of EPTS, some of their team have participated in the EPTS 4th Event Processing Symposium. One of the missions they have talked upon themselves the task to bring the event processing ideas and concepts of event processing closer to the Japanese market, they have sent me the URL of their Japanese portal
and in specific they have translated the EPTS glossary to Japanese
I salute the initiative, we indeed need to reach out for more parts of the universe. I don't understand the Japanese part of the site, but trust the CAC team. Hope to have an opportunity to visit Japan some day.

The issue of translation from language to language reminded me about the still chaotic situation in the area of event representation on which I have blogged about a year ago.

Much of the existing event processing systems see the event representation as an internal issue and make their own decision about which structure should be supported: flat positional like 1-NF in relational database, tagged semi-structured (XML), or Java objects. Furthermore the "header" of an event varies. This reminds me a discussion a few years ago with some of my IBM colleagues who worked on standard for management events. They included "managed entity" and "severity" as mandatory attributes, since in their domain -- event is always a reported problem. We explained them that there are other types of events, e.g. money withdrawal. For this type of event there is no severity and no "managed entity". This has been a funny discussion since they did not really think that what we are talking about is event, however, in their domain, what they have done made perfectly sense. This leads to a thought that we may have several layers of attributes:

• Universal attributes that should be mandatory in any event --- there should be very few of them --- Marco from RuleCore has suggested in his recent Blog:
• Id - every event is unique and have an unique id.
• Type - Every event is of a specific type.
• Detection time - every event is detected at a specific time.
• Entity id - The entity that changed its state.

I agree to the first three, but not every event changes the state of a single entity, and in
some cases (e.g. derived events) it is not really important which entity changes it state, so I
would move the "Entity id" to the next category.

• Domain common attributes (which also may have several level of domain, sub-domain, sub-sub-domain etc...) -- this may include: event-location, entity that changed its state, time-zone, severity and much more.
• Event type specific attributes -- individual for each event type. Some of them may be reference to entities, some descriptive attributes.

There is even more interesting discussion about derived events - but I'll defer it to one of the next postings.

The Babylon tower and event formats

Still in the USA, after the OMG meeting in the Washington DC area, I got to the Boston area, and now I am in Burlington, MA, visiting the (former) Aptsoft guys. When driving abroad I am renting a car with GPS (see above to determine which one), and it typically gets me to where I want - however, it is still not totally reliable, in the last day it confused me twice, one yesterday night to find the hotel, it told me to turn left, and meant in the next turn, not in the current turn, but it did not say so, thus, I found myself back on the I95, and had to turn around at the next exit, and try again. Today it sent me to some shopping center instead of the Aptsoft site, and after I ignored it and found it - the local people told me that the GPS maps have the numbers of the street in the wrong directions (starting from the other end) - well, sometimes the technology is fine, and the weakest link is the data it uses.

Talking about data, one of the topics that were discussed in the OMG meeting about standards is the topic of -- semantic/structural standards for events. I have used the term "Babylon tower" in one of the earliest postings in this Blog and meant that we have Babylon tower of languages - like the original tower who separated the languages. However, there is another Babylon tower that relates to event format - syntax and semantics, and here the problem is even harder, since there are multiple formats in multiple domains, nobody even made an inventory survey. One of the presenters said (and it is true in some cases) that 80 percent of the cost of building EP application is to set up the events from the sources and transform them to a processable format by composing adapters (hand coded, or by using transformation engines). This is some of the domains that need more investigation, and perhaps we need a meta-data standard here.

On deleted event, revised event and converse event

First, congratulations to my student Ayelet Biger, who has successfully taken today her M.Sc. thesis defense exam. Ayelet's thesis topic has been - Complex Event Processing Scalability by Partition which deals with parallel execution of CEP logic, when there are complex dependencies among the different agents. I'll discuss this issue in one of the later postings - we still need to compose a paper for one of the coming conferences on this thesis. Ayelet is my 17th M.Sc. student that has been graduated (together with 5 Ph.D. students makes it the 22nd thesis exam). Most of the students have done theses on - active databases, temporal databases (my past interest areas) and in the last few years to event processing. Supervising graduate students is a great way to work on new ideas that I don't have ability to work on in my regular work, the only thing that is needed are three more hours in each day...

Today's topic is inspired by a recent blog that I have recently read by Marco Seiriö. Marco is one of the pioneers in EP blogging, I've started reading his blog in January 2006, when he started the blog as "Blog on ESP", however at some point his blog became "Marco writes about complex event processing", another evidence that the name ESP has disappeared. Anyway, in his Blog, Marco talks about event model, I'll not discuss event model today, but concentrate in one interesting point that Marco raises about "undoing events". This is indeed a pragmatic issue with some semantic difficulties. There are systems in which events can be deleted, and some actions can be triggered by the event deletion. However, event is not a regular data and cannot be treated as such, since event represents something that happens in reality, then conceptually events are "append only" - in database terms, one can only insert events, but not modify or delete them. Deleting events also blocks the way from the ability to trace decisions/actions or have retrospective processing of the events. So - when in reality we need to delete/undo/revise events:

1. when event is entered by mistake - typically not the event itself, but some details in the event attributes, we'll need a possibility to revise event.
2. when we wish an event no longer to effect the processing.
3. when the event itself expired or we'll not need it anymore, and don't need to use it in any other processing - including retrospective.

The first case is a revision case - if we are in an "append only" mode, then the way to do it is to enter another event, and have the possibility that it will override an existing event (or set of events) for the purpose of processing. Example: somebody sent bid for an electronic auction and realized that one of the details (say: the price he is ready to buy) is wrong, then he can add another bid that will override the first bid. Why not delete the original bid ? it may be possible that the original bid is already in process, and the overriding cannot stop this process, even if not, there is a possibility that for retrospective processing we'll need to reconstruct past state which includes the original bid (these considerations are actually not new, we have thoroughly discussed these issues within the temporal database community a decade ago when we (Sushil Jajodia, Sury Sripada and myself) edited a book about temporal databases research and practice

The second case is even more interesting, but similar in type of thinking, here we would like to eliminate an event from taking effect, this can be done by sending a "converse event" that reversing the effect of the event - e.g. cancel bid. The implementation problem is that this event, and maybe its descendant events may have being flowing all over the event processing networks, with some even getting out from the EPN with actions triggered, some in process, and some are part of a state, but have not been processed yet (e.g. since a pattern has not been detected yet). Theoretically there is a possibility to apply something similar to "truth maintenance system" in AI that includes also the action and compensate for all actions, but this complicates the system, so recommended only when it is critical to do it (I'll discuss such cases in another postings), when the event has not gone out from the EPN, it is still possible to stop it, most system does not provide a language primitive to do it globally in an EPN, and recently I have watched a concrete customer case, where they had to do it manually.

The third case is the "vacuuming" case - when an event is no longer needed (in agents' state, in the global state etc..), I never got deep into this issue, but thought intuitively that it is a relatively easy problem; however, when this issue has been discussed in the Dagstuhl seminar last year, the claim was that the general issue of event vacuuming is still an open question.

I'll stop here now -- spent enough time on this one... more - later

On Event Representation

Back to micro-oriented issue, and today I'll start discussion about -what's behind the definition of the event processing glossary and get to the issue of event representation. As the glossary says - event is something that happens in reality. We also tend to call "event" to the representation of this reality for the purpose of processing by a computer. This notion in event has in the glossary several aliases: event object, event message and event tuple. The various aliases are indications that the space of event representation is not uniform, some think about event as a message that moves around, some thinks of it as a tuple, which is part of a stream, and really the twin brother of a tuple in relational database, some think of it as an object with arbitrary structure (which may also be hidden). Obviously, there is no "universal event", and unfortunately, since in many cases, events are already given from the sources with their given formats, and the event processing designer has little to say about it, then a generic event processing system has to support multiple type of events, or have adapters that translate all types of event to some cannonic type of event (and typically both -- supporting some cannonic type of events and having adapters translating other types of events to the cannonoic type). Event can be structured, semi-structured (XML), and unstructured (the area of unstructured events processing deserves more focused attention). One of the questions is - whether there are common attributes that each event should have to enable event processing. In the data world - the answer is no - there is not a single attribute that must exist in all relations (besides the fact that each tuple should be a member of some relation - no floating tuples). For event processing -- there are some attributes that have been proposed as common attributes:

• Event-type
• Source
• Time-Stamp (or Time-Interval)

Let's look about the question - are they mandatory or not:

• The first question is whether each event is an instance of an event-type (or event-class). The glossary says - yes ! "all events must be instances of event-type". This seems reasonable, however, we may think of some exceptions - such as rare events that have not been classified.. I need to drill down on rare events in some other post.
• The second question is whether the source should be mandatory - again, this is desirable if we want to have lineage or tracing back actions/decisions, but there may be cases in which the source is indefinite, or we wish to hide the source (e.g. leaking of information).
• The third question is whether each event must have a time-stamp (or time-interval in case it happens over an interval - another area that needs more discussion) - the answer is that many event processing patterns are time related, and if we want to know which event occurs first, or if two events occurred within 5 minutes of each other, we need to know WHEN this event occurred in reality. However - in some cases it is not known, in other cases it is not really needed.

It seems that all common attributes are useful, but may be optional in some cases.

There are attributes that are common for types - such as probability for uncertain events, spatial coordinates for spatial events etc -- this is before relating to the content.

The content is determined according to domain related ontologies - and there is a lot of work today in different application domain or industry to define such ontologies. XML is the ontology language, and it has its own benefits, it also carries overhead relative to "flat" events in which the attributes are positional oriented and not keyword oriented.

Events also carry semantic information - such as: reference to entities in certain roles. In fact, event can be thought of a transition between one state to another and the information included in the event refers to a change in the universe such as:

what was changed ? what entities are affected? when it was change ? where did the change take place ? what other information is important about the change ?

This short discussion raised already several open issues that deserve further discussion - so I'll put these topics on the queue for further postings.... more - later.