On events vs. data - the semantic view

I have written in the past about events vs. data, talking about the various roles that data stores can play in event processing.   recently I had a discussion in the more semantic level., about the differences between events and data.   The first distinction is that data (and also objects) do not have semantics in general, since anything can be stored in data, however event has a semantic meaning: something that happens!.    So the best comparison is to semantic data models, and in the picture above you can see the most simple (and common) semantic data model - the entity-relationship model.    As seen in this picture, entity is a term that designates some common name for individual beings that are classified to that term.   Person, department, equipment and experiment show in the picture are all entities.   None of them represent events,  since event is a different creature,  as mentioned it designates something that happens.   An event typically references entities.  For example, the event - Paul sold the bike to Peter is an event of type "sell" (if this is a general auction domain)  or "bike sell" (if this is a bicycle store domain).    In this case the event references three entities: 

• Buyer:  entity of type person
• Seller:  entity of type person
• Item:   entity of type bike

Of course, in more general case the buyer and the seller can be corporates, and the item can be of any item type.    If we look at the description as a sentence,  the entity type is represented by a noun, while the event type is represented by a verb.    Note that not all verbs represent events though,  if we look at the picture above, the relationship  "requires" is a verb that designates that an experiment requires equipment, but it is not an event,  it is a static relationship among two entities,  so verb is not a sufficient condition for event, event represents a verb that indicate that something happened.     Models like the entity relationships model should be extended to include events.    

A simple example of agents and contexts

In the last few days, dishes in my house are washed by hand. Our dish washer have seen signs of old age, thus I acquired a new one. First step was to order delivery from the store. There is the software engineering principle of "separation of concerns", thus those who do the delivery don't install it. I am not allowed to install it myself, since if I open the package I am losing the warranty, only a technician of the service company related to the importer is authorized to open the package. After some coordination this technician arrived today, looked at our kitchen and said -- "this is very common, they have changed the standard, now there is a thermostat on the pipe, so the size of the hole that your need in the kitchen closet is larger". Since there is a separation of concerns, he does not do holes, and I had to call the Carpenter who came later today, so I'll be able to call the technician again tomorrow... This is the result of the fact that the event of "standard of pipe changed, thus it effects the size of the pipe" was not reported by the store, since they don't install dish washers so they probably don't know --- this separation of concerns, and not seeing event over a larger story, wasted a lot of time and money to the society...

Anyway -- enough complaining.

I got a question from Hans, related to my posting on context, I am copying the question here:

" I would be interested to hear about how the context concept works in a particular use case: Let's say I have events that contain, among other things, a number. I would like to capture the third quartile (a data point that sits immediately above 75% of the other points) of the number, every hour. I would then like to perform some operation over the previous 8 of these hourly numbers.

How would this be expressed as contexts and agents?"

OK -- here it is:

We start by event of type E1 that flows into the system from some consumer and has some numeric attribute A, for which we want to capture the third quartile.

• First we define a context C1 of type of sliding fixed temporal interval with both duration and period of 1 hour.
  
• Then we define an agent A1, which is valid within C1, and calculates the third quartile and produces event of type E2, with an attribute B which that captures the derived value.
  
• Further we define a context C2 that is a sliding event temporal interval on event E2, with event count period and event count duration both of 8.
• And finally we define an agent A2 that subscribe to events of type E2, and is valid with context C2, that does some operation on E2, and may in turn produce (say) event of type E3, which is flowing to some consumer.
  

This is the basic model. Of course, it is a bit more complicated, if our agent library does not contain an agent that calculate quartiles may not be a basic, this can be somehow combined from some combination of agents, since an agent can be recursive and include mini-EPN, but as such we can still model it as a single agent in the high level view.

The nice thing about this abstraction that it is quite simple to model such problems... More - Later.

On the "Return on Investment" in Event Processing

As part of the orders that I got from my physician, to which I am humbly comply, is to spend around one hour walking every day, if I have time I am doing it outside, if not I am doing it at home on an electric walker. Yesterday I walked around, not far from home, and decided to take a shortcut via a woodland, not far from home, but not a familiar one, I saw a trail that seemed to go up in the hill, where the top of the hill was supposed to lead me back to somewhere in my neighborhood, there was a split in the trail and I chose a random one, and after five minutes realized that it leads to a dead end, however, I did not feel like going down, so I continued to climb up and pave the way among bushes, fallen trees etc -- quite irresponsible of me, especially as it was getting dark outside -- after 40 minutes of wondering I saw the back yard of an house, navigated there, and got safe and sound indeed to somewhere in the neighborhood. Somehow felt as return to childhood -- but not for long.

Anyway, today I would like to write something about the "Return on Investment" in event processing.


Mark Palmer, the current CEO of Streambase, has recently blogged about the fact that CEP is not about "feeds and speed" but about "ease of use", it is actually refreshing to see it from a Streambase person, since in the past some Streambase people claimed that the only reason to use a CEP engine is due to its scalability properties. Actually I have written one of my first postings on this Blog, entitled "the mythical event per second" saying something about it. I agree that there are some applications that require to satisfy high throughput or any other QOS metrics as a crucial requirement, but this is a secondary ROI type. The major one is providing abstractions that reduces the cost of the development and consequently the maintenance of event-driven applications. This is similar to what the DBMS discipline provides us -- as a grey-bearded old timer who is not completely senile, I still remember the times we have worked with file systems, DBMS provided many abstractions that makes the data oriented applications much easier to develop. The same goes for event processing, I am constantly saying to people who ask -- is there something new in event processing ? the answer is -- not really, event processing were hard-coded within regular programming for ages, however, since traditional programming languages and environments were not created to process events, the manual work required is quite substantial. The reduction in cost relative to hard coding can be substantial, and some customers have estimated it in 75% reduction. It will be interesting to do an empirical study about it, probably a challenge for our EPTS use case work group. More about ROI -- at later posts.

On Semantic Event Processing

This nice castle seems European, but it is really located in Tarrytown, NY. I am not living in castles, but in a hotel in the small town of Tarrytown that has many hotels. After a busy week - starting in Pasadena, visiting Mani Chandy in Cal Tech, spending three days in IBM's Impact 2008 and visit the IBM research center in Hawthorne yesterday, and before going home this evening, I have some time for Blogging. Catching up with the recent Blogs I have found several ones that has gone semantic: Jack Rusher from Aleri, Marco from Rulecore, Paul Vincent from TIBCO all have written about "semantic CEP", with the idea to use domain dependent onthologies. The idea to use semantic relations between entities as part of event processing is something we played with a few years ago (during my previous tenure in IBM research) and I believe it has a big future, certainly when the trend is to move parts of the development to the business user, this can certainly help. So - this is certainly in our road map. Will write more on this topic later.

The notion of context and its role in event processing

I have copied this pyramid from an academic project site from Monash University in Australia that explores the notion of context in pervasive computing
It represents quite well the location of context - between the raw events ("sensory originated data" in this picture), and the derived events ("situation" in this picture).

The rationale behind this term is that under different circumstances, we identify different situations from the same event or combination of events. This can be temporal : "during working hours" we detect different situations than "during off-working hours" or "as long as red alert is in effect, it can be spatial : "if it happens within the county limits", or it can be entity-driven such as "if it relates to platinum customers".

Is context a first level abstraction ? - not in SQL oriented languages, where the "where" is semantically overloaded, although there is a limited support to temporal contexts with the notion of "time window" (which is usually just a time offset). Some works of context support in rules exist, though not in the main-stream products in this area. However, the benefits of getting context to be first class citizen:

• It is a first-class semantic abstraction: people think in contexts.
• It provides computational efficiency. E.g. in event processing network - context can determine the partition of agents, each agent does "function within context" (i.e. detecting a "complex event processing pattern", thus events are routed only to the agents that meet their context requirement.

I'll continue to discuss contexts in later blogs.