On Event Derivation

Today I have participated in the local IBM championship in Backgammon (in Hebrew we use the Turkish name "shesh-besh"), however, my participation did not last very long, I lost the first game, and did not move to the second round, thus, returned to my office... At least the one who won the game over me is the technician in charge of the video conference equipment, so if one wants to conduct video conferences, make sure he likes you, since he is the only one in the lab who knows how to operate this simple equipment.

Today I would like to write a little bit about "event derivation" - in one of the past postings I've used this illustration in order to explain relationships between terms. Why do we need all these terms? -- well - we don't, but they are "legacy" terms, thus it is a good to show the relations among them.


When saying "derivation" -- let's compare it to two other types of derivations:

• Inference -- where the input are: facts and inference rules, for example: fact 1 -- Jack is the father of Mike; fact 2 -- Mike is the fater of Alex; inference rule: X is grandfather of Y if X is father of Z and Z is father of Y. From this system we can infer that Jack is the grandfgather of Alex. This is a deductive inference, there are some other types of inferences, and can be obtained by logic programming, or inference rule engines.
• Data derivation --- although the term is broader --- if we'll take the relational database variation --- it takes a collection of relations as an input, apply query, and gets a relation as an output -- here there is no logical inference, but computation over data. Other variation of data derivation is a spreadsheet formula, typically performing some aggregation operator (sum, count, average...) on neigboring data in the table.
  

Event derivation is neither of the above, but has similar nature to both. The input and output are events, if we present it as functions then:

Inference: Facts X rules ----> Fact
Data derivation: Relations X Query ---> Relation
Event Derivation: Events X Event Derivations ---> Events

Now looking closely at event derivation. First -- why does event derivation occur ?

Event derivation may occur from various reasons -- while in mind it is closely related with pattern detection, it may also be associated with other event processing functions such as: enrichment and transformation.

Some examples:

Example 1:

Input event: Order that consists of the attributes (not including the header) -- customer, item, quantity.
Functionality type: enrichment -- add attributes as: customer-type, customer-reliability-rank to the order.
Output event: the enriched event. In this case the derived event is identical to the event that has been created by the main function (enrichment) and the derivation is just copying.
Output event occurs immediately when function is done

Example 2:
Input event: Order (same as first example)
Functionality: Aggregation -- quantity by item.
Output event: Item-Demand: Item, Quantity. Multiple events - one for each item.
Output event occurs at the end of the context ("deferred").
comment: here a composite event is being accumulated, and at the end of the context is decomposed to individual events.

Example 3:
Input events: Order (same as first), Item-Demand (Output of example 2).
Functionality: Detect Pattern, the pattern is: Quantity of a single order > 2/3 from quantity of the same Item.
Output event: Dominant-Order: Order-Id (from header), Item, Dominance Ratio = (Order.Quantity / Item. Quantity)
Output event occurs as soon as detected.

Example 4:
Input events: Temperature-Reading (every 1 minute).
Functionality: Aggregator/collector -- collects all reading of a certain hour and creates one event.
Output event: Every hour -- Composite event that consists of 60 raw events.

These are some examples. More formal definition of event derivation - later.

On relationships among: derived event, composite event, complex event and situation

This is the water park "Luna Gal" where I am planned to spend the late afternoon with my children (it is my wife's turn to have a business trip this week) as part of a fun day for employees, meanwhile I am again at my (air-conditioned) office with the morning coffee and the Blog. One of my somewhat neglected obligations that I am trying to catch up these days are writing some terms to the encyclopedia of database systems, since we are in the glossary era, I would like to answer a FAQ question about the relationship between four terms: derived event, composite event, complex event and situation. The first three are glossary terms, the last is not, but is being used a lot in the community jargon. Tim Bass has recently posted in his Blog a simple solution model which is originated in the experimental psychology research.


The following illustration clarifies the relationships:

Derived event is an event that is created as a result of a computational derivation process as a function of one or more events.

Complex event is an event that is an abstraction of other events.

Composite event consists of collection of events that satisfy some pattern.

These are not the glossary definitions but they are explanations of the definitions that will enable to determine the relationships. As we can see not every derived event is a complex event - a derived event can be just an enrichment of a single event, and as such not a complex event, on the other hand, complex event can be created by explicit grouping of events and not by computational derivation, thus a complex event may not be a derived event. The relationships between derived events and complex event are - partial overlapping.

On the other hand, composite event is a subset of both complex event (since it is a collection of event and as such an abstraction) and derived (since it is derived by a computational process). Interestingly composite event is not the intersection between derived event and complex event, since the intersection also include scalar derived events that aggregate the raw events but don't keep the collection of events. Thus composite event is a subset of the intersection.

How does situation fit in ? - well, situation is a different beast, it is not some syntactic operation on event, but it is a semantic abstraction that denotes -- something that requires reaction. An event of any type may designate or approximate a situation, for example: fraud suspicion is the situation and a certain pattern is an indication (either exact or approximate) to the occurence of this situation, when there is a determination that a situation has occurred - a derived event may be issued to designate this situation, and the pattern is the condition (again - exact or approximate) for the situation. Note that an event pattern may have some certainty measure (e.g. probability) for the fact whether it designates a situation , not all cases are deterministic.

Does every derive event represent a situation? --- not really. A derived event can be used as an interim event or part of dissemination system and not have semantic role of situation. The same is true for all other event types.

Last but not least --- Paul Vincent in his Blog reported on a tutorial he has given about the ETPS glossary (useful presentation, I'll ask permission to borrow it). He asks a question - whether event inheritence is derived event ? well - let's check the definition of derived -- creating an event as a result of computational process as a function of one or more events. Does it describe the inheritance process ? -- IMHO it does, thus I think that inherited event is a type of derived event, though the derivation process is somewhat different from the derivation process in other cases. more - later.

On the classification of events

I am returning to the area of "classification of events", although it has been discussed in this blog several times regarding certain aspects, the reason is - some discussion in which I realized that the confusion is still alive and kicking.

Events can be classified according to their structure, semantics, or pragmatics. This classification is orthogonal.

Based on Structure:

• Simple or atomic event - an event that reports on a single occurrence.

The event structure may be flat (like relational database), or semi-structured (XML) - which also can express non-flat structure (e.g. hierarchical). In some cases events are represented as objects. Typically events contain "header" - information about this event (event class, time stamp, source...), and "payload" - the content of the event.

• Complex event - an event that is composed of multiple events

There is some difference between this and the term "composite event" from active databases which mixes structure and semantic aspect, but I don't see it useful to use this term any more.

Based on source

• Raw event: Event that has been reported to the event processing system by an external event producer.

Note that a raw event can be simple or complex. The term "raw" is relative, since it can be result of some computation process in the event producer that is transparent to the event processing system.

• Derived event: Event that is created by the event processing system as a result of some processing.

Again, the derived event can be simple/atomic - where the attributes are computed as functions of raw and derived events, or complex - in which the derived event is a concatenation of events.

Based on ontology:

• Real event: An event that designates something that occurs in reality
• Virtual event: Hypothetical, simulated or uncertain event.

Again, this is a separate dimension - can be in any structure and by any source.

Based on pragmatics

• Reactionable event (AKA situation): An event that is consumed by an external consumer at the end of the event processing network (either notify or orchestrate).

We used the term "situation" for a phenomenon in the consumer's domain of discourse that the consumer wishes to react to. This can be raw event (and in this case the event processing system is reduced to "simple event processing" - i.e. filtering and routing) or derived event, it can also be real or virtual, and in any structure.

BTW - ECA - "Event-Condition-Action" rules are rules in which the events are situations, and they are executed within the consumer domain, but I'll continue to discuss the conceptual model in the next posting.

On simple events and simple event processing

This is a picture I have borrowed from Siemens, however, I'll use it talking about simple events and simple event processing. There is a constant confusion around the terms "simple" (and "complex") here due to the ambiguity of the phrase: simple (complex) event processing - does it mean: processing of simple (or complex) events? or does it mean simple (or complex) processing in events. In this posting, I am drilling down to the simplicity notion.

Let's start with simple event - I prefer to contrast simple event with composite event where the contrast is in the structure - composite event is a collection of simple events and simple event is an atomic event, nothing said about the processing yet.

Now - is there different processing for simple events and composite events ? - in principle no - there are some functions on collections that are not applicable for atomic events, but if we take a collection of simple events that has not been concatenated we can apply the same processing for them.

Thus - my preference is to attach the simple to the processing and not to the event type, and define simple event processing as simple type of processing, no matter what the event structure is. What are the characteristics of simple event processing ?

• processing is done on a single event - not looking on other events, but events are processed "one at a time".


• only types of processing possible are: filtering and routing


• filtering decides whether this event should be passed


• routing decides to whom this event should be passed

Basic pub/sub with filtering is simple event processing.

ECA (Event-Condition-Action) rule is also simple event processing -- this does not say that the event cannot be derived/complex/composite - but regardless of the history of the event, its structure, the reason why it is created, and its source the ECA rule still performs a simple event processing, and processes one event at a time, where the condition provides filtering, and the action is indeed routing to somewhere to do this action.

Furthermore, in many cases, the "simple event processing" is a preamble to the event processing network done by the event producer, or post-processing done by the consumer, however, it can still be part of the Event Processing Network.

More related concepts - in the next postings.