On the Internet of EVERYTHING

Cisco came out recently with the concept of  "The Internet of EVERYTHING".

While the "Internet of Things" deal with connecting anything to the Internet, the Internet of EVERYTHING deal with the things and the semantic connection among things that can make the world actionable in real-time.  A simple example is the car theft example. 

A car is connected to the Internet through its GPS sensor that reports its location, it has also semantic relations to a list of eligible drivers who are permitted to drive in this car, each of these persons is also connected to the Internet using his or her mobile phone, thus the Internet knows the person's location  (disregard the privacy issue for this scenario!),  so if the car is moving (inferred from the GPS change in location), and all the eligible drivers are not in the car - it means that the car is stolen, and it can then report to the police and have them track its location.  

This scenario is based on - things,  contexts of things, and processing events about things..   It is actually quite straightforward from technology point of view.  I wonder if the "Internet of EVERYTHING" will survive the buzzword test of time... 

Putting event-driven rules in context in real life

We are often talking about the fact that event processing is done within a context,  but what about event-driven rules in real life (not in computerized systems)?


Earlier this week there was an incident that the engine of one of the cars in the parking lot of our lab was on fire (I did not take a picture of that incident, the picture you see above is just a car on fire I found in the Web).


There was an announcement to all employees to move the cars from a certain potion of the parking lot.


The rule that local administration has is:  When a fire event occurs,  shut down all elevators.  
However the one who composed the rule forgot to put it in context and say: when a fire event occurs in the building, shut down all elevators, and the rule has been activated.   The result was counter-productive, since people ran in the stairs  (we have an 8 story building).    


More context-sensitive rule would be helpful.   

DEBS 2010 --- tutorial days and our context related tutorial

This is King's College in Cambridge, a palace shaped building in which I am staying now, and in which the DEBS 2010 conference is being held. This nice picture was taken by Jacklyn Banergee and can be found is a website called Victorian Web. Cambridge is a very nice place and I took some time both yesterday evening and this evening to walk around. There was one thing in DEBS 2010 that reminded of DEBS 2008 in Rome -- it is July, the weather is quite hot, and this place is not very strong on air-conditioning (there is none in the guest rooms!)-- it seems that it is not an important part in the European culture.

The first day of the DEBS conference was yesterday - a PhD workshop, I have attended part of it, today was the tutorial day, I have attended two tutorials - the one about wireless sensor networks (the first half) and the second about event processing reference architectures - you can view the tutorial page of the DEBS 2010 conference for more details. We also had a tutorial that was delivered by Ella Rabinovich and myself on context related stuff. I have written about contexts several times in this Blog, and the tutorial provides some details about the roles of tutorial, in-depth look about tutorials, and implementation of tutorials in practice. I've uploaded the tutorial on slideshare, and you can view it from there, I hope that the rest of the tutorials will follow soon. Tomorrow - the paper presentation part is starting.

Some footnotes to Brenda Michelson's post on TUCON

Las Vegas is a place where companies like to do customer conferences; I guess that the quantity and sizes of hotels is a major factor. Last week IBM has done the big customer conference: IMPACT 2010, after a few years of attending this conference, I have skipped this one, so I did not cover it in the Blog. This week TIBCO has done its customer conference - TUCON 2010, naturally I have not been there either, as I am not a TIBCO user, but looked at some impressions on the Blogland, and came across Brenda Michelson's Blog describing the keynote sessions. With the disclaimer that my information on this is second hand (but a good hand !), I'll make some footnotes:

1. I believe in the direction that events will be more and more in the center of enterprise (and other applications); we see it happening and we'll see much more.

2. Like Brenda, I am not sure that the phrase "enterprise 3.0" will catch, at the past there was an attempt to push "SOA 2.0" but it did not catch either, I think the world became tired from recycling the n.0 phrase.

3. "2 second advantage" is catchy, and may be true for some applications, but in some of them 2 second might not be enough to make a difference. In general there are two aspects: immediate reaction -- react fast (or faster than others); and proactive --- react in time to mitigate or eliminate future event; I am not sure it that if the Interpol will know about a crime 2 seconds earlier it will be enough time to eliminate the crime.

4. About NoSQL and relations to event processing - I'll write a separate post, it is an interesting insight that NoSQL is gaining traction.

5. Context as additional dimension -- I have written several times in the past about contexts, and am planned to deliver a tutorial about contexts in DEBS 2010; I believe that contexts are starting to be a major factor in computing.

It is good to see that the event-centric universe is getting more traction. More -later.

More on spatial event processing

This is a map of my home city, Haifa, which shows it northern part, I live in the south part outside this picture. I had some internal discussions this week about the role of location information in event processing, this is somewhat similar to the roles that time has in event processing, though the use of time is more pervasive, we see increasing number of applications that require spatial support. Some of the major utilizations of location information are:

1. Location as a context: The location is used to group events together as they serve as input for any event processing function such as: aggregation of pattern matching.
2. Location as enabler for spatial patterns: The locations of events is used in the function that determined whether a certain pattern is being matched.

Note that these two roles of locations are orthogonal.

An example for the first role: count all the police vehicles that are within 5 KM from a crime scene, where there are periodic events that identify the location of the police vehicles using GPS devise. In this case, location is done to find out which events are "in context", and the event processing function itself is a simple aggregation (count).

An example for the second role: A certain company experienced over one week, 20 events of breaking into employees houses, it an an attempt to figure out whether this is a work of a single person, the pattern that is matched is: the maximal distance between two breaking in events that occurred within a single night is less than 15KM. In this case the context is not spatial, since the events involved are determined by time ("a single night") and segmentation ("events related to houses of employees"), but the pattern that we are looking for is spatial, since it involves location.

As said, time can also play in the context role, in the pattern role, in both, or in none.

More on spatial event processing - later.

On spatial relations between events and entities

During the final pass of revisions for the EPIA book, we have re-visited the issue of spatial context, one of the options of spatial context is to group or partition events according to the relationship between event and a certain entity. However, location is not necessarily a point, there are three main type of location: point, line (or poly-line) and area (polygon). This is true for entity, but it is also true for an event, if the event is located using GPS, then a point coordinate is obtained, but if an event is located using the signal sent from a cellular phone, then the location of the event is a cell, i.e. it occurs within an area. We can also say that a vehicle is somewhere on road M1, which is a poly-line. Given these three data types, we can define several relations among the location of the event, and the location of the entity, as shown in the illustration above, such as: contains --- the entity contains the event, disjoint etc... The illustration above shows the various cases between events of various location types and entities of various location types. Some examples of use cases for the various relations will follow in one of the next postings, meanwhile -- you can think about applications yourself

On reasoning about events and states

Yesterday, we watched in our local theater, the play "Amadeus". I have seen the movie when it was first launched, and another version of this play in the nineties. Still very impressive, and an opportunity to hear some of Mozart's good music.

Thanks to Paul Vincent's Blog, I clicked the link and got to Paul Haley's posting entitled "time of the next generation of knowledge automation". Paul Haley starts his postings by classifying four types of reasoning:

1. Reasoning at a point within a [business] process
2. Reasoning about events that occur over time.
3. Reasoning about a [business] process (as in deciding what comes next)
4. Reasoning about and across different states (as in planning)

The claim is that the first kind is being supported by business rules (called EDM in the original), the second kind is being supported by event processing, while the third and fourth kinds are not really supported by the state of the art. The third type requires getting the notion of context into reasoning while the fourth type requires getting cross state reasoning (which is different from cross event reasoning). I agree with the classification, and will write more in the future about the third and fourth types in depth. Actually, we also need reasoning that will combine the different types of reasoning as well.

On the ingredients of pattern definitions

In the previoius posting I started the discussion about the notion of pattern, and stated that it is a function that selects an event subset; continuing to drill down on patterns, I'll say a few words about what are the ingredients of pattern definitions, what information do we need in order to perform pattern matching in a well-defined way:

• Pattern type: which determines WHAT the pattern matching is looking for. There are a variety of pattern types, and I'll dedicate a posting to the pattern type I collected so far. Some examples: sequence (temporal pattern), moving north (spatio-temporal pattern), trheshold pattern (e.g. related to average of some value over a set) and more.
• Participant set: the set of event types whose instances issue the pattern matching set.
• Context: the context to which the pattern is associated with (actually the agent executing the pattern is associated with).
  
• Pattern assertion: Some patterns have assertions associated with them. Assertion can determine if events are relevant (e.g. if we are looking at a sequence of two events, say event of type E1, and event of type E2, where in order to do a pattern matching we require that E1.A > E2.B, where A and B are names of attributes. There are also some pattern associated with certain pattern type, e.g. if the pattern type is a threshold pattern than there is an assertion associated with it, e.g. Average (e1.A) [over each context partition] > 40.
• Pattern policy: Pattern policy determines when a matching set is going to be activated, how many times, can event count for more than one pattern, how repeated events are treated and more.
  

This was only in title level, and in the next postings I'll provide more information about pattern types and pattern policies.

On composite contexts

Today is a happy day to the Israeli scientific community it was announced that Professor Ada Yonath from the Weizmann Institute won a Nobel prize in chemistry, this adds to two Israeli scientists that won a Nobel prize in chemistry a few years ago, so Israel is a chemistry super-power. Computer science was not exist when Nobel decided on prizes, the equivalent is Turing Awards, that, if I am not mistaken, have been already awarded to three Israeli scientists, so we are not bad in this area either..

Back to event processing thinking --- getting progress on the EPIA book, yesterday we had two hours review with the editor on editorial stuff around the last three chapters, and upon revision, it will get to the 2/3 review. Our target date for publication is now April 2010, and this will probably be the final target.

Anyway, to continue my previous posting on contexts, I would like to discuss the notion of context composition. Recall that context is grouping of events based on one of the following: time, space, state and segment, for either grouping in order to apply operations on this group, or make distinct behavior for distinct groups. Composite context -- as its name suggests is just a multi-dimensional context, i.e. it contains cross-section of several contexts.
In the picture below this is a composition of segmentation ("per customer") and time (in this case fixed sliding temporal interval - per hour, each square is a context partition.


This is just example of combination, there are other useful compositions such as: spatio-temporal context: partition with one dimension is time and one is location oriented, or space and state contexts combination, example: spatial context == within the city of Trento, state of weather = {sunny, cloudy, rainy, snowing}, in each of these partitions other agents are applicable.

Somebody asked me -- what is the benefit of using the context abstraction anyway --- the answer is, like any other abstraction -- it saves work. The same application can be written with much less code and is much simpler to develop, maintain and understand --- the use of context is quite useful in this sense, see also some discussion by Marco. Next -- I'll write more about the next chapters on event processing patterns, stay tuned.

A simple example of agents and contexts

In the last few days, dishes in my house are washed by hand. Our dish washer have seen signs of old age, thus I acquired a new one. First step was to order delivery from the store. There is the software engineering principle of "separation of concerns", thus those who do the delivery don't install it. I am not allowed to install it myself, since if I open the package I am losing the warranty, only a technician of the service company related to the importer is authorized to open the package. After some coordination this technician arrived today, looked at our kitchen and said -- "this is very common, they have changed the standard, now there is a thermostat on the pipe, so the size of the hole that your need in the kitchen closet is larger". Since there is a separation of concerns, he does not do holes, and I had to call the Carpenter who came later today, so I'll be able to call the technician again tomorrow... This is the result of the fact that the event of "standard of pipe changed, thus it effects the size of the pipe" was not reported by the store, since they don't install dish washers so they probably don't know --- this separation of concerns, and not seeing event over a larger story, wasted a lot of time and money to the society...

Anyway -- enough complaining.

I got a question from Hans, related to my posting on context, I am copying the question here:

" I would be interested to hear about how the context concept works in a particular use case: Let's say I have events that contain, among other things, a number. I would like to capture the third quartile (a data point that sits immediately above 75% of the other points) of the number, every hour. I would then like to perform some operation over the previous 8 of these hourly numbers.

How would this be expressed as contexts and agents?"

OK -- here it is:

We start by event of type E1 that flows into the system from some consumer and has some numeric attribute A, for which we want to capture the third quartile.

• First we define a context C1 of type of sliding fixed temporal interval with both duration and period of 1 hour.
  
• Then we define an agent A1, which is valid within C1, and calculates the third quartile and produces event of type E2, with an attribute B which that captures the derived value.
  
• Further we define a context C2 that is a sliding event temporal interval on event E2, with event count period and event count duration both of 8.
• And finally we define an agent A2 that subscribe to events of type E2, and is valid with context C2, that does some operation on E2, and may in turn produce (say) event of type E3, which is flowing to some consumer.
  

This is the basic model. Of course, it is a bit more complicated, if our agent library does not contain an agent that calculate quartiles may not be a basic, this can be somehow combined from some combination of agents, since an agent can be recursive and include mini-EPN, but as such we can still model it as a single agent in the high level view.

The nice thing about this abstraction that it is quite simple to model such problems... More - Later.

On Contexts

Sometimes I feel out of context, but today I would like to return to the notion of context, which is the topic of chapter 7 of the EPIA book. I have discussed the notion of context first time almost two years ago, and since then mentioned it in some other postings; so now it is a good time to talk more about it. A context, in essence, is a grouping of events that are processed together. This grouping may be: temporal, spatial, state oriented, or segmentation oriented, as seen in the figure below.


Temporal context may be fixed interval, interval that start and end in the occurrence (or detection) of events, and also sliding interval, according to time, or number of events. Spatial contexts can relate to fixed location, to distance from an entity, and to distance from the location of event occurrence, state oriented refers to external state (e.g. weather), and segmentation oriented, refers to partition of the event space according to some assertion (e.g. simple partition can be: platinum customers, gold customers and other type of customers). A context can be composite such as: spatial and temporal together.

What is the role of context in event processing ? first - an event processing agent lives within a context, in different contexts, the same event can be processed by different event processing agents. Second - it determines which events are processed together as input for the same pattern detection (AKA "complex event processing"). For example, if we are looking for the behavior of a single customer within a single day, then this is a combination of a temporal context (day) and a segmentation context (partition by customer), and each of them is processed in looking for a distinct pattern. This is a description of contexts in a nutshell. Some languages support kinds of temporal contexts ("window" or "sliding window"), some have more developed notion of context, and some have none. I'll write more about context types in one of the next postings.

On event channels

Last week, the Disney Channel arrived to the Israeli cable system and enriched the set of already existing children channels; so speaking about channels, it is a good time to discuss another type of channel -- an event channel, which is discussed in chapter 6 of the EPIA book draft. Some people view channel as an edge in the event processing graph, but we view channel as a type of node, since it has some processing associated with it. We define a channel as a processing element that receives events from one or more source processing elements (We refer to EPA, producer and consumer as processing elements), makes routing decisions, and sends the input events unchanged to one or more target processing elements in accordance with these routing decisions. Note that like the term Event Processing Agent, channels are abstractions and can be implemented in various ways (e.g. through messaging systems, through buffers, through persistent stores etc...). Channels are classified according to their routing schemes. Some of the common routing schemes are:

• Fixed routing scheme: The channel has predefined input terminals wired to predefined processing elements, and predefined output terminals wired to predefined processing elements. Every event that is received on any input terminals is sent to all output terminals. Note that this type of channels can be defined implicitly.
  

• Subscription-based: EPAs or consumers can subscribe to the channel dynamically. The routing decision is determined according to the list of subscribers that is valid at the time that a decision is made.
  

• Itinerary-based: The sink's input terminal identifier or identifiers are obtained from some attribute in the event's payload, this is used to send an event to a specific consumer instance, when the EPN node is the consumer class.
  

• Context-based: The channel makes routing decisions based on the context to which the EPA belongs. This is applicable for pattern detection ("complex event processing") type of EPA. The channel selects the appropriate run-time EPA based on the context defined in the pattern- I'll discuss contexts in length in one of the next postings, as this is the topic of the next chapter in the book.
  

• Type-based: The channel makes routing decisions based on the event type of the event that is being routed.

• Content-based: The routing decision is based on the event's content, this can be phrased as assertions, rules, decision trees or decision tables, and are based on the input event content, as well as context information.

This is just the basic definition, in one of the next postings I'll show example of how all these concepts fit together.

On temporal aspects of event processing

In the past I was involved in work on temporal databases, in the picture you can see a 1998 book about temporal databases that I co-edited with Sury Sripada and Sushil Jajodia. Although there were some attempts to create substantial extension to SQL with temporal capabilities, and move temporal databases to the mainstream. This did not work, and there are several reasons, the event processing area provides a second chance for these idea to come to the mainstream now, as event processing have strong relations to temporal issues. Bob Hagman from Aleri (former Coral8) has recently written some survey of implementation alternatives related to time aspects in the Aleri Blog. In the DEBS 2008 language analysis tutorial we had dealt quite briefly with the topic of time. Earlier this year I have written a chapter in the upcoming book of the book "Handbook of Research on Advanced Distributed Event-Based Systems, Publish/Subscribe and Message Filtering Technologies; edited by Annika Hinze and Alejandro Buchmann"

This chapter is entitled: "Temporal Perspectives in Event Processing".
Here is the chapter's main topics:

• Temporal dimensions: in temporal databases we dealt with the temporal semantics of a collection of snapshots (states), in event processing we deal with the temporal semantics of events (transitions). Are the temporal dimensions the same ? do they have the same semantics ?
• The "instantaneous" issue -- do event occur over a time-point or an interval, and if it is interval what does it mean from computational point of view ?
• Time granularity -- in temporal databases we introduced the term "chronon" which stands for the time granularity that makes sense for a particular use. This idea is also applicable to event processing, for different events, different chronons make sense.
  
• Temporal contexts: the term "time window" in stream processing is a kind of a temporal context. What kinds of temporal contexts are required, and what is the computational implications of them. I'll write more about contexts soon, as this is the topic of chapter 7 of the EPIA book.
• Temporal patterns: "complex event processing" is about finding patterns among collections of events; some (but not all) of these patterns are temporal in nature -- what are the temporal oriented patterns ?
• Temporal properties of derived events: An event processing system derives events as result of its processing. What is the time properties of the derived events? this is a rather tricky question that deserves a discussion.
• Ordering events: for some temporal patterns, knowing the order of events is important. What are the issues associated with keeping such an order, how out-of-order events should be handled ?
• A related issue is "retrospective events" -- what happens if events that relate to the past are detected, where the assumption that they did not occur already triggered some processing ?

• Issues of time in distributed environment -- clock synchronization, time-zone handling, time validity for mobile clients --- are all applicable for event processing.

As written, this is an outline of topics surveyed at that chapter, I'll write more about some of them in the future.

On state based event processing

Early morning in Nashville. I arrived here yesterday, and after some rest, went to dinner with some of the people in the community (this was initiated by Matthew Cooper from Eventzero who invited some people). In the dinner I was sitting next to Dieter Gawlick from Oracle, one of the most active people in the community. Dieter is recently advocating the concept of "state based event processing", based on some work he is doing. Today I'll hear more about it in the use cases tutorial, and then in a later meeting with Dieter to get feedback from the use cases workgroup to the language analysis and reference architecture work groups. My initial impression is that in the term "state" he more or less means what we call "context". I have just written the first draft of the chapter in EPIA book defining and describing the term context (will be available on the Web within a few weeks I hope). It will be interesting to see if how the notion of context will further evolve. Most event processing languages have very modest support in context these days (e.g. time window as a simple case of temporal context). I'll write again about contexts and states in one of the next postings. Today -- the DEBS conference start, and the tutorial day (last year it was the most interesting day of the conference, let's see if we'll keep the standard high). Now -- going over my part in the event processing language tutorial, so I'll not mix up the English words...

On Event Processing Building Blocks

Back to work for one day in the office, with five conference calls (one with Germany, one with France, one with UK, and two with USA...) and then back to home for the weekend. When I have free time I like to read books, the current book I am reading is "A Lion Among Men", 0ne of the books of Gregory Macguire, who writes stories that take as background famous children stories (in this case - the Wizard of Oz), actually this is the third one behind the scene of the Wizard, now taking the Lion as its main character. I have another book of the same author still waiting...

We also submitted the draft of chapter 3 of the "Event Processing in Action" book to the publisher, which hopefully be posted on the MEAP site soon.

The approach we have taken in the book, as I have written before, is to use the "building block" approach, describing event processing principles, and the use case whose construction demonstrates the application, using building blocks, which are like the chemical elements. The application itself is being built by using "definition elements" which are like atoms (my partner for writing this book, Peter Niblett, has come with the analogy from the world of chemistry). we believe that this is the right approach to teach what event processing is -- in the "deep dive" part of the book we dedicate a chapter for each of the major seven building blocks and then dive deeper into the types of event processing agents (which deserves a different discussion). We'll also provide samples of how each building blocks is realized in different models.

The seven building blocks are:

• Event type: defines the event schema
• Even producer: the projection of the event producer over the event processing network (note that the event producer itself is outside the scope)
• Event consumer: same -- the projection of the event consumer over the EPN.
• Event channel: the glue that holds the EPN together
• Event processing agent: the brain that does the entire work; each agent is doing a specific task of processing.
• Context: the semantic partition of events and agents
• Event derivation: A building block that is possibly part of each EPA that specifies the derived event.

There are some more building blocks that are used to support these ones, but our claim is that this set of building block is what needed to build an event processing application.

Chapter 4 which is in advanced phases of being written starts the deep dive by discussing the event type building block, and in one of the next posts I'll say more about it.

EPDL expression for the "on off windows"

Yesterday, I drove my 11 year old (youngest) duaghter, Daphna, with four of her friends (all boys), 65 kilometers to the nearest branch of Max Brenner (see pictures above) a chocolate store which also has a restaurant, most of the menu is the chocolate oriented menu, but for parents there is also some real food. They still did not get to Haifa, but I think that they have brnaches in New York. Anyway, we found the nearest branch and got there...

I have posted an explanation about the approach to the "on off window" that Pedro posted; somebody asked me beyond the explanation -- how this scenario can be expressed in the EPDL meta-language that we are working on. Actually this is a very simple scenario, it is expressed like this:

Let E be the event type; with two attributes param1 and param2 (as given); we also need to route the event e to the agent a, and the output event to a producer - lets define two output pipes, p1 and p2, both of them carry events of type e.

We define:

• Context, name = c, type = temporal, initiated-by = event e with param1 = 2, terminated-by = event e with param2 = 0.

• Agent, name = a, type = filter, within-context = c, filter-condition = all, input = p1, output = p2
  

Explanation: Context determines which subset of the "event cloud" are applicable, there may be multiple agents within a context, but each agent has a single context. In this case - we can use the simplest type of agent - filter - with a trivial filter-condition - "all". Agent a will not be active before cotnext c will be active. With the event that satisfies the condition, the context is opened, thus agent a receives input, does nothing with it, and post it as output.

This is of course very simple case -- we can also concatenate all selected events to one composite event; we can also use more sophisticated agent e.g. pattern detection.

More educational material about languages -- later.

On contexts and separation of concerns - the case of "On Off Windows"

The window in the picture cannot be openned or closed without some fixing, which is typical to some of the event processing language that support in fixed window. I have been quite amused in the last couple of days to read the discussion in the CEP-Interest Group on Yahoo!, started by
a query posted by Pedro Bizarro.
If you do not subscribe on this group I am copying Pedro's query:

Hi there,

A company contacted me to solve a query puzzle that they have. They
are reading incoming tuples and want to select all tuples that are
between two tuples. For example, assume that I want all tuples between
the first tuple with PARAM1=2 and the first tuple after this one with
PARAM2=0. This would select all tuples with time between 3 and 8 in
the example below.

TIME | PARAM1 | PARAM2
1 1 0
2 1 1
3 2 2 <== ON
4 2 3
5 3 5
6 3 4
7 2 3
8 2 0 <== OFF
9 3 1

I have tried a number of event processing engines but cannot write
such a query. (However, I can write the query on a table on Oracle 11g
using Oracle's Analytic functions with the help of a simple PL/SQL
function.)

Do you have a solution for the OnOff Window puzzle?

Thanks!
-pedro



You can browse and see the variety of answers given by different vendors - my intention is not to analyze the results, but look at the semantics of what is asked by Pedro. Pedro is talking about window, typically window is used in order to select a collection of events and perform some operation on them, in Pedro's case the operation is trivial - just select all of them, but we can slightly change the requirement and change it to "find if a pattern P is satisfied in this collection of events" (example of a pattern -- there are at least two events in which PARAM1 > PARAM2 for the same event) and if this pattern is detected perform some action (e.g. send me SMS).

Here we have three types of conditions:

1. Condition for opening the window
2. Condition for closing the window
3. Condition for detecting a certain pattern (in this case the result of the detection is a "situation" since it triggers action outside the event processing network.

Condition 3 is a normal type of "pattern detection" processing, however, conditions 1 and 2 have different roles -- they are used in order to decide which events will be used as input for the pattern detection operation. One of the answers was given by my colleague from IBM Haifa, Guy Sharon, who talked about the AMiT implementation of "lifespan" as a temporal context. Again, without going to the particular language, I would like to point out that the principle of "separation of concerns" --- have distinct separation between conditions to select the events that participate in an operation and the operation itself provides better clarity to the end result. Since a window is indeed a subset of the term context on which I have blogged several time before and pointed out about the importance of looking at context as a first class citizen in event processing. Context can have several dimensions -- the temporal dimension of context subsumes the notion of window, and there are other dimension (e.g. spatial - that selects events based on their location).

Bottom line: my belief is that the consumability of event processing will increase if we'll succeed to make them available to semi-technical people and have clear semantic roles to the building blocks instead of trust the developers that they will find a way to hack a bit in order to satisfy requirements -- more on this topic -- later.

On the rain in the window -- windows and temporal contexts

I realized that I have not written for a while, I am not out of topics, just trying to do too many things in parallel... Anyway, I am typically late in changing from summer clothing to winter clothing relative to most others, but it happened yesterday, maybe the noise of the heavy rain in the window, brought me to change from short shirt and sandals to long shirt and shoes.
Winter is a relative term, people who live in some climates, will not call our winter as winter.

Last night I attended the conclusion session of "students exchange trip" in which my 13 years old daughter Hadas has participated, they visited a school in Foster City, California, this is a plan called "ambassadors", and they had also to give speeches about various aspects on Israel, one of their challenges was to convince their host to come to Israel as a counter-visit. Since the international media create the preconception that Israel is a dangerous place to be, with wars in the streets etc.., some people (typically those who have never been here) are afraid to come... It seems that the children were successful to convince that in Haifa we live normal life, and there is no war in the streets... Actually I am used to people I ask me, that I feel much safer in Haifa then in New-York, London, or Paris. Paris is the only place I was attacked by thieves, so it is the most terrifying city for me.


Back to the rain in the window. The notion of "window" that came from stream processing, is used to process a sub-stream that is bounded by time (or by number of occurrences). In some cases a window can be specified by some starting time and duration, or slide at certain time intervals, however, in other cases we need to process events in a time interval "while it is raining" - this is done either to find certain patterns that are only relevant in raining time, or use the stream processing classic application --- aggregate within a sub-stream. In any case, this is not determined by fixed time, and the duration is not known in advance. This can be either "while something is in state S" or a time interval that starts by the occurrence of event E1 and ends by the occurrence of event E2. An interval may also expire if the state lasts too long...


I'll re-visit the notion of context and its formal definition soon.

On semantics of synonyms

Holidays time - and I had to spend time in replacing my home wireless router (not of the same firm that is shown in the picture) who did not work well - to get another expensive one, whose signal is not received well in the lower floor well -- I hate wasting my time on restore things to work, this is never-ending story, every time something else need fix... we need more robust appliances..

Well - back from my personal frustrations to event processing thinking. I have posted a couple of postings about the different possible interpretations of "event_1 occurs before event_2", however, different interpretations are not unique to temporal issues, and the specific anomaly of having different temporal semantics. Let's take a case in which time does not matter - the function defined this time as follows:
Detect a pattern that consists of conjunction of two events (order is not important) - e1, e2.
e1 has two attributes = {N, A}; e2 has also two attributes = {N, B} ; the pattern matching is partitioned according to the value of N (on context partitions I'll write another time).
For each detection, create a derived event e3 which includes two attributes = {N, C}; E3 values are derived as: E3.N := E1.N ; E3. C = E1. A * E2. B.

Let's also assume that the relevant temporal context is time-stamps = [1, 5] - and the events of types E1 and E2 that arrived during this period are displayed in the table below:

The question is: how many instances of event E3 are going to be created, and what will be the values of their attributes? Think about it -- I'll discuss it next week.

On Event Processing Network and Situations - the semantic bridge

One of the challenges in building a semantic meta-language that captures event processing behavior is to bridge the gap between terms that are in different domains. I have written before about Situations following the CITT meeting, in which this term has been discussed. In fact, we have used the term "situation" in AMiT , we also called its core component the "situation manager" We used the term situation to denote a meta-data entity the defines combination of pattern and derivation, but I must admit that this has been a mismatch of terms, although there is a strong correlation among these terms.

First - let's go back to the operational world, in this world events are flowing within event processing networks, in the EPN there are agents of various types: simple (filtering), mediating (transformation, enrichment, aggregation, splitting), complex (pattern detection) and intelligent (uncertainty handling, decision heuristics... ), in the roots of the network there are producers, and in the leaves of the network there are consumers. This is an operational view of what is done, the term "situation" is in fact, not an operational term, but a semantic term, in the consumers' terminology and can be defined as circumstances that require reaction.

How can we move from the operational world to the semantic world - we have two cases here:

• Deterministic case: there is an exact mapping between concepts in the operational world and the situation concept;
• Approximate case: there is only approximate mapping.

In order to understand it, let's take two examples:

Example 1 - toll violation (determinstic, simple)

• The cirumstance that require reaction is the case that somebody crosses a highway toll booth without paying (in Israel the only toll highway is completely automatic, but we'll assume that it occurs elsewhere, and that the toll is not applicable between 9PM - 6AM and in weekends).
• Getting it to the operational domain - there are two cases - one: go in the EZ pass lane and don't pay, two: go in a manual lane and somehow succeed to cross the barrier (obstacle?) without paying.
• The action in both cases: apply camera to capture the license plate, SMS the picture to officer on duty in the other side of the bridge.

From the EPN perspective, we have events of car cross a certain section of the road (the raw event), the EZ pass reading is an attribute of this event, and if no EZ pass it gets a "null" value. There is context information which is --- temporal (it is in hour and day where toll is in effect), spatial (the location of EZ pass lane), note: sometimes people mix the notion of context with the notion of situation, I have explained the difference in the past. Within this context a filter agent that looks for null value in the EZ pass reading is applied, if the filter agent evaluates to true then the situation phrased has been applied in deterministic way, and indeed the edge going out of the filter agent is going directly to a consumer (the camera snapshot may or may not be considered as part of the EPN). This is a case of "simple event processing", stateless filtering, whose output is a situation. This gives a counter example to the misconception that situation is closely related with complex event processing [I can continue the example to the other case, but I think that you got the point by now]

Example 2 - Angry Customer (approximate, complex)

The setting is a call center, the situation is -- detect an angry customer -- refer him or her to the "angry customers officer".

Here the life is not that easy, a human agent can detect angry customers by the tone of their voice (or electronic message), but this does not include all cases of angry customers, so we can look at some pattern saying -- a customer that applied 3rd time in a single day is an angry customer, and then we need to have a "pattern detection" agent that detects the pattern "3rd instances of application" where the context partition refers to the same date, same customer, same product. In this case also a leaf edge is mapped to a situation, but there are two differences from the previous case:

1. The agent is now complex event processing agent since it detects pattern in mu;tliple agents;

2. The edge represents the situation in an approximate way, which means that it can have false positives (the CEP pattern is satisfied but the customer is not really angry, just asked for a lot of information to install the product), or false negatives (the customer called twice, and does not talk in an aggressive tone, yet he is furious).

In some case it also makes sense to associate "certainty factor" with the leaf edge, approximating the measure of belief in the fact that this edge represents the situation. I'll leave the discussion about uncertain situations to another time.