In the previoius posting I started the discussion about the notion of pattern, and stated that it is a function that selects an event subset; continuing to drill down on patterns, I'll say a few words about what are the ingredients of pattern definitions, what information do we need in order to perform pattern matching in a well-defined way:
- Pattern type: which determines WHAT the pattern matching is looking for. There are a variety of pattern types, and I'll dedicate a posting to the pattern type I collected so far. Some examples: sequence (temporal pattern), moving north (spatio-temporal pattern), trheshold pattern (e.g. related to average of some value over a set) and more.
- Participant set: the set of event types whose instances issue the pattern matching set.
- Context: the context to which the pattern is associated with (actually the agent executing the pattern is associated with).
- Pattern assertion: Some patterns have assertions associated with them. Assertion can determine if events are relevant (e.g. if we are looking at a sequence of two events, say event of type E1, and event of type E2, where in order to do a pattern matching we require that E1.A > E2.B, where A and B are names of attributes. There are also some pattern associated with certain pattern type, e.g. if the pattern type is a threshold pattern than there is an assertion associated with it, e.g. Average (e1.A) [over each context partition] > 40.
- Pattern policy: Pattern policy determines when a matching set is going to be activated, how many times, can event count for more than one pattern, how repeated events are treated and more.
This was only in title level, and in the next postings I'll provide more information about pattern types and pattern policies.
My last trip abroad was a mix of vacation and business trip (to attend the Tretno event processing symposium, and the meeting of the EASSy consortium that is writing a proposal for EU project), as such I took a piece of luggage with me, and deposit it (in Hebrew the words "to deposit" and "to abandon" are very close --: "lehafkid" vs. "lehafkir", thus I am typically saying that I have abandoned the luggage in the airline...), anyway, standing near the carousel (something I really hate to do !), I realized that I am actually doing a pattern matching. I have a mental model of my luggage in my mind (shape, color, other characteristics), and I am trying to match this mental model with the pieces of luggage flowing on the carousel. There may be more than one that fits my mental model, and sometimes I grab one, look closely, and decides that this has been a "false positive". Likewise, we are matching patterns many times in our daily life.
So - what exactly is an event processing pattern? In the EPIA book, we have dedicated a (long!) chapter to pattern matching in event processing, and started it with the following definition:A Pattern is a function that takes a collection of input event instances and produces a matching set that consists of zero or more of those input events This is, in essence, what we do in the luggage case, selecting one out of a set. Interesting cases are cases in which we should find events that the pattern relate to their combination (this, according to some definitions, what makes the "Complex" in the "Complex Event processing" pharas), so they need to satisfy some pattern as a collection of events, and not as individual events. An example is we wish to determine if a moving object (say: motorcycle) is moving north, by looking at the location of the motorcycle over some period of time, we can observe whether the progression of location satisfies the pattern of "moving north". I'll write more about pattern types soon.
Today is a happy day to the Israeli scientific community it was announced that Professor Ada Yonath from the Weizmann Institute won a Nobel prize in chemistry, this adds to two Israeli scientists that won a Nobel prize in chemistry a few years ago, so Israel is a chemistry super-power. Computer science was not exist when Nobel decided on prizes, the equivalent is Turing Awards, that, if I am not mistaken, have been already awarded to three Israeli scientists, so we are not bad in this area either..
Back to event processing thinking --- getting progress on the EPIA book, yesterday we had two hours review with the editor on editorial stuff around the last three chapters, and upon revision, it will get to the 2/3 review. Our target date for publication is now April 2010, and this will probably be the final target.
Anyway, to continue my previous posting on contexts, I would like to discuss the notion of context composition. Recall that context is grouping of events based on one of the following: time, space, state and segment, for either grouping in order to apply operations on this group, or make distinct behavior for distinct groups. Composite context -- as its name suggests is just a multi-dimensional context, i.e. it contains cross-section of several contexts.
In the picture below this is a composition of segmentation ("per customer") and time (in this case fixed sliding temporal interval - per hour, each square is a context partition.
This is just example of combination, there are other useful compositions such as: spatio-temporal context: partition with one dimension is time and one is location oriented, or space and state contexts combination, example: spatial context == within the city of Trento, state of weather = {sunny, cloudy, rainy, snowing}, in each of these partitions other agents are applicable.
Somebody asked me -- what is the benefit of using the context abstraction anyway --- the answer is, like any other abstraction -- it saves work. The same application can be written with much less code and is much simpler to develop, maintain and understand --- the use of context is quite useful in this sense, see also some discussion by Marco. Next -- I'll write more about the next chapters on event processing patterns, stay tuned.
In the last few days, dishes in my house are washed by hand. Our dish washer have seen signs of old age, thus I acquired a new one. First step was to order delivery from the store. There is the software engineering principle of "separation of concerns", thus those who do the delivery don't install it. I am not allowed to install it myself, since if I open the package I am losing the warranty, only a technician of the service company related to the importer is authorized to open the package. After some coordination this technician arrived today, looked at our kitchen and said -- "this is very common, they have changed the standard, now there is a thermostat on the pipe, so the size of the hole that your need in the kitchen closet is larger". Since there is a separation of concerns, he does not do holes, and I had to call the Carpenter who came later today, so I'll be able to call the technician again tomorrow... This is the result of the fact that the event of "standard of pipe changed, thus it effects the size of the pipe" was not reported by the store, since they don't install dish washers so they probably don't know --- this separation of concerns, and not seeing event over a larger story, wasted a lot of time and money to the society...
Anyway -- enough complaining.
I got a question from Hans, related to my posting on context, I am copying the question here:
" I would be interested to hear about how the context concept works in a particular use case: Let's say I have events that contain, among other things, a number. I would like to capture the third quartile (a data point that sits immediately above 75% of the other points) of the number, every hour. I would then like to perform some operation over the previous 8 of these hourly numbers. How would this be expressed as contexts and agents?"
OK -- here it is:
We start by event of type E1 that flows into the system from some consumer and has some numeric attribute A, for which we want to capture the third quartile.
- First we define a context C1 of type of sliding fixed temporal interval with both duration and period of 1 hour.
- Then we define an agent A1, which is valid within C1, and calculates the third quartile and produces event of type E2, with an attribute B which that captures the derived value.
- Further we define a context C2 that is a sliding event temporal interval on event E2, with event count period and event count duration both of 8.
- And finally we define an agent A2 that subscribe to events of type E2, and is valid with context C2, that does some operation on E2, and may in turn produce (say) event of type E3, which is flowing to some consumer.
This is the basic model. Of course, it is a bit more complicated, if our agent library does not contain an agent that calculate quartiles may not be a basic, this can be somehow combined from some combination of agents, since an agent can be recursive and include mini-EPN, but as such we can still model it as a single agent in the high level view.
The nice thing about this abstraction that it is quite simple to model such problems... More - Later.
This is the illustration of the event processing network for the Fast Flower Delivery example that accompanies the EPIA book we are writing, don't spoil your eyes reading it. We'll provide the readers with an open source editor to the EPN meta-language we have developed (developed by Ronen Vaisenberg, a Ph.D. student in UCI). I'll have posting about it with more instructions in one of the next postings.
In this posting I would like to go back to the nodes in EPN and discuss why do we need multiple nodes and what are these nodes. One version is that there is a single type of node called EPA -- event processing agent, but my preference is to have multiple nodes, since they have different roles. Here are the types of nodes in our EPN model:
- Event Processing Agent (EPA): a node that gets one or more events as an input and produces one or more event as an output.
- Event producer: An entry point to the event processing system that emits events to the EPN.
- Event consumer: An exit point from the event processing system that receives events from the event processing system
- Event channel: A node that make routing decisions within the EPN, I have recently discussed event channels in this Blog.
- Global states: This is a node that holds various type of state (reference data, global variables)
- EPN node: An EPA can represent an EPN, and thus make the EPN recursive.
One of the frequent questions is whether an event producer which is also an event consumer is considered an EPA. In my opinion, the answer is no. Event producers and consumers operate on the borderline of event processing network, and is actually have have two nodes one for each of its roles. The information that a consumer is also a producer is important for traceability and lineage, but we'll discuss this aspect another time.
Christoph Emmesberger, whom I met in Trento, has proposed to extend the notion of channel to provide aggregation and filtering, this idea is rooted in pub/sub systems where event processing seems as an extension of some event processing functionality. However, when we deal with event processing system, I prefer to make a clear distinction between event processing agents that is doing -- filtering, aggregation, transformation and event pattern detection , and leave channel as a routing mechanism. thus we'll ensure proper separation of concerns, and have a cleaner model. We are now finishing the chapter about pattern detection, and I'll write about patterns soon.
This is Escher's famous picture "relativity" reporduced in Lego blocks. I reminded of the relativity issue while reading Paul Vincent's Blog entitled: "Is CEP subset of BPM"?
Well, if you ask the BPM marketing persons they will probably say yes, as today we can see event processing as part of a BPM stack (well, for some vendors). However, this does not say that event processing is a subset of BPM, the same question can be asked about databases, BPM suites store stuff on databases, so databases can be, by the same token, considered as subset of BPM. In the 5th Event Processing symposium, last week in Trento, we had sessions dedicated to the relationships of event processing with BPM, IT management and Robotics. IT management also viewed, for years, anything that has to do with events as a subset of IT management, and they are also right, since IT management deals with events, again, it also deals with databases. Talking with a simulation person he told me that event processing, of course, is a subset in simulation, and actually invented in the world of simulation (I thought that it has been present also in ancient operating systems). So everybody have their own relativistic view about event processing being a subset of something.
IMHO, Event processing, like databases, is a generic technology (BTW, some database people view event processing as a subset of data management, which IMHO is also not true) that can be embedded in various frameworks. The question is whether we need one generic event processing technology for say - all three domains I mentioned: BPM, IT Management and Robotics, or does each of them need its own variation, maybe they need different (possibly overlapping) subsets of the same technology as seen in this Venn Diagram? More about it - later.
Chameleon is an adaptive animal, it can adapt it color to the environment. In the next few weeks my main task will be to complete a proposal for EU project that deals with adaptive services. I have (mistakenly?) agreed to coordinate a consortium that creates a proposal. We had a meeting in Trento following the EPTS event processing symposium. EU project has a benefit of getting funding for research that can explore more advanced topics then can be funded by commercial corporates, and also provide an opportunity to collaborate with some very good people both in industry and academia. The down side is that there should be much investment in the proposal, since it became extremely competitive.
The proposal is about the idea of adaptive services -- which means in plain words that the behavior of the system should be adapted to (unexpected?) events. This adaptation may require human interaction (e.g. modifying medical treatment protocols), or self-adaptation (e.g. change of emergency handling protocol on the fly). The research challenges here are in area of event processing -- having more advanced features that are beyond the state-of-the-art; adaptation, modeling and methodology of how to build such systems. More about this topic - later.
In the previoius posting I started the discussion about the notion of pattern, and stated that it is a function that selects an event subset; continuing to drill down on patterns, I'll say a few words about what are the ingredients of pattern definitions, what information do we need in order to perform pattern matching in a well-defined way:
