Weekends is catch-up times for any workaholic. In my case, I dedicate the spare time in weekend to advance in the EPIA book, which, as expected, is somewhat behind the original schedule. We are now cleaning up the main chapters of the books that deal with : event processing network, contexts and patterns. I'll dedicate some of the coming postings to book related stuff.
In chapter 6 we get a deep dive into the notion of EPN, and its various components: event processing agents, channels and global states. Consumers and producer are discussed earlier in the book. Event Processing Agent (EPA) is a major abstraction we are using. In recent posting I've discussed that the conceptual EPN may be mapped in various ways to implementations, EPA is part of the EPN, as such it is conceptual creature, and can be mapped to implementation in various ways.
The illustration on the top of this page illustrates the (logical) anatomy of an EPA. It illustrates a kind of SCA like component (SCA has now a proposed events extension, I'll discuss it another time), with input terminals and output terminals. In this cases, the EPA gets events from input terminals, each of them has a distinct type of event. Let's take a simple three event conjunction example. A virtual car dealer matches used car sellers and buyers, once there is an agreement on the transaction, it supervises that it is carried out. The used car buying transaction is a conjunction of three events (the order of them may not be important) :
Event1: The seller physical hands over the car to the buyerEvent2: The buyer transfers to the seller the agreed upon amount for the carEvent3: The department of transportation approved and records the ownership transfer -- in Israel it is a service given in post offices, the buyer and seller identify and sign, and the post office clerk has access to the transportation ministry vehicle registration system and can verify who is the registered owner, and if the car has any incumbrance
The three logical steps of EPA are: filtering, matching (if the EPA is looking for a pattern), and deriving. The filtering part may be done outside the EPA, e.g. if the EPA instance deals with a certain transaction, then the channel should route only events related to this transaction to the EPA, I'll discuss more of the notion of channel in the near future. However, there are cases that the events should be filtered by the EPA, e.g. when it involves a condition on more than one event. After the filtering part some subset of the input events survive to be the players in the EPA. If the EPA is doing also pattern matching, it creates matching sets according to the requested pattern. In the case of the used car selling, the pattern is a conjunction of these three events that relate to the same transaction, the order is up to the buyer and seller. When the pattern is matched, it means, in this case, that three events have been detected, which satisfy together the pattern. These three events create a matching set, for each EPA instance there may be zero , one or multiple matching sets during the lifespan of this EPA instance. After the matching set is created, the third phase - the derivation phase is aimed to determine what should be reported out of the results of this EPA. It can just report the composite event that contains the three events in the matching set, and it can contain any other raw or calculated value that is a function of the attributes of these three events. Furthermore, for different purposes, different derived events can be required - consumed by other EPAs or by consumers. This is the way we describe EPA, note that this description is general enough to cover EPAs from different types. The input terminals can take either individual events or sets. I'll discuss more about EPA types, channels, contexts and patterns later
One cannot ignore these days the change of president in the USA, something with affects the entire universe. One minority the Mr. Obama belongs to is the minority of left-handed people, as can be clearly seen in the picture, while four of the last USA presidents were left handed (which make his fifth in the last seven presidents), conference rooms in the USA or university classes - all of them have desks only for the right handed majority. Here is a picture of a left handed desk, 
I am sure that the USA president has much more urgent items on his agenda, however, unlike his predecessors he may also do something for the deprived minority of left-handed people. BTW - the situation in Israel (whose current prime-minister is, surprise...a left-handed person) is somewhat better. We, Left handed people may be a small minority (around 10% of the population) but our collective impact on humanity is unproportionately huge -- starting from Alexander the great, Julius Caesar, Napoleon, Queen Victoria, Lewis Carrol, Mark Twain, Escher, Michelangelo, Leonardo da Vinci and many more....well enough of that for now.
Going back to one of my previous posts
that has explained the difference between event pattern detection and event pattern discovery.
In the wake of some questions, here is more about the relationship between these two terms:
- Event pattern detection is performed for patterns that are known in advance, the pattern detection in done "on-line" when the event occur.
- Event pattern discovery is performed typically off-line, it can use machine learning techniques on past events in some cases, it can also use some natural language understanding technique to derive pattern from legal documents (e.g. regulations) in other cases.
- A pattern discovery creates patterns that are detected on-line by pattern detection, so they are complementary techniques.
- In some cases there is continuous discovery, and thus the patterns are updated in a dynamic way, however, still the discovery feeds the detection part on-line, and the respective roles are preserved.
- Last but not least, the discovery process may use simulation techniques that use detection of simulated events in order to check assumptions about patterns.
Typically, event processing products contain event pattern detection capabilities, in one form or another. The event pattern discovery is considered as add-on, typically using techniques that are not particular to event processing.
This drawing, in various forms, has been used by us for many years to illustrate the notion of pattern, actually in the PowerPoint version it is animated, and the geometric shapes are keep moving. The term pattern is a bit overloaded in event processing, as noted my DEBS 2008 tutorial on this topic, but this illustration refers to the pattern which shows some combination of events, to be more accurate it is a predicate on the event history that if evaluated to the value of
"true" something should happen. This illustration was created by Tali Yazkar-Haham from IBM Haifa Research Lab as an exercise in a presentation course, and was used in dozens of presentations ever since (including presentations of some people outside IBM who typically forgot to give credit to the source).
Paul Vincent, in a continuous debate with Tim Bass, on the complex events forum, has written about "detection of new instance" and "detection of new type". While these terms make sense, I prefer not to overload the term detection and use the terms event pattern detection and event pattern discovery.
Pattern detection deals with detection that a predefined pattern has happened. This is what illustrated in the picture above. Some example can be: a patient is hooked up to a heartbeat monitor, and the physician is pre-setting a pattern "the heartbeat is monotonically increasing within 10 minutes, and the amount of increase is more than 30 during that period". This is actually a predicate over a part of the event history of a single source and type (other examples can involve multiple sources and types, but the principle is the same).
So event pattern detection is defined as detection that a predefined patterns has occurred.
This is equivalent to what Paul called: Pattern instance Detection.
In contrast, when we talk about event pattern discovery we mean that the pattern is not known in advance, and the pattern discovery function determines what is the pattern. The legend says that Archimedes discovered his famous laws about floating bodies when sitting in the bathtub and shouted: Eureka (this illustration was taken from the homepage of a company who has the word Eureka in its name, again animated in the source). 
A pattern can be discovered by machine learning techniques using decision trees, statistical modeling, Bayesian Networks and numerous other methods. At the end when a pattern is discovered then it also need to be detected in reality; there are also cases in which there is a continuous detection since the patterns are changing after a short time.
Getting back to the previous example about the heartbeat, it may be the case that this pattern has bot been set by a physician, instead it was detected by some method that has looked at past events and found out that this pattern has some significance.
Most people thinking about "complex event processing" are actually talking about pattern detection, regardless of whether the patterns were composed by a human or discovered by machine learning. The illustration at the top of this page illustrates what people typically mean. As stated in the past, I don't want to get into the meaning of TLAs, and leave it to my colleagues who are doing marketing. Thus the term that I used "event pattern detection" is the more accurate one.
Another observation about the difference is that event pattern detection can be applied as COTS -- a user can use such a product, compose some patterns, hook it up to event sources, and get the pattern to be detected.
On the other hand --- while there are many tools that can help in the event pattern discovery, we cannot hook it up to event source and tell it: discover all. There is a need to do some formal modeling of the system, kind of patterns that are sought etc... In other words, this is not something that a typical developer or business analyst can do, since it requires some expertise,
It is getting late - so I'll finish at this point and return to this topic at some later point.
