Weekends is catch-up times for any workaholic. In my case, I dedicate the spare time in weekend to advance in the EPIA book, which, as expected, is somewhat behind the original schedule. We are now cleaning up the main chapters of the books that deal with : event processing network, contexts and patterns. I'll dedicate some of the coming postings to book related stuff.
In chapter 6 we get a deep dive into the notion of EPN, and its various components: event processing agents, channels and global states. Consumers and producer are discussed earlier in the book. Event Processing Agent (EPA) is a major abstraction we are using. In recent posting I've discussed that the conceptual EPN may be mapped in various ways to implementations, EPA is part of the EPN, as such it is conceptual creature, and can be mapped to implementation in various ways.
The illustration on the top of this page illustrates the (logical) anatomy of an EPA. It illustrates a kind of SCA like component (SCA has now a proposed events extension, I'll discuss it another time), with input terminals and output terminals. In this cases, the EPA gets events from input terminals, each of them has a distinct type of event. Let's take a simple three event conjunction example. A virtual car dealer matches used car sellers and buyers, once there is an agreement on the transaction, it supervises that it is carried out. The used car buying transaction is a conjunction of three events (the order of them may not be important) :
Event1: The seller physical hands over the car to the buyerEvent2: The buyer transfers to the seller the agreed upon amount for the carEvent3: The department of transportation approved and records the ownership transfer -- in Israel it is a service given in post offices, the buyer and seller identify and sign, and the post office clerk has access to the transportation ministry vehicle registration system and can verify who is the registered owner, and if the car has any incumbrance
The three logical steps of EPA are: filtering, matching (if the EPA is looking for a pattern), and deriving. The filtering part may be done outside the EPA, e.g. if the EPA instance deals with a certain transaction, then the channel should route only events related to this transaction to the EPA, I'll discuss more of the notion of channel in the near future. However, there are cases that the events should be filtered by the EPA, e.g. when it involves a condition on more than one event. After the filtering part some subset of the input events survive to be the players in the EPA. If the EPA is doing also pattern matching, it creates matching sets according to the requested pattern. In the case of the used car selling, the pattern is a conjunction of these three events that relate to the same transaction, the order is up to the buyer and seller. When the pattern is matched, it means, in this case, that three events have been detected, which satisfy together the pattern. These three events create a matching set, for each EPA instance there may be zero , one or multiple matching sets during the lifespan of this EPA instance. After the matching set is created, the third phase - the derivation phase is aimed to determine what should be reported out of the results of this EPA. It can just report the composite event that contains the three events in the matching set, and it can contain any other raw or calculated value that is a function of the attributes of these three events. Furthermore, for different purposes, different derived events can be required - consumed by other EPAs or by consumers. This is the way we describe EPA, note that this description is general enough to cover EPAs from different types. The input terminals can take either individual events or sets. I'll discuss more about EPA types, channels, contexts and patterns later
Back to work for one day in the office, with five conference calls (one with Germany, one with France, one with UK, and two with USA...) and then back to home for the weekend. When I have free time I like to read books, the current book I am reading is "A Lion Among Men", 0ne of the books of Gregory Macguire, who writes stories that take as background famous children stories (in this case - the Wizard of Oz), actually this is the third one behind the scene of the Wizard, now taking the Lion as its main character. I have another book of the same author still waiting...
We also submitted the draft of chapter 3 of the "Event Processing in Action" book to the publisher, which hopefully be posted on the MEAP site soon.
The approach we have taken in the book, as I have written before, is to use the "building block" approach, describing event processing principles, and the use case whose construction demonstrates the application, using building blocks, which are like the chemical elements. The application itself is being built by using "definition elements" which are like atoms (my partner for writing this book, Peter Niblett, has come with the analogy from the world of chemistry). we believe that this is the right approach to teach what event processing is -- in the "deep dive" part of the book we dedicate a chapter for each of the major seven building blocks and then dive deeper into the types of event processing agents (which deserves a different discussion). We'll also provide samples of how each building blocks is realized in different models.
The seven building blocks are:
- Event type: defines the event schema
- Even producer: the projection of the event producer over the event processing network (note that the event producer itself is outside the scope)
- Event consumer: same -- the projection of the event consumer over the EPN.
- Event channel: the glue that holds the EPN together
- Event processing agent: the brain that does the entire work; each agent is doing a specific task of processing.
- Context: the semantic partition of events and agents
- Event derivation: A building block that is possibly part of each EPA that specifies the derived event.
There are some more building blocks that are used to support these ones, but our claim is that this set of building block is what needed to build an event processing application.
Chapter 4 which is in advanced phases of being written starts the deep dive by discussing the event type building block, and in one of the next posts I'll say more about it.
Today I have participated in the local IBM championship in Backgammon (in Hebrew we use the Turkish name "shesh-besh"), however, my participation did not last very long, I lost the first game, and did not move to the second round, thus, returned to my office... At least the one who won the game over me is the technician in charge of the video conference equipment, so if one wants to conduct video conferences, make sure he likes you, since he is the only one in the lab who knows how to operate this simple equipment.
Today I would like to write a little bit about "event derivation" - in one of the past postings I've used this illustration in order to explain relationships between terms. Why do we need all these terms? -- well - we don't, but they are "legacy" terms, thus it is a good to show the relations among them.
When saying "derivation" -- let's compare it to two other types of derivations:
- Inference -- where the input are: facts and inference rules, for example: fact 1 -- Jack is the father of Mike; fact 2 -- Mike is the fater of Alex; inference rule: X is grandfather of Y if X is father of Z and Z is father of Y. From this system we can infer that Jack is the grandfgather of Alex. This is a deductive inference, there are some other types of inferences, and can be obtained by logic programming, or inference rule engines.
- Data derivation --- although the term is broader --- if we'll take the relational database variation --- it takes a collection of relations as an input, apply query, and gets a relation as an output -- here there is no logical inference, but computation over data. Other variation of data derivation is a spreadsheet formula, typically performing some aggregation operator (sum, count, average...) on neigboring data in the table.
Event derivation is neither of the above, but has similar nature to both. The input and output are events, if we present it as functions then:
Inference: Facts X rules ----> Fact
Data derivation: Relations X Query ---> Relation
Event Derivation: Events X Event Derivations ---> Events
Now looking closely at event derivation. First -- why does event derivation occur ?
Event derivation may occur from various reasons -- while in mind it is closely related with pattern detection, it may also be associated with other event processing functions such as: enrichment and transformation.
Some examples:
Example 1:
Input event: Order that consists of the attributes (not including the header) -- customer, item, quantity.
Functionality type: enrichment -- add attributes as: customer-type, customer-reliability-rank to the order.
Output event: the enriched event. In this case the derived event is identical to the event that has been created by the main function (enrichment) and the derivation is just copying.
Output event occurs immediately when function is done
Example 2:
Input event: Order (same as first example)
Functionality: Aggregation -- quantity by item.
Output event: Item-Demand: Item, Quantity. Multiple events - one for each item.
Output event occurs at the end of the context ("deferred").
comment: here a composite event is being accumulated, and at the end of the context is decomposed to individual events.
Example 3:
Input events: Order (same as first), Item-Demand (Output of example 2).
Functionality: Detect Pattern, the pattern is: Quantity of a single order > 2/3 from quantity of the same Item.
Output event: Dominant-Order: Order-Id (from header), Item, Dominance Ratio = (Order.Quantity / Item. Quantity)
Output event occurs as soon as detected.
Example 4:
Input events: Temperature-Reading (every 1 minute).
Functionality: Aggregator/collector -- collects all reading of a certain hour and creates one event.
Output event: Every hour -- Composite event that consists of 60 raw events.
These are some examples. More formal definition of event derivation - later.
