On event processing components exchange

I haven't blogged for a while, hectic week is ending (our working week is Sunday - Thursday, so this is the evening of the last working day!). Meanwhile I have noticed that Streambase has declared "component exchange program", looking closer, this includes a collection of stuff, starting from Twitter adapter, and even including the Streambase implementation to our "Fast Flower Delivery" example. Very good idea. Now, the challenge is to have a global components exchange that is not specific to a single vendor. In my previous posting I have written about event processing building blocks, and having a building block exchange is the ultimate way to implement this idea, everybody will be able to mix and match building blocks. The inhibitor is the lack of standard. Let's say that I want to use two of the exchange components, and that one should send events to the other, then of course they need to agree on how event is being represented, writing adapters for each pair of components is not very appealing. Furthermore, if one develops an application out of various programmable components, one would want them to be speak the same language. So -- having an exchange around a single product is nice, having an exchange in the industry level will be even nicer. More -later.

On event processing building blocks

As response to my posting about stand alone or part of a bigger picture, Marco made a comment that in the event-driven world there is an opportunity to provide building blocks that may be used in various platforms and part of multiple bigger pictures. This is a valid point, as event processing is indeed a collection of capabilities of various types -- transformation, aggregations, pattern detection and more, each of them can also be of various types --- e.g. supporting variety of aggregating functions, variety of transformation, multiple patterns etc, coupled with the fact that event-driven computing is decoupled, thus the interfaces to these components can be quite simple --- receive events and send events, can provide an opportunity to provide variety of such components, kind of plug-and-play event processing. The key, and currently main obstacle in letting it happen is standardization and current lack of standards.

• Interoperability standards are needed for standard interfaces
• Event meta-data standards are needed so that the events exchanged between components
• Languages and meta-modeling standards so people can model and program in a seamless model.

I believe that this is the right direction going forward, and it is a direction we in Haifa Research Lab are investigating; this might be the key to make event processing a pervasive technology - more about it, later.

More on states and event processing

During my last visit in the USA I have purchased and now reading the book "the last oracle" (nothing to do with the software company that bears this name), one of the thrillers that are based on some history and span around the world, exposing some scheme that will bring disaster on the universe. It seems that this type of books became popular recently.

Anyway, I wanted to come back to the issue of states that I have recently blogged about. Actually the term state in event processing are somewhat overloaded. A state may refer to any of the following:

• The internal state of a single agent -- when trying to detect a pattern (say - E1 and E2) there is a need to keep the state, e.g. the fact that E1 already arrived but E2 did not. This can be kept either by keeping the input event as an object, keeping another object somehow derived from the original event, or keeping some internal data structure that helps in the detection.
  
• An event store where events are kept for further consumption by other agents -- this is a global (or bounded) state.
• Reference data that is being used by event processing agents for enrichment, this data is not maintained by the event processing system, but can be considered as part of the event processing state, since the event processing results may depend upon values of this data.
• Global variables (whether persisted or used on a shared memory) that are being used as a global state across event processing agents.
• Context -- on which I have blogged several times, which may have temporal, spatial, segmentation and external state (which is indeed a global variable) dimensions.
  

All of them are in a way part of the system state, they have different roles, though.

BTW -- we have revised the building blocks that we are using in the EPIA book, by replacing event derivation (which is actually part of the event processing agent) with global state.



Among the types of states we mentioned before -- context is a first class citizen in the model, internal state is hidden inside the implementation and is not explicitly modelled (although some languages allow also to model the internal state explicitly) and global state contains all the rest (reference data, event store, global variables). More - Later.

On the methodic use case used in the EPIA book

The EPIA (Event Processing in Action) book that Peter Niblett and myself are writing went to its first milestone -- the first 1/3 of the book (five chapters) have been completed in a draft form (chapter 4 and 5 should still be posted on the web by the publisher), and sent to a set of reviewers that the publisher has selected. One of the main issues in the book is, of course, to make things concrete by using a concrete example. We chose to use one example that accompanies the entire book and is developed step-by-step during the book chapters. One of the good advices we received from one of the initial reviewers on the book outline, was to use an example that does not require any domain knowledge (e.g. trade example), since this can issue a communication barrier with readers not familiar with that domain. Taking this advice, we decided to go for a methodic use case, that takes things from various applications we are familiar this and wraps them up in a single story, which does not require any prior domain knowledge - and this is the "Fast Flower Delivery" use case that I present below.

We present this use case in the descriptive language that we build througout the book, a sample of a building block in this language describing the event structure has been presented in past posting. The intention is also to go beyond that and add (as appendix?) some code samples from languages of various kinds -- SQL extension, rule language, script language etc... some language owners have already agreed to help, and we'll solicit more help from them soon.






Here is the current draft of the "Fast Flower Delivery" use case. It is fairly simple to understand, yet, it includes many (not all) the concepts we explain in the book:

General description

The flower stores association in a large city has established an agreement with local independent van drivers to deliver flowers from the city’s flower stores to their destinations. When a store gets a flower delivery order it creates a request which is broadcast to relevant drivers within a certain distance from the store, with the time for pick up (typically now) and the required delivery time if it is an urgent delivery. A driver is then assigned and the customer is notified that a delivery has been scheduled. The driver picks up the delivery and delivers it, and the person receiving the flowers confirms the delivery time by signing for it on the driver's mobile device. The system maintains a ranking of each individual driver based on his or her ability to deliver flowers on time. Each store has a profile that can include a constraint on the ranking of its drivers, for example a store can require its drivers to have a ranking greater than 10. The profile also indicates whether the store wants the system to assign drivers automatically, or whether it wants to receive several applications and then make its own choice.

Skeleton Specification

Phase 1: Bid Phase

The communication between the store and the person who makes the order is outside the scope of the system, so as far as we are concerned a delivery’s life-cycle starts when a store places a Delivery Request event into the system. The system enriches the Delivery Request event by adding to it the minimum ranking that the store is prepared to accept (each store has different level of tolerance for service quality). Each van is equipped with a GPS modem which periodically transmits a GPS Location event. The system translates these events, which contain raw latitude and longitude values, into events which indicate which region of the city the driver is currently in. When it receives a Delivery Request event the system matches it to its list of drivers. A filter is applied to this list to select only those authorized drivers who satisfy the ranking requirements and who are currently in nearby regions. A Bid Request event is then broadcast to all drivers that pass this filter.

Phase 2: Assignment phase

A driver responds to the Bid Request by sending a Delivery Bid event designating his or her current location and committing to a pick up time. Two minutes after the broadcast the system starts the assignment process. This is either an automatic or a manual process, depending on the store’s preference. If the process is manual the system collects the Delivery Bid events that match the original Bid Request and sends the five highest-ranked of these to the store. If the process is manual, the store makes the assignment and creates an Assignment event that is sent to the system; if the process is automatic then the first bidder among the selected drivers wins the bid, and the Assignment event is created by the processing system. The pickup time and delivery time are set and the Assignment is sent to the driver.

There are also some alerts associated with this process: If there are no bidders an alert is sent both to the store and to the system manager; if the store has not performed its manual assignment within one minute of receiving its Delivery Bid events then both the store and system manager receive an alert.

Phase 3: Delivery process

When the driver arrives to pick up the flowers from the store, the store sends a Pick Up Confirmation event; when the driver delivers the flowers, the person receiving them confirms by signing the driver's mobile device, and this generates a Delivery Confirmation event. Both Pick-Up Confirmation and Delivery Confirmation events have time-stamps associated with them, and this allows the system to generate alert events. A Pick-Up Alert is generated if a Pick-Up Confirmation was not reported within five minutes of the committed pick up time. A Delivery Alert is generated if a Delivery Confirmation was not reported within ten minutes of the committed delivery time.

Phase 4: Ranking Evaluation

The system performs an evaluation of each driver’s ranking every time that that driver completes 20 deliveries. If the driver did not have any Delivery Alerts during that period then the system generates a Ranking Increase if the driver has had more than five delivery alerts during that time then the system generates a Ranking Decrease to reduce the ranking by one point. If the system generates a Ranking Increase for a driver whose previous evaluation had been a Ranking Decrease then it generates an Improvement Note. event indicating that the driver’s ranking has increased by one point. Conversely

Phase 5: Activity Monitoring

The system aggregates assignment and other events and counts the number of assignments per day for each driver for each day on which the driver has been active. Once a month the system creates reports on drivers' performance, assessing the drivers according to the following criteria:

• A permanent weak driver is a driver with fewer than five assignments on all the days on which the driver has been active.
• An idle driver is a driver with at least one day of activity which had no assignments.
• A consistent weak driver is a driver, whose daily assignments are at least two standard deviations lower than the average assignment per driver on each day in question.
• A consistent strong driver is a driver, whose daily assignments are at least two standard deviations higher than the average assignment per driver on each day in question.
• An improving driver is a driver whose assignments increase or stay the same day by day

More about the building blocks used in the book will be discussed in future postings.

On Event Processing Building Blocks

Back to work for one day in the office, with five conference calls (one with Germany, one with France, one with UK, and two with USA...) and then back to home for the weekend. When I have free time I like to read books, the current book I am reading is "A Lion Among Men", 0ne of the books of Gregory Macguire, who writes stories that take as background famous children stories (in this case - the Wizard of Oz), actually this is the third one behind the scene of the Wizard, now taking the Lion as its main character. I have another book of the same author still waiting...

We also submitted the draft of chapter 3 of the "Event Processing in Action" book to the publisher, which hopefully be posted on the MEAP site soon.

The approach we have taken in the book, as I have written before, is to use the "building block" approach, describing event processing principles, and the use case whose construction demonstrates the application, using building blocks, which are like the chemical elements. The application itself is being built by using "definition elements" which are like atoms (my partner for writing this book, Peter Niblett, has come with the analogy from the world of chemistry). we believe that this is the right approach to teach what event processing is -- in the "deep dive" part of the book we dedicate a chapter for each of the major seven building blocks and then dive deeper into the types of event processing agents (which deserves a different discussion). We'll also provide samples of how each building blocks is realized in different models.

The seven building blocks are:

• Event type: defines the event schema
• Even producer: the projection of the event producer over the event processing network (note that the event producer itself is outside the scope)
• Event consumer: same -- the projection of the event consumer over the EPN.
• Event channel: the glue that holds the EPN together
• Event processing agent: the brain that does the entire work; each agent is doing a specific task of processing.
• Context: the semantic partition of events and agents
• Event derivation: A building block that is possibly part of each EPA that specifies the derived event.

There are some more building blocks that are used to support these ones, but our claim is that this set of building block is what needed to build an event processing application.

Chapter 4 which is in advanced phases of being written starts the deep dive by discussing the event type building block, and in one of the next posts I'll say more about it.