More on the DEBS use case tutorial

I am now in my hotel room in Tarrytown, NY; one of the nearby towns is known as "Village of Sleepy Hollow", I always wondered about the origin of this name, and who wants to say -- I live in sleepy hollow.. I have seen some other strange name. For example, there is a USA small town called - "bird in hand".

I have noticed that Pedro Bizarro has posted the DEBS use case tutorial provided by the EPTS use case work groups on slideshare in form of 7 presentations. Enjoy!

It is interesting to note the "lessons learned" presentation, it summarizing ten lessons.

• Architecture
  • Lesson 1: State-based event processing
    
  • Lesson 2: Incident objects
  • Lesson 3: Integration with other data management systems
  • Lesson 4: System of systems
• Languages
  
  • Lesson 5: Query languages
  • Lesson 6: Classification and rules groups
  • Lesson 7: Customization
• Glossary
  • Lesson 8: Changes to glossary
• Use Cases
  • Lesson 9: Changes to questionnaire
  Lesson 10: Better instructions to describe a use case
  

I agree with the state-based lesson; I am advocating the notion of context which can include various types of state and segmentation, and I view it as a first class citizen in the model. This is both architecture and language issue.


What they call "incident object" seems to be an object that describes derived event that has some internal structure which can be updated by programs. This seems to be quite trivial notion that events need to have some structure, and be able to participate in processing other events, may be retained over time; I think that this is true for any event, as event may also mean "event object". About the question whether it can be updated by programs, this is a matter of taste, since they want to keep these objects in temporal databases, temporal databases has adopted an "append only" model, where changes create other objects, and there is some reasoning behind it. I may dedicate a postings to this topic.

Integration with other data management systems -- given that Dieter Gawlick is co-leading this workgroup, one can expect that there is a database-centric view of event processing, his views on this topics are well known. I agree that databases can be integrated with event processing systems for various reasons (state keeping, reference, meta-data holding, high availability...), but event processing may also be integrated with a messaging system, with business process management system, with a BAM system, and with sensor networks, so I would extend it to integration with other parts of the enterprise architecture.

System of systems has to do with modularization and some implementation issue (e.g. bounds of local and global states). In my view EPN can be defined recursively such that an EPA can represent an EPN, thus we can express "system of systems". I am not sure that this requires any conceptual difference.

Query languages -- I am not sure I got the exact point here, but it seems that all event processing styles require to express conditions such as predicates for various uses. I think this is already happening. The issue whether internal state can be queried is different, and not relate to the language issue.

Classification / customization-- This is again one of the context dimensions, as context can classify populations according to classification predicates. Languages that are more syntactic oriented like the SQL derivatives hide classification inside SQL Where, but where context is supported as a primitive concept then classification is an explicit construct; this also allows customization. The use of vocabulary, as been used in rule languages, can be helpful here.

Glossary -- I'll leave the glossary discussion to the glossary workgroup for a while.

Use cases description --- the goal of this entire exercise was to "debug" the use cases template; I hope that the use case workgroup will finalize the template soon, and move to the next phase -- getting a substantial number of using cases and draw conclusions about their types, requirements etc--- it will be a great help to the community if we'll succeed to classify the applications to a finite, small, number of classes, where in each of them we'll be able to specify important functional and non-functional requirements. More on this -later.

DEBS 2009 - first day

This is the Vanderbilt University Law School, where the DEBS conference is being held this year. Today I spent the day in a U-shaped seminar room, with a lot of ex-Deans of the school hanging on the wall. The first day has been the tutorials day. In the first half of the day I attended the "use cases" tutorial in which the EPTS use cases workgroup, chaired by Pedro Bizarro and Dieter Gawlick, who, together with some more members of this workgroup presented some use cases, together with lessons learned. Paul Vincent has written a review (with some criticism) about that tutorial. I wonder how many bearded academics he counted in the conference -- indeed, once in academic conferences one could see all the prominent scientists carrying a beard, but today, most of them are shaving their entire face... Dieter who presented some of this tutorial has a beard, but he is from Oracle and not from the academia. Anyway, the idea to have an EPTS award (donated by TIBCO) to the application of a year is a good one, I'll bring it to the EPTS steering committee. We also discussed (after the end of the program) topics related to the use cases workgroup and relations with other EPTS workgroups.

In the second half of the day, Jon Riecke, Adrian Paschke and myself have presented the "event processing languages" tutorial. I am not reviewing my own talks, so instead -- you can look at the slides and review it yourself. More about DEBS 2009 -- tomorrow.

On the methodic use case used in the EPIA book

The EPIA (Event Processing in Action) book that Peter Niblett and myself are writing went to its first milestone -- the first 1/3 of the book (five chapters) have been completed in a draft form (chapter 4 and 5 should still be posted on the web by the publisher), and sent to a set of reviewers that the publisher has selected. One of the main issues in the book is, of course, to make things concrete by using a concrete example. We chose to use one example that accompanies the entire book and is developed step-by-step during the book chapters. One of the good advices we received from one of the initial reviewers on the book outline, was to use an example that does not require any domain knowledge (e.g. trade example), since this can issue a communication barrier with readers not familiar with that domain. Taking this advice, we decided to go for a methodic use case, that takes things from various applications we are familiar this and wraps them up in a single story, which does not require any prior domain knowledge - and this is the "Fast Flower Delivery" use case that I present below.

We present this use case in the descriptive language that we build througout the book, a sample of a building block in this language describing the event structure has been presented in past posting. The intention is also to go beyond that and add (as appendix?) some code samples from languages of various kinds -- SQL extension, rule language, script language etc... some language owners have already agreed to help, and we'll solicit more help from them soon.






Here is the current draft of the "Fast Flower Delivery" use case. It is fairly simple to understand, yet, it includes many (not all) the concepts we explain in the book:

General description

The flower stores association in a large city has established an agreement with local independent van drivers to deliver flowers from the city’s flower stores to their destinations. When a store gets a flower delivery order it creates a request which is broadcast to relevant drivers within a certain distance from the store, with the time for pick up (typically now) and the required delivery time if it is an urgent delivery. A driver is then assigned and the customer is notified that a delivery has been scheduled. The driver picks up the delivery and delivers it, and the person receiving the flowers confirms the delivery time by signing for it on the driver's mobile device. The system maintains a ranking of each individual driver based on his or her ability to deliver flowers on time. Each store has a profile that can include a constraint on the ranking of its drivers, for example a store can require its drivers to have a ranking greater than 10. The profile also indicates whether the store wants the system to assign drivers automatically, or whether it wants to receive several applications and then make its own choice.

Skeleton Specification

Phase 1: Bid Phase

The communication between the store and the person who makes the order is outside the scope of the system, so as far as we are concerned a delivery’s life-cycle starts when a store places a Delivery Request event into the system. The system enriches the Delivery Request event by adding to it the minimum ranking that the store is prepared to accept (each store has different level of tolerance for service quality). Each van is equipped with a GPS modem which periodically transmits a GPS Location event. The system translates these events, which contain raw latitude and longitude values, into events which indicate which region of the city the driver is currently in. When it receives a Delivery Request event the system matches it to its list of drivers. A filter is applied to this list to select only those authorized drivers who satisfy the ranking requirements and who are currently in nearby regions. A Bid Request event is then broadcast to all drivers that pass this filter.

Phase 2: Assignment phase

A driver responds to the Bid Request by sending a Delivery Bid event designating his or her current location and committing to a pick up time. Two minutes after the broadcast the system starts the assignment process. This is either an automatic or a manual process, depending on the store’s preference. If the process is manual the system collects the Delivery Bid events that match the original Bid Request and sends the five highest-ranked of these to the store. If the process is manual, the store makes the assignment and creates an Assignment event that is sent to the system; if the process is automatic then the first bidder among the selected drivers wins the bid, and the Assignment event is created by the processing system. The pickup time and delivery time are set and the Assignment is sent to the driver.

There are also some alerts associated with this process: If there are no bidders an alert is sent both to the store and to the system manager; if the store has not performed its manual assignment within one minute of receiving its Delivery Bid events then both the store and system manager receive an alert.

Phase 3: Delivery process

When the driver arrives to pick up the flowers from the store, the store sends a Pick Up Confirmation event; when the driver delivers the flowers, the person receiving them confirms by signing the driver's mobile device, and this generates a Delivery Confirmation event. Both Pick-Up Confirmation and Delivery Confirmation events have time-stamps associated with them, and this allows the system to generate alert events. A Pick-Up Alert is generated if a Pick-Up Confirmation was not reported within five minutes of the committed pick up time. A Delivery Alert is generated if a Delivery Confirmation was not reported within ten minutes of the committed delivery time.

Phase 4: Ranking Evaluation

The system performs an evaluation of each driver’s ranking every time that that driver completes 20 deliveries. If the driver did not have any Delivery Alerts during that period then the system generates a Ranking Increase if the driver has had more than five delivery alerts during that time then the system generates a Ranking Decrease to reduce the ranking by one point. If the system generates a Ranking Increase for a driver whose previous evaluation had been a Ranking Decrease then it generates an Improvement Note. event indicating that the driver’s ranking has increased by one point. Conversely

Phase 5: Activity Monitoring

The system aggregates assignment and other events and counts the number of assignments per day for each driver for each day on which the driver has been active. Once a month the system creates reports on drivers' performance, assessing the drivers according to the following criteria:

• A permanent weak driver is a driver with fewer than five assignments on all the days on which the driver has been active.
• An idle driver is a driver with at least one day of activity which had no assignments.
• A consistent weak driver is a driver, whose daily assignments are at least two standard deviations lower than the average assignment per driver on each day in question.
• A consistent strong driver is a driver, whose daily assignments are at least two standard deviations higher than the average assignment per driver on each day in question.
• An improving driver is a driver whose assignments increase or stay the same day by day

More about the building blocks used in the book will be discussed in future postings.

On Top Down and Bottom Up

My B.A. degree is in Philosophy (well, to be accurate most of the studies were related to logic), later I have learned business administration and Computer Science, but I think that Philosophy was the most important thing I have learned, as the other studies provided techniques, while Philosophy provide more basic competencies.

When I have decided to study Philosophy, my late father, who has been very practical person, invested an entire evening trying to convince me that I am going to waste my time, and why I cannot study engineering, medicine or law like everybody else, well, I have listened carefully, and went to study philosophy, a decision I have never regretted - actually some of my friends switched to study philosophy when they realized that I am enjoying my studies and they are not.

I was reminded about this episode with time wasting when I caught up in what happened in Blog-land during my vacation, and discovered the blog posting entitled: fallacies of self-fullfiling CEP use case studies. I am quite amused to read these type of Blogs, I hope that the author is also feel amused when writing them... well, after being amused for a few minutes, I thought -- well, today there is a conference call of the EPTS use case workgroup, maybe I should cancel the call, since all the participants including myself are going to waste their time in addition for the time already wasted on fallacies... Fortunately, I have studied Philosophy, and moreover investigated fallacy types a long time before Wikipedia was created, and failed the fallacy in the identification of fallacies (an absence of fallacy event..), so I decided not to cancel the call.... I have a feeling that the participants did not think that they have wasted their time, but who knows...

Enough humor, time to have some serious stuff on this posting -- so I think that top-down and bottom-up approaches is a good topic to discuss. While the original posting used the metaphor of space rockets

I'll stay on the earth and use the metaphor of a gadget - let's imagine that some inventor invents a gadget maybe like this:

This gadget has a lot of features, and it was designed in a totally top-down manner.

At some point customers start to acquire this gadget and are using it in different ways for different purposes. The inventors of this gadget have a lot of ideas of what else to do in the next version of the gadget, but besides the top-down innovation there is also a bottom-up process that is called in control theory: feedback. Typically, a large enough sample of customers is being interviewed to understand - what is this used for ?, how it is used ?, are various features needed? understood ? used the same way as imagined by the inventors or otherwise? are there some requirements for this gadget to connect to other gadgets? some requirements about the operational aspects? this information can be used both to explain new customers about the experience gained, and information that can tune up the priorities and ideas of the next generation.

Back from the fascinating gadget world to our not less fascinating EP world -- the use case workgroup study is intended to understand both about the ways in which EP technologies are used today, and about additional requirements that customers who already used EP technologies are looking at (customers that did not use a certain technology typically have difficulties to express requirements about such a technology, unless they have studied the area...). .

My assumption is that the end result of this study will be beneficial for the entire community - customers who would like the best practices, vendors that design their next generation, researchers who wish to analyze this market etc... However, assumptions are just philosophy and as such, my assumptions are as good as the counter-assumption that this is all a self-fulfilling fallacy. Since I have grown up and am wearing the scientist hat these days, I suggest to take the empirical approach, namely, be patient to see the end result and judge it.

And bottom line about bottom-up and top-down: Top-Down and Bottom-up work have in general complimentary roles, and are used in different phases in the life-cycle of products/technologies/areas. Important concepts such as: use patterns and best practices are, by definition, Bottom-Up...