On automatic translation

A known urban legend about automatic translation is that an automatic translation program got as an input the phrase "the spirit is willing but the flesh is week" and translated it from English to Russian and then back to English, the end result was "the vodka is good but the steak is lousy", there are some translation pearls collected all over the Web. I am using automatic translations from time to time, mainly since my good friend Rainer von Ammon has a habit of forwarding me Emails and documents in German, the automatic translation programs I can find on the web are not that good, but I can understand more or less what is written. However, last night I had my moment of loud laughing. While searching the Web for something using the almighty Google search, I came across a webpage written in Hebrew, I realize that most of the Blog readers don't read Hebrew so I'll summarize the reading experience: first -- it looks like a collection of words in the wrong order and syntax that does not make any sense, second --- looking closer I realized that I actually wrote it, well - it is not that I forgot how to write in Hebrew, on the contrary, my Hebrew is still much better than my English, but it seems that it is supposed to be a translation to Hebrew of a Blog posting I have written in English in January 2009. Trying to get to the bottom of it, I've found that there is a site called the "Unix and Linux form" which copied some of my Blog posting (not sure in what context) using some crawler that is called "Linux Bot", it seems that it did not just copy it, but also translated it to Hebrew. Since Hebrew is not the most popular language in the universe, I wonder to how many other languages it is translated, and if somebody is making any quality control. Funny.

Revisiting race condition with FFD example

In the past I have written about race conditions and this triggered some responses. We recently realized that in the example we created for the EPIA book (the Fast Flower Delivery and has got already around ten different implementations, six of them can be viewed on the book's webpage, some more will be added) there is an case that if will not be handled carefully may yield wrong results due to race conditions. Here is the case:

There is an aggregate EPA per driver and day that collects assignment events for a driver and in the end of the day creates a derived event which counts the number of assignment per driver, there is a second EPA per day that collects all the drivers count at that day and calculates mean and standard deviation for the number of assignments per active drivers in that day; there is a third EPA, again per driver and day, which gets the derived events from the first two EPAs and calculate for each driver its deviation from the mean, in standard deviation units. These three EPAs are all aggregation type EPA which has some order among them, until now -- no problem. Now, the issue is that all these calculations occur at the end of the day, and have causal dependencies. If we are not careful, the first EPA calculates the count per driver at the end of the day, but until it finishes the calculation the time is say, 12:01, so the result is classified to the next day, but it is required to calculate the statistics for this day, and then if it gets into the statistics of the next day, then we get some inconsistency in the system. Obviously a naive implementation will get wrong results here. There are various ways to handle it and ensure correctness, however the main issue is whether the developer needs to be aware of it while designing the application, or the compiler that takes the definition of these EPAs and creates the actual implementation should be the one which will do the job. My opinion is that if the developer will have to take care of such things in hard coding, the life will be quite difficult, as this is only one case of race condition, and it is better that it will be transparent to the developer. This will eat the cake and have it too --- both using high level tool that makes the programming easier and lower the total cost of ownership, and fine tune the semantics in a way that require typically dedicated, and even complicated programming. More about other aspects of semantic fine tuning - later.

On new event processing course and some ways to explain what event processing is

This is the Computer Science building at the Technion; yesterday I have started to teach an event processing course there (in the previous semester I have taught in the Information Systems Engineering program, which is my academic home for many years), the course is given as an "advanced topics in computer science" course, and the students that have shown up were mostly graduate students from CS as well of EE (there are software people in EE also). Unlike the previous semester in which the students projects evolved around validating the solutions that different vendors put for the EPIA book's website, this time I would like to concentrate in the project level about the view of those building event processing platform (this is a slightly different view). The first class is always an introduction, since the students don't have a clue about what event processing is, thus I am starting with some examples about what is it used for.

Always there is somebody who asks the question -- what is new, haven't such applications done in the past also? My answer is, consistent with the answer I am giving to this question for years, event processing does not bring any new functionality to the table, it takes functions that people have done in one or other way using regular programming and make it both -- easier to use since it introduces high level abstractions (analog to the abstraction of database query against the way we did database programming in the distant past, and other abstractions), and providing platforms that can execute these abstractions using optimizations specific to these abstractions.

I am also talking about the event-driven decoupling programming vs. the more traditional request/response.

One thing that I am making sure that the students will understand is that (like databases) event processing can be used for many different purposes and is not bound to a single type of application, or single industry. I am doing it by starting with 10 examples that are quite distinct (taken from chapter 1 of the EPIA book), and concluding by trying to generalize some type of applications, using the picture below (taken from an IBM academy study a few years ago that examined what event processing applications are), this is just one of possible classifications.

This is an important thing, since there are some misconceptions around.

As I have written in a previous posting, event processing does not have a single purpose.

Creating event aggregations to check key performance indicators is certainly an application, but not the only one; likewise "detecting threats and opportunities in the event cloud" is a phrase going around (I heard it first from Roy Schulte, but I don't know if he is the copyrighter), and indeed, this is an application of event processing, but many event processing application detect neither threats or opportunities, but serve for diagnosis, or operational decisions. The message - event processing is a discipline with set of concepts, and various ways to implement these concepts that serve many purposes that need event-based computing. I Shall write more about the course as it will further develop.

More on spatial event processing

This is a map of my home city, Haifa, which shows it northern part, I live in the south part outside this picture. I had some internal discussions this week about the role of location information in event processing, this is somewhat similar to the roles that time has in event processing, though the use of time is more pervasive, we see increasing number of applications that require spatial support. Some of the major utilizations of location information are:

1. Location as a context: The location is used to group events together as they serve as input for any event processing function such as: aggregation of pattern matching.
2. Location as enabler for spatial patterns: The locations of events is used in the function that determined whether a certain pattern is being matched.

Note that these two roles of locations are orthogonal.

An example for the first role: count all the police vehicles that are within 5 KM from a crime scene, where there are periodic events that identify the location of the police vehicles using GPS devise. In this case, location is done to find out which events are "in context", and the event processing function itself is a simple aggregation (count).

An example for the second role: A certain company experienced over one week, 20 events of breaking into employees houses, it an an attempt to figure out whether this is a work of a single person, the pattern that is matched is: the maximal distance between two breaking in events that occurred within a single night is less than 15KM. In this case the context is not spatial, since the events involved are determined by time ("a single night") and segmentation ("events related to houses of employees"), but the pattern that we are looking for is spatial, since it involves location.

As said, time can also play in the context role, in the pattern role, in both, or in none.

More on spatial event processing - later.

Towards an event processing manifesto - take one

The word manifesto brings to many people an association with the Communist Manifesto by Marx and Engels, which is an example how manifesto can be used, reused and abused. The term manifesto has been adopted by the computing community, and we have seen variety of works entitled: manifesto, some of them are: object oriented database manifesto, the manifesto for agile software development, and the business rules manifesto - just to name a few examples.

In May 2010 I am co-chairing (together with Mani Chandy and Rainer von Ammon) a Dagstuhl seminar, whose task is to compose "event processing manifesto", which we strive to make it as influential effort. A Dagstuhl seminar is a five days retreat in an isolated place in which a selected group of people are convened to make deep discussions about a certain area in computer science, many of these seminars are geared towards the academic community, but it this particular one we strive to keep the balance and have half of the participants from industry (vendors, independent consultants, analysts, and some innovative users), the number of attendees in each seminar in limited, and there is a big competition on this resource (and long waiting list after approved, we have put the application to do it in 2008). Here is a picture of Schloss Dagstuhl (located in Saarland, Germany):

I guess that I'll write a lot about the Dagstuhl seminar, the event processing manifesto, and related stuff in the next few months, so this is the modest starting point.

The intention of the manifesto is simple (but not easy) -- define what event processing is, what is the scope of event processing, are there mandatory features in event processing, what are the various options and variations, what are the synergies with other disciplines, and have as an appendix a call for action for the community: what are the research challenges and what are the pragmatic challenges.

I'll start today with some of my own thoughts on the scope issue.

I view "event processing" as a name of a computing discipline like "data management", "image processing" and "information retrieval" - relatively wide disciplines that cover wide collection of applications. This is also true for event processing. Alas, we still have some misconceptions even on the scope, I have used before the metaphor of blind people touching various parts of an elephant and concluding that each of them has identified what an elephant is, so this time I'll use another animal - ant whose sight is limited to its immediate environment.

I am using ant, since one of the great Hebrew poets has written a poem saying (in free translation): "my world is narrow as the world of an ant". Some examples of it:

1. people who view event processing as aggregation of time-series events, this is definitely an event processing application, but not the only one;
2. other people are looking at detection of anomalies within event collections, where the anomalies types are not predefined as what event processing is all about, this is indeed another event processing application, but not the only one;
3. yet other people are looking at real-time matching of predefined patterns with streaming events, which is somewhat different from the two previous examples.

One might claim that there is no event processing discipline at all, but there are very distinct areas that process events in different ways for different purposes, probably using different technique; while this may be a possible conclusion from the Dagstuhl meeting, my experience shows that there are many applications that require blend of event processing functionalities, and that the space is not discrete, but has some continuity, thus event processing in the wide sense should deal with all of them (and some more) and the synergies between them. So this is a point for discussion.

While the amount of people that will be involved in the actual composition of the manifesto is restricted, we'll strive to disseminate the draft to a wide discussion, both within the EPTS community and other avenues. As a comment, the Dagstuhl seminar is not directly related to EPTS, many of the attendees are not EPTS members, and some are members of adjacent communities and not the event processing community, however I think that EPTS should have a role in follow-ups.

On proactive behavior and application performance management

David Mavashev from Nastel has pointed me to his posting entitled:"who owns application performance management", it deal with the issue of whether operations group or application group own the SLA type performance management, and gets to a conclusions that this is being handled reactively instead of proactively. This observation is true for other areas as well, in most services operations, when a problem is created, it is solved in a reactive way; if in the supermarkets the line in the checkouts become too long, they add more checkout personnel, the same goes in service centers. Move to more proactive behavior will save time, money and getting patient-less people like me from getting high blood pressure. The use of proactive systems is still in its beginning, event processing is a key part in it, but there are some missing pieces to make it proactive, as the current technology is geared towards reactive systems. More about proactive extensions to event processing -- later.

Book review: The Decision Model

The last package from Amazon brought me the book entitled: "The Decision Model: A Business Logic Framework Linking Business and Technology" by Barbara von Halle and Larry Goldberg.

I have read a draft of the book before, at Barb's request, and wrote a review, from which one line was quoted on the back cover; I believe that the trend of modeling decisions and look at them in perspective of higher level abstractions will become more pervasive, and I view technologies like business rules, event processing and various analytics as building blocks in decision platforms that are going to be notable part of enterprise computing and managing much of the operational decisions. The book has three sections:

Section I puts the decision model in context, explains what is decision model, providing a background comparing decision models with data models, and positioning decision model in the SOA and BPM universe, it also explains the business value. This section is intended mostly for business users and managers that want to get an overview.

Section II explains the decision model in detail, discussing the structural, declarative and integrity principles, and comparing the decision model to the relational model, a motive repeating in previous books by Von Halle. There is even a chapter that is called "The decision model formally defined", but the formalization is in terms of explanations and tables, and not by formal writing, which I guess fits the target audience.

Section III is called "Commentaries" and is actually a collection of articles by the authors as well as by various people active in this space (John Zachman, James Taylor, Bruce Silver and more) discussing specific related issues such as: relations to enterprise decision management, standards, business decision maturity model.

Event based decisions and event processing are mentioned several times within the book, but are not thoroughly discussed. The focus is on facts and rules kind of terminology; a combined model that combines both rules and events is a natural extension, from the point of view of this decision model as well as from the point of view of event processing modeling. I have written before about decision agents, and since that time advanced on the thinking about such a decision agents framework. I'll revisit this issue in one of the following postings.

Bottom line -- the decision model book is a very good book to explain the book to various types of readers (the introduction maps the chapters of the book to the various types of users) and possible basis for both pragmatic foundations of rules technology, as well as a possible basis for a more formal basis for extended decision agents framework. More on this topic -- later.