On use of technology and unfairness

This is somewhat related to event processing, but the issue is more general. It relates to an article published today in the NYTimes, claiming that high speed traders gain unfair benefit over regular traders. The relationship to event processing is obvious, since typically event processing system run under the cover of the "high speed trading"; however, the point is whether technology progress should be stalled due to fairness consideration. I live in a country which has a socialist root; When I took the matriculation exams in high school, we were forbidden from using calculators, since at that time it was relatively expensive, and not every high school student could purchase it, I actually learned how to use slide rule, which I think can be found only in museums today, which is somewhat equivalent to calculator; it did not take long until calculators were allowed. Another example from the Israeli history, when the TV started in Israel, it was black and white, but programs that were purchased from other countries were in color, so people have started to purchase more expansive TV sets that supported color; the government thought it is not fair that the "rich" people can see movies in color, and the "poor" people can see movies only in black-and-white, and ordered the TV station (at these days it was only one, owned by the government) to erase the colors so nobody will be able to see anything in colors, even if they have color TV. However, Israel is also a high-tech country, and an engineer called Mooly Eden (who is now a senior executive in Intel) developed a device called "anti eraser" that restored the color, by reversing the erase function. At the end, the government gave in, and the "anti eraser" became obsolete. These are two examples where fairness considerations tried to bit technology progress, but not for long.

Technology is not the only issue of fairness, we send our children to private school, since we see that the public schools here are inferior; not really fair towards those who cannot afford it. the same goes for health. This actually goes to every facet of life, citing the immortal phrase from "start trek: the next generation" -- resistance is futile.

On logical and physical interpretations of EPN and EPA

My youngest daughter Daphna has finished last week her summer course in the Technion in the framework of the program of "science seeking youth". She studied her first programming course using "Microworlds", a variation of the rather old Logo language, this is of course been translated to lower level language when executes in practice, by this fact is totally transparent to those who program in Microworlds. I am using this analogy since there seems to be some terminology discussion going on recently about the terms EPA and EPN. These terms were introduced in the past by David Luckham, who used them to describe a physical operational view of event processing application. Thus, an EPA is mapped in 1-1 fashion to a software module, and the EPN describes the running software modules and connections among them using physical channels, the first version of the EPTS glossary reflects this view.

However, the way I am using the terms EPN and EPA is slightly different, the physical view is of interest to system administrators, but for the users, designers and developers, the logical view is more relevant, thus I am using these term in a logical way and not a physical way. In order to demonstrate the difference, let's look at the following simple example: There are many patterns that relate to the management of a call center, one of them is the frustrated customer detection: if a gold customer complains three times within a single day (possibly on multiple issues), then a supervisor should call this customer immediately.

However, there is a spectrum of ways that this application can be implemented in reality:

• It is possible to have a centralized implementation with a single software module that executes all the different functions within this applications, and actually the EPN is internal to this module;
• On the other extreme we can have a software module implements any single function instance, for example, an agent that detect the frustrated customer pattern for Alice, where a different agent detects the frustrated customer pattern for Bob.
• Another possibility is a context oriented implementation --- all patterns related to the Alice are processed within a single software module
• Yet another possibility is a functional partition -- there is a single module for detecting the frustrated customer pattern for all customers
  
• There can be also some more combination.

Should the user / system designer / developer care about it and build a different EPN for each variation ? In the past when event processing was hard coded in general purpose programming languages, the logical EPN was also the physical EPN, but one of the gains from using dedicated event processing languages are the ability to abstract the implementation out.
The actual mapping of functions to software modules is left to an optimizer, and can be dynamically changed based on change in the system behavior, load balancing etc.. Actually the paper we presented in DEBS 2009 is part of such an optimization scheme. Thus, the way I am using the term EPA is a single logical function and not necessarily a software module. In the EPIA book we are building our entire concept based on a logical level meta-language that can be translated to various implementations, and even programming styles. As said, there is also an interest in the physical realization of EPN, but it is more of interest to system administrators and implementers of event processing products, but it should be transparent to the user of event processing applications. More on this topic - later.

On the EPIA chapter about event consumers

Back to writing about the EPIA, the next chapter out, will be the chapter dealing with event consumers. As anyone who has developed an event processing application may realize, the event processing, is just part, and sometimes the small part of the picture, and in order to be useful, the consumers and producers have to be set. A consumer consumes the events (either raw events or derived events created by the event processing system), in this chapter (written by my partner Peter Niblett) there is a classification of event consumers, and plenty of examples. Here is some classification

And here are some cool consumers:


This, believe it or not, is an application that consumes Twitter events. This map following a bus that notifies Twitter anytime that it gets to a station, it reports where the bus is, and how many passengers are on board. You can read more details and even follow this service on Twitter (169 followers, when this Blog is written). The map is an event consumer that consumes the raw Twitter event and displays the bus route and location within a certain time interval.

This is another cool event consumer called "Ambient Orb"; this gadget is actually an event consumer, an event processing application determines the color of the ambient orb, the application can be anything from the state of your favorite stock, the traffic load outside, the security risk, or some key performance indicator - how many purchases have been done through the company's website today. The output is a color (in the picture it is green, meaning that whatever the indication is -- the status is OK).

More about the EPIA book - later.

On citations and the scope of the languages tutorial

Yesterday, I went with (most of) my family to watch the movie : Harry Potter and the Half-Blood Prince. I have to admit that among other things I am also a Harry Potter fan, and going to new Harry Potter film is for us a family event. The movie took some shortcuts from the book, so that some things remain fuzzy for those who did not read the book, but this is a very thick book, so there is a challenge to present everything in 2.5 hours movie.

Anyway, I am following with some amusement the discussion between Paul Vincent and Tim Bass on the scope of the event processing languages tutorial (0n the linkedin CEP users group). Typically I do not answer Mr. Bass' musings, as I believe in free speech and in the general intelligence of the readers, however, today I would like to deviate from this habit (just for one posting), and sincerely congratulate Tim on the achievement that he mentions -- his paper has been cited by an Australian patent application, Wow -- I am truly impressed !, and as I am a supportive person I want to really congratulate Tim, it is very flattering when one cites you.

There is just a tiny comment, Tim adds, in the height of his self-esteem:

I noticed that the Australian government's patent application did not reference your work a decade ago nor did it reference other "leaders" in the EPTS. (Sorry about that!) The 2008 patent application did not even reference "The Power of Events"... imagine that!

I did not look at the Australian patent application, but I have no reason to doubt the fact written, well.

BTW, I personally have a patent that has been accepted some years ago by the USA patent office, and last time I visited it was cited by 13 patents that were already accepted (not just applications) in USA;

And BTW -- the patent I am talking about won at that year the award for being among the top 5% IBM patents (among several thousands of patents that were accepted for IBM at that year) which brought me not only a financial award but also the title of "IBM Master Inventor";

And BTW -- this was only one of several patents I filed, not to mention around 70 papers I published (typically with other partners) on topics related to event processing over the last 20 years that were published in peer-reviewed journals and conference papers and were cited by hundreds of other papers and patents;

And BTW -- I am only one of 18 well qualified members in the EPTS event processing languages group, and some of them may have even better credentials than me.

This paragraph was not intended, heaven forbid !, to dwarf the achievement of Tim that his (single) paper has being cited, and he has all reasons to be proud and satisfied in his well deserved achievement, but it was just intended to put things in proportion and show that in professional credentials there are some others people who have been cited and recognized by the professional community as well...

And for the discussion issue about the scope of the languages tutorial, thanks to Paul for trying to defend the team, but for me this is a non-issue. The tutorial was not about general purpose languages that people sometimes use to process events, but about event processing languages defined as languages whose main purpose is to process events - very simple definition.

There are people who indeed hard-code event processing applications in C, Java, Perl, Payton, Lisp and other general purpose languages, but the scope of the tutorial was not to discuss general purpose languages, there are conferences on programming languages which deal with such surveys. I assume that there will always be event processing applications that will be written in general purpose languages as there are always programs written in assembly languages, or many systems that use file systems and not DBMS systems to store structured data. However, we observe that the use of dedicated event processing languages is growing, and I believe that in time it will cover substantial amount of the event processing applications. Our challenge in EPTS is to help mature that state-of-the-art to make it happen;
And BTW, we don't really need education about event processing or event processing languages.

As said, this was a one time comment, and I'll not tempt to turn it to a discussion.