On use of technology and unfairness

This is somewhat related to event processing, but the issue is more general. It relates to an article published today in the NYTimes, claiming that high speed traders gain unfair benefit over regular traders. The relationship to event processing is obvious, since typically event processing system run under the cover of the "high speed trading"; however, the point is whether technology progress should be stalled due to fairness consideration. I live in a country which has a socialist root; When I took the matriculation exams in high school, we were forbidden from using calculators, since at that time it was relatively expensive, and not every high school student could purchase it, I actually learned how to use slide rule, which I think can be found only in museums today, which is somewhat equivalent to calculator; it did not take long until calculators were allowed. Another example from the Israeli history, when the TV started in Israel, it was black and white, but programs that were purchased from other countries were in color, so people have started to purchase more expansive TV sets that supported color; the government thought it is not fair that the "rich" people can see movies in color, and the "poor" people can see movies only in black-and-white, and ordered the TV station (at these days it was only one, owned by the government) to erase the colors so nobody will be able to see anything in colors, even if they have color TV. However, Israel is also a high-tech country, and an engineer called Mooly Eden (who is now a senior executive in Intel) developed a device called "anti eraser" that restored the color, by reversing the erase function. At the end, the government gave in, and the "anti eraser" became obsolete. These are two examples where fairness considerations tried to bit technology progress, but not for long.

Technology is not the only issue of fairness, we send our children to private school, since we see that the public schools here are inferior; not really fair towards those who cannot afford it. the same goes for health. This actually goes to every facet of life, citing the immortal phrase from "start trek: the next generation" -- resistance is futile.

On logical and physical interpretations of EPN and EPA

My youngest daughter Daphna has finished last week her summer course in the Technion in the framework of the program of "science seeking youth". She studied her first programming course using "Microworlds", a variation of the rather old Logo language, this is of course been translated to lower level language when executes in practice, by this fact is totally transparent to those who program in Microworlds. I am using this analogy since there seems to be some terminology discussion going on recently about the terms EPA and EPN. These terms were introduced in the past by David Luckham, who used them to describe a physical operational view of event processing application. Thus, an EPA is mapped in 1-1 fashion to a software module, and the EPN describes the running software modules and connections among them using physical channels, the first version of the EPTS glossary reflects this view.

However, the way I am using the terms EPN and EPA is slightly different, the physical view is of interest to system administrators, but for the users, designers and developers, the logical view is more relevant, thus I am using these term in a logical way and not a physical way. In order to demonstrate the difference, let's look at the following simple example: There are many patterns that relate to the management of a call center, one of them is the frustrated customer detection: if a gold customer complains three times within a single day (possibly on multiple issues), then a supervisor should call this customer immediately.

However, there is a spectrum of ways that this application can be implemented in reality:

• It is possible to have a centralized implementation with a single software module that executes all the different functions within this applications, and actually the EPN is internal to this module;
• On the other extreme we can have a software module implements any single function instance, for example, an agent that detect the frustrated customer pattern for Alice, where a different agent detects the frustrated customer pattern for Bob.
• Another possibility is a context oriented implementation --- all patterns related to the Alice are processed within a single software module
• Yet another possibility is a functional partition -- there is a single module for detecting the frustrated customer pattern for all customers
  
• There can be also some more combination.

Should the user / system designer / developer care about it and build a different EPN for each variation ? In the past when event processing was hard coded in general purpose programming languages, the logical EPN was also the physical EPN, but one of the gains from using dedicated event processing languages are the ability to abstract the implementation out.
The actual mapping of functions to software modules is left to an optimizer, and can be dynamically changed based on change in the system behavior, load balancing etc.. Actually the paper we presented in DEBS 2009 is part of such an optimization scheme. Thus, the way I am using the term EPA is a single logical function and not necessarily a software module. In the EPIA book we are building our entire concept based on a logical level meta-language that can be translated to various implementations, and even programming styles. As said, there is also an interest in the physical realization of EPN, but it is more of interest to system administrators and implementers of event processing products, but it should be transparent to the user of event processing applications. More on this topic - later.

On the EPIA chapter about event consumers

Back to writing about the EPIA, the next chapter out, will be the chapter dealing with event consumers. As anyone who has developed an event processing application may realize, the event processing, is just part, and sometimes the small part of the picture, and in order to be useful, the consumers and producers have to be set. A consumer consumes the events (either raw events or derived events created by the event processing system), in this chapter (written by my partner Peter Niblett) there is a classification of event consumers, and plenty of examples. Here is some classification

And here are some cool consumers:


This, believe it or not, is an application that consumes Twitter events. This map following a bus that notifies Twitter anytime that it gets to a station, it reports where the bus is, and how many passengers are on board. You can read more details and even follow this service on Twitter (169 followers, when this Blog is written). The map is an event consumer that consumes the raw Twitter event and displays the bus route and location within a certain time interval.

This is another cool event consumer called "Ambient Orb"; this gadget is actually an event consumer, an event processing application determines the color of the ambient orb, the application can be anything from the state of your favorite stock, the traffic load outside, the security risk, or some key performance indicator - how many purchases have been done through the company's website today. The output is a color (in the picture it is green, meaning that whatever the indication is -- the status is OK).

More about the EPIA book - later.

On citations and the scope of the languages tutorial

Yesterday, I went with (most of) my family to watch the movie : Harry Potter and the Half-Blood Prince. I have to admit that among other things I am also a Harry Potter fan, and going to new Harry Potter film is for us a family event. The movie took some shortcuts from the book, so that some things remain fuzzy for those who did not read the book, but this is a very thick book, so there is a challenge to present everything in 2.5 hours movie.

Anyway, I am following with some amusement the discussion between Paul Vincent and Tim Bass on the scope of the event processing languages tutorial (0n the linkedin CEP users group). Typically I do not answer Mr. Bass' musings, as I believe in free speech and in the general intelligence of the readers, however, today I would like to deviate from this habit (just for one posting), and sincerely congratulate Tim on the achievement that he mentions -- his paper has been cited by an Australian patent application, Wow -- I am truly impressed !, and as I am a supportive person I want to really congratulate Tim, it is very flattering when one cites you.

There is just a tiny comment, Tim adds, in the height of his self-esteem:

I noticed that the Australian government's patent application did not reference your work a decade ago nor did it reference other "leaders" in the EPTS. (Sorry about that!) The 2008 patent application did not even reference "The Power of Events"... imagine that!

I did not look at the Australian patent application, but I have no reason to doubt the fact written, well.

BTW, I personally have a patent that has been accepted some years ago by the USA patent office, and last time I visited it was cited by 13 patents that were already accepted (not just applications) in USA;

And BTW -- the patent I am talking about won at that year the award for being among the top 5% IBM patents (among several thousands of patents that were accepted for IBM at that year) which brought me not only a financial award but also the title of "IBM Master Inventor";

And BTW -- this was only one of several patents I filed, not to mention around 70 papers I published (typically with other partners) on topics related to event processing over the last 20 years that were published in peer-reviewed journals and conference papers and were cited by hundreds of other papers and patents;

And BTW -- I am only one of 18 well qualified members in the EPTS event processing languages group, and some of them may have even better credentials than me.

This paragraph was not intended, heaven forbid !, to dwarf the achievement of Tim that his (single) paper has being cited, and he has all reasons to be proud and satisfied in his well deserved achievement, but it was just intended to put things in proportion and show that in professional credentials there are some others people who have been cited and recognized by the professional community as well...

And for the discussion issue about the scope of the languages tutorial, thanks to Paul for trying to defend the team, but for me this is a non-issue. The tutorial was not about general purpose languages that people sometimes use to process events, but about event processing languages defined as languages whose main purpose is to process events - very simple definition.

There are people who indeed hard-code event processing applications in C, Java, Perl, Payton, Lisp and other general purpose languages, but the scope of the tutorial was not to discuss general purpose languages, there are conferences on programming languages which deal with such surveys. I assume that there will always be event processing applications that will be written in general purpose languages as there are always programs written in assembly languages, or many systems that use file systems and not DBMS systems to store structured data. However, we observe that the use of dedicated event processing languages is growing, and I believe that in time it will cover substantial amount of the event processing applications. Our challenge in EPTS is to help mature that state-of-the-art to make it happen;
And BTW, we don't really need education about event processing or event processing languages.

As said, this was a one time comment, and I'll not tempt to turn it to a discussion.

Call for active participation in the 5th EPTS event processing symposium - Trento, September 21-23, 2009

As we are now in the high gear on planning the 5th event processing symposium, here is a call for active participation. It will be phrased in Q&A format:


Question: What is the Event Processing Symposium ?

This is an annual meeting, intended primarily for EPTS members, but open to other invitees up to the capacity limitation. It features some presentations and a lot of panels and discussions in the areas that are geared towards the understanding and advancing the state of the art and state of the practice in event processing. The previous symposium's program can be found on the ep-ts site.

Question: Where will the symposium take place?

The symposium will take place in Trento, Italy. The four previous meetings have been held in the USA, and since there is a big event processing community outside the USA, we thought of having the next one in Europe and call the large EP community in Europe to participate.
Trento university offered to host the meeting, they also host the symposium's website which currently includes information on the venue, and a registration form, and will also contain the program in the future.

Question: Who can participate ?

As said, the meeting is intended for EPTS members, however, other interested persons can apply and ask to participate. Based on the past experience, there will be some slots for non EPTS members. Participation is by invitation only, and requests for invitations can be done using the symposium's website.

Question: Are there participation fees ?

All EPTS activities are free of charge. Besides the symposium, there will be a banquet, to be held in Castel Toblino, a 12th century castle situated in the Toblino lake, in the evening of September 22. The banquet requires pre-registration and payment for the event and transportation. Details can be found in the registration form.
This is the Castle Toblino where the banquet is planned to be held.


Question: What is the notion of "active participation" ?

Unlike conferences where people come to hear presentations and maybe ask the presenters some questions, the notion of symposium is having the participants actively participate, this is done through the various panels that exist, which provide the possibility to present position statement, and discussions held with the audience. There are also some presentation opportunities. Below I am bringing an annotated plan of the symposium's session, annotating the opportunities for active participation, and then summarize the active participation opportunities.

Question: What sessions will be included in this year's program this year?

Here is an annotated list of the sessions. The meetings will last from 8:30AM to 6:45PM in the first and second day, and until 1PM in the third day.

The first day -- September 21 Monday.


Session I: EPTS - current state and further missions

We shall start (and end) the symposium by discussing the role of EPTS, what was achieved so far, and its missions for the future. This is an opportunity for active participation.

Session II: EPTS Use Case Working Group

This session will report the progress of the EPTS use case working group and will discuss the results and further activities.

Session III: Event Processing customers feedback

In this session we'll have a keynote address by a customer (TBD) and a panel made by customers providing feedback about the state of the practice and further directions from customers' point of view. This is an opportunity for active participation.


Session IV: Event Processing Research Contributions

In this session we'll hear several contributions from the research community about interesting research projects that are being held, there are already some requests for presentations, but there is a room for a couple of more presentations. This is an opportunity for active participation

The Second Day - September 22 - Tuesday

Session V: Reaching out – Event Processing and Business Process Management

During this symposium we shall have two sessions dedicating to "reaching out", dealing with the relationships between "event processing" as a discipline to other disciplines. The first of them is for relationships with the BPM area, following recent and planned discussions and meetings about even-driven BPM, the session will feature a keynote address by professor John Mylopoulos, a prominent researcher on the topic of Business Level Monitoring, followed by a panel that consists of people from both communities, and discussion with the audience. This is an opportunity for active participation

Session VI: EPTS Event Processing Languages Analysis workgroup

This session will report the progress of the EPTS languages analysis working group and will discuss the results and further activities.


Session VII: Reaching out – Event Processing and Network/Systems Management

This will be the second reaching out session for the area of network and system management which is a kind of event processing area that have been restricted to the boundaries of an application so far (see David Luckham's interview with Tom Bishop, the CTO of BMC).
The session will start with a keynote address of Dr. Kristian Stwart, the network availability architect in IBM Tivoli, formerly engineering director of Micromuse Netcool/Omnibus. This will be followed by a panel that consists of people from the two communities. This is an opportunity for active participation
Session VIII : Glossary and Interoperability session (tentative)

This session will deal with EPTS work groups about glossary and interoperability analysis, its format has not been determined yet. This is an opportunity for active participation


The Third Day - September 23 – Wednesday

Session IX: Architecture

This session will report the progress of the EPTS reference architecture working group and will discuss the results and further activities.


Session X: Grand Challenges

This is the closing session, it will feature a panel and open brainstorming about the grand challenges that are ahead of the event processing community, and discuss the preparation for the event processing Dagstuhl seminar planned for May 2010. This is an opportunity for active participation

An EPTS Business Meeting will follow on Thursday afternoon in the same place, but it is not an official part of the symposium.




Question: Can you summarize all the active participation opportunities ?


Here is the list:

1. Participation in the panel about EPTS missions.
2. Participation in the panel about customers feedback about the state of the practice.
3. Providing presentation in the event processing research projects presentation.
4. Participating in the panel about the relations between event processing and business process management
5. Participating in the panel about the relations between event processing and network/system management.
6. Participating in the panel about event processing grand challenges.

In addition, there will be opportunities for participating in open discussions.


Question: What do I need to do in order to participate in any of these activities?

If you are interested to participate in any of the panels or provide a research presentation, please let me know and I'll forward to the appropriate session chair; A session chair will be assigned soon and will be responsible for posing questions to the panel and participants, prior to the symposium, and moderate the session.

Question: What should I do now ?

If you want to participate, then use the symposium's website in order to register.

If you want to actively participate in any of the panel, then use Email to notify me about it.

If you know of anybody who might be interested to participate, and is not included in any of the notification forums and mailing list, then please forward him/her the link to this call for participation.

On Declarative programming in event processing

The city of New York has issued a 3 cents value stamp for its 300 anniversary, I am not in the stamp printing business, and wonder how long stamps will survive, but the blogger tool tells me that this is my 300 posting on this blog, in almost two years. Since recently I am dealing in languages, today I'll share some thoughts about declarative programming and its implications on event processing. John Bates told in his DEBS keynote about his experience, in the assumptions they had when developing Apama in the lab, and how these assumptions were slightly changed when got out of the lab. This is a phenomenon that I know well from a few times over my life that I have been involved with getting ideas from the lab to reality. Of course, the closer that your assumptions made in the lab are to reality, the faster time-to-market you have, and better success probability. Among the opinions that I used to hold is that declarative programming is always superior on imperative programming, in imperative programming, people need to work difficult needlessly, while in declarative programming, programming is more elegant and easier. I did not really changed my mind, however, reality shows that its ain't that clear cut. Getting to a neighboring area, it is well known half-secret in the business rules industry, that while all BRMS products support RETE style declarative inference rules, most users prefer to use a more imperative looking "sequential rules; I think that the reason is that in inference rules the flow is hidden, each rule is defined individually, and the flow is derived from the fact that some rule derives some fact, and some other rule has such a fact in its conditions, but this makes all the programming as "programming in the small", and the "programming in the large", controlling on how various rules relate to each other, does not exist in this programming style. It turns out that it is important for users to have visibility, and even control on the programming in the large aspects. Coming back to event processing, I believe in a combination of declarative programming for the programming in the small, i.e. to define any individual event processing agent, and making the programming in the large, i.e. the event processing network, explicit. This is not really an imperative model, since the EPN can be implemented in various ways, but it is not pure declarative either. We see that this type of programming model is getting traction.
I'll discuss the various dimensions of event processing languages in depth, in further postings - stay tuned.

Here is a picture I got recently from Matty Cooper, from EventZero, who made all the way from Australia to Nashville to attend DEBS 2009. This picture was taken during the event processing language tutorial

On the smarter planet and the big brother

An interesting comment to my previous posting on smart cities, said as follows:


Even if I am a big supporter of event based systems, I have a question on this topic. Can you explain me, who is the owner of the event based routing, filtering integration services?

I hope, that you do not think about weather the government, or some industrial companies, cause this would empower them to rule the world.

There need to be a self organizing way like e.g. the Internet, but getting there will be even a larger competition than introducing the web, because of already competing companies in that market segment.

I have written before on the big brother when talking about the previous NY governor whose felling out of power began when a computer program indicated him as suspicious in money laundering. This reminds my of somebody who told me that he is afraid that information is gathered in computerized systems and used for other purposes, that person lived in NY, and moved at some point to live with his girl-friend in New Jersey. He did not want to change his NY address for various reasons (among them not to bother changing his driver licence). He said that he never used EZPASS to pay for toll roads, since somebody could conclude from the fact that he drives every working day in the "garden state parkway" northbound, that he actually lives in NJ.

So the question is -- using all the smarter planet services will have the possibility to gather much information about the individual, who owns this information and what can this information be used to. Governments typically do not own infrastructures, but they have powers to make laws that will compel infrastructure owners to provide information, they do it today. Actually Internet service providers has a potential to know a lot about us. The same goes for events, if we'll have smarter planets, there will be a lot of events about individual person floating around, and if somebody will be able to join events from various sources, this somebody have a potential to know much about us. I guess that here there will be a need to have some legal structure that will prevent infrastructure suppliers as well as governments to abuse this information, like the bank secrecy laws in Switzerland.