On Occurrence time: a footnote to the UAL fiasco

As a past Dungeon Master the word crawler always reminds me about the "carrion crawler", a monster you can see in the picture above, but recently a combination of the allmighty Google crawler, and automatic trading programs based on event processing has caused a fiasco that crashed the stock of United Airlines, some of the blogs have referred to it: Brenda Michelson in her Blog have talked about the butterfly that lead to the computer glitch. Mark Palmer thinks that news should be regulated (some people I know who were borne in countries were news are indeed regulated shiver to hear the idea that news - of any type - should be regulated).

I will not go back to the story, but as a footnote - two issues come to mind - event validation and the issue of occurraece time. So I'll write today about occurance time since it is easier...

The works in the temporal area are talking about several time dimensions - the bi-temporal model talks about: transaction time -- the time that a fact is recorded, and valid time -- the time interval in which the fact is valid. In event processing we also look at a bi-temporal time similar to this: detection time -- the time that the message that represents the event was detected by the processing system, and occurence time -- the time which the event happened in reality (occurrence time can be considered as the starting point of a valid time that ends when the event becomes irrelevant, but let's get it out of the scope and concentrate in occurrence time).

Some of the implementation of event processing base the order of event on the detection time, some support occurance time, and some base the built-in temporal capabilities based on detection time, and enable defining times as an an attribute, but then the temporal operators have to be hand-coded as regular predicate.

One of the common fallacies is that detection time is good enough as a metrics for temporal operations on event (e.g. trends), first - event from the past can suddenly pop up out of the blue (I know a person who has an habit to catch-up in Email every two weeks or so, and answer to the Email before realizing that there has been a whole thread of Emails that make answering the original Email quite obsolete), second - the order may not be kept even if the delay from the occurance time to the detection time is very small. The order of medical exams may not be consistent with the order of results reaching, and knowing the real order may be important for the differential diagnosis.

Thinking about standard structures for events -- I would think that having "standard header" with some mandatory properties for each event - is a good candidate for having standard (I am less optimistic about standards for the content of the event), and in the header - the occurrence

time should be a mandatory.

Occurrence time has some inherent issues associated with it - but I'll discuss it another time.

On events about events

Today, our entire family has travelled to the "instruction basis" of the Israeli Navy, where my (second) daughter had the ceremony of finishing her basic training, above is the logo of the Israeli Navy, the Navy is the only branch of the army who has ceremonial white uniforms, so it was all white day...

This is also, somehow, the time in the year where many things related to event processing happen. Yesterday, IBM (the company who pays my salary) has done what was known internally as the "events event" in Boston for analysts and press, some report on it exist in the media, I think that David Berlind's report in "information week" is the most thorough one and includes video and slides. I have also seen several announcement by other vendors.

Next week there will be the two back-to-back events, the Gartner EPS - second of its kind, follows by the EPTS event processing symposium - which is fourth of its kind, but first one after the formal EPTS launch. The first one to hear analysts reports and some other talks geared to customers, the second - highly interactive, discussion oriented meeting geared to the EP community. This is primarily for EPTS members, but we have invited some guests. The EPTS meeting will be video-taped by CITT, and will be posted on the Web, so additional people willl have access to the material.

I'll blog more on these events next week. Looking forward to see the active those active in the community and also some new faces...

A footnote to the streamSQL paper

The comment that my good friend Claudi (AKA Pattern Storm) made in the complexevents forum made me curious to actually read this paper; reading it I had the uncomfortable feeling that since people insist to use a language style that implies type of thinking about event processing, and this creates semantic problems which they try to solve by use the same type of thinking, with more complicated constructs.

I'll use one simple example taken from the paper, which they had to deal with semantic problems that were caused by the way the language semantics.

The scenario (translated to my language - without the "streams") -- Events are reported about cars that move through some segment of the road; each event consists of

There are also simultaneous events, i.e. several events that happen in the same time unit (what ever the time granularity is). The inputs are events of this type, the output is - for each event, generate a derived event that include the original attributes of the events and the average speed of cars in the same time unit. If you want to see the types of problems that the SQL implementators see in this simple example, read the streamsql paper. Instead of discussing SQL, I would like to show an alternative way to think about the same problem.

The slide below shows an alternative way to think about this problem - this is a very simple EPN (Event Processing Network) which has two functional agents, one producer (e.g. an event emitter that create events from video stream produced by a camera that looks at the road) and one consumer (whoever wants to see the output events)..

The two agents work under the same temporal context (it can be spatio-temporal if we also want to group by road segment) - in this case, a temporal context is opened and closed every beginning and end of 1 time unit.

• The raw event is called "car position event" and it goes to both agents.
• The first agent is an aggregator which calculates (incrementally) the average, since it is bounded to the context, the average is of events from the same time unit, at the end of the time unit it produces a single event "speed-average-event" with the structure

• The second agent is a "pattern detector" which takes two input events - the "car position event" again, and the derived event "speed-average-event"; the pattern that need to be identified is AND, and the "speed-average-event" for that agent has a consumption policy of "reuse" (which means that if an event can be used for multiple patterns). The agent produces a derived event - for each AND pattern that consists of the "output-event" whose structure is:

This EPN does not involve "streams" - the thinking is "event oriented" and it attempts to provide natural thinking about event processing functionality.

Comments:

1. This is rather simple example, can also be solved by putting the average speed event on a global state (or event store/database) and then enrich it back - but the event-oriented is closer to the spirit of the original example which work on streams.

2. Aggregator and pattern detector are type of agents, there are some (not many) more types. Typically, an event processing network consist of multiple types of agents.

3. "Pattern Storm" claims that stream SQL ignore causality. One can view the relation between input events and output events of the same agent as a causality relation (he is using another scenario from the paper), and this can be set while defining the EPN.

One general comment (not related to this posting) - to "anonymous" - I'll gladly answer your question if you'll send it back and identify yourself. I don't publish anonymous comments.

I can post the solution to the rest of the examples in the stream SQL paper if anybody is interested...

On AITC (Arab Israeli Technology Center)

Today I am blogging about an "off topic" issue -- an initiative that occupy part of my time -- AITC (Arab Israeli Technology Center). Some background - as you probably know, Israel is high-tech country, out of the 3.3 Million people in the Israeli work-force, 200,000 work in the ICT area (AKA "high tech"), something around 6%. This is an impressive number, considering the fact that the salaries and thus standards of living in the high-tech industry is relatively high, and that the supply of high-tech professional does not meet the demand. However, the Israeli high-tech does not have even participation of all populations. There are two populations that are extremely under-represented, one is the ultra-orthodox Jewish population, which parts of it chose to live in isolation, and not seek employment outside their community - they are outside the scope of this discussion now, the other under-represented population is the Arab population, which is around 20% of the population of Israel (I am talking about the Arab citizens of Israel, not the residents of the Palestinian territories). Some survey that has been done revealed that the participation of the Arab population in the Israeli High-Tech is around 0.2% - two orders of magnitude difference. Why did it happen? Twenty years ago much of the high-tech was associated with the defense industry. While there still is a defense industry, most of the high-tech is civilian, furthermore - many multinational companies: Intel, IBM, Oracle, HP, SAP, BMC, CA, Motorola, Microsoft, Google - to name a few - have substantial activity in Israel, many of them have also acquired Israeli start-ups and continue operate them as part of the bigger company. However, the image of high-tech being closely associated with the defense industries, who don't occupy Arabs, is still pervasive in the Arab society. One of the Arab Israeli notable families decided to break this reality, and to launch a project that will first convert people from the target population who already have academic degrees in various engineering, science or mathematics disciplines, in the sequel establish a private academic institute, and then also incubation for start-ups. This family has recruited a known (and controversial) person in Israel, who is known in his ability to push things, as the President of this center, and he recruited me to be the Academic Director of this center (based on some bullets in my CV, I guess)... I have decided to take the offer (in my spare time... at least for now), in order to make contribution to the society. The first mission has been to recruit the support of the Israeli high-tech industry, and this has been relatively easy. We have now full support, including participation in advisory committee, ability to do projects, and willingness to recruit graduates of this program (well - if we prove that we meet the high standards of the industry). I also compiled a curriculum draft, which have gone major revision after consulting with the different companies (we cannot make everybody happy, but it seems that everybody is happy about something, and at the end - this make them happy). Besides the high-tech industry, various politicans are also showing involvement (I am not crazy about politicians in general)... and the Israeli president, Shimon Peres, is giving his sponsorship (which will earn the entire team a dinner with him in 2 months or so)...

Now there are preparations for the launch, and the actual studies are planned to start in February 2009. I am in a phase of recruiting teachers - blend of people from academia and industry...

BTW - the picture above is of Nazareth, the biggest Arab town, where the center will be located.

How does it relate to event processing? Well- this launch is a big event, but like David Luckham's example of the stock market crash, it does not happen over a day, and consists of many smaller events. So - I'll blog in th future, about some of these smaller events that will happen. More -later; back to EP Blogging.

On subjective view about discussion agenda

I tend to believe in positive thinking, my previous Blog posting had a somewhat negative tone, since it explained why certain discussions over the blog-land turn me off (I am quoting Giles Nelson from Apama). Sometimes one has to voice a negative tone to make a point, but today I would like to quickly return to positive tone, and say what I do want to see on the blog-land. This is, of course, my own subjective view, as noted in the title, and other people may have other interests.

First - there are both macro-level issues and micro-level issues that worth discussing; macro-level issues relate to the "event processing" discipline in general, while micro-level issues relate to particular topics that are of interest. I'll partition the discussion to -- applications and technology.

Macro issues about event processing applications:

• What are the types of event processing applications, what are the motivation/ROI for using event processing?
• What are the conditions for EP COTS to "beat" hand-coded EP
• Business evaluation about trends, markets, industries etc...
• Relationships with other areas - BPM, SOA, XTP, CODA, Cloud computing and more.

Micro issues about event processing applications:

• Description of interesting applications
• Experience report of customers - what worked well? What needs improvement.
• Patterns of use and methodologies for best practices.

Macro issues about event processing technology:

• Technology trends and Research trends
• Technology challenges and engineering challenges
• Standards

Micro issues about event processing technology:

• New functional features (e.g. supporting uncertain events)
• Non-functional requirements (e.g. what is latency?)
• Languages
• Development tools
• Optimizations

Some of the recent postings on Blogs are exactly on such topics: The Smart Order Routing application, The Value of state and more.

This is probably not a complete list, but indicates on directions, this is more or less the topics that will be discussed in the 4th event processing symposium, and in the next postings on this Blog - I already have a backlog of several topics to post...

Is my cat CEP or not?

One of my favorite cinema artists is Mel Brooks, and one of the great films of Mel Brooks is known as "silent movie" (see flyer above). To those who don't know or don't remember, silent movie is a film about doing a silent movie, and most of the plot involves around Mel Brooks and his gang going to famous actors asking them to participate in a silent movie, they agree and this sums up their participation, since their voice is not heard; there is one exception - when asking the world famous mime Marcel Marceau to be in the film he replies, "Non", the only spoken word in the entire film, since he talked, he did not participate in the silent movie.

So here is my dilemma -- recently the blog-land is full of postings on the topic "is X CEP?" - personally I don't find that this debate leads anywhere, so my inclination is not to participate in this discussion, so I'll contribute my part by explaining why I don't find this discussion interesting.

There are different reasons why I don't find it interesting to spend time on these discussions are:

1. I don't see that the results of this debate are important to the reality.


2. One can debate terms forever - but this is not useful, there is a better way.


3. This debate will not converge anyway, the same arguments are being repeated and it does not enrich the reader's intellect to read them again.

Let's start with the reality -- what the current vendors attempt to do is to build generic products that will enable to develop various sorts of applications in the area of event processing. This is different from the past, where event processing has been done in hard-wired, hand-coded, ad-hoc fashion to build a single application. While any single application may be challenging, the challenges in building generic products that will be a cost-effective alternative to hand-coding, are somewhat different from building ad-hoc applications. The content of these products is determined by several factors - not by the boundaries between "simple" and "complex" - there are multiple dimensions of complexity, and different applications may exhibit different type of complexity (functionality, topology, scalability, event types)... No matter where we'll put the border between the "simple" and "complex", a generic product will have to support both sides of the border. Bottom line -- the importance is ROI for the customers and not whether something is called "complex" or not.

Debate over terms: when we have done the first event processing symposium, in March 2006, the conference started by nine presentations about the "state of the art", and a very quick observation has been that each of us has invented his own set of terms. Consequently we have decided that the first community effort will be to produce a glossary edited by our non-vendor colleagues, to reduce bias. They have collected input from many sources with different opinions, sometimes strong opinions (it is amazing that people are so caring about terms). The glossary has been published recently and its announcement was endorsed by many people in the community. I am sure that nobody is happy about all terms, but as a compromise among all opinions, the editors have done a pretty good job.

Personally I don't have strong feeling towards terminology, so the glossary definition is a possible one, there are others as well, there is no absolute truth here, but - I think that is good to have a precise definition of terms that are accepted by large community vs. continue debating on terminology forever, even if I personally would have defined it otherwise. Among the possible interpretations, the editors chose to define CEP: Computing that performs operations on complex events, where complex event is defined as:An event that is an abstraction of other events called its members. As far as I am concerned, this is what CEP is.

Last but not least -- If somebody decides to insist on his own (different) definition of a term T, and judge everything according to this own definition, then obviously, if a term T have two different defintions, then X may be T according to one definition, and not T according to the other definition, when the two sides stick to their definitions, then no one will convince the other anyway, and arguments will still repeat themselves. Actually, I have not learned any new insight from this discussion. Have I mentioned that the arguments repeat themselves? in case that I have not done it , I'll mention that the arguments repeat themselves.

Bottom line -- is my cat CEP ? well - I don't have a cat(I wonder if Roger King who has written the famous article "my cat is object oriented" had a cat).

There are plenty of more useful topics to Blog about - so no more on that issue.

On event processing as a paradigm shift

The readers are probably familiar with this picture where it shifts between seeing two faces facing each other (in black) and a white vase. I came across a (relatively) new blogger in this area, Pern Walker, blogging for Oracle's "event driven architecture". The title of the posting is:

Event servers, a disruptive technology. It describes the components of the (former) BEA framework, nothing new here, but the interesting part is the conclusion - event processing COTS is a disruptive technology, it displaces custom code in event processing, since it is more cost-effective.

This reminds me of a discussion we had in May 2007 in the Dagstuhl seminar on event processing, it was a night discussion with wine, and was lead by Roy Schulte, the question that Roy has posed to the participants : "Will Event Processing (EDA) become a paradigm shift in the next few years or not?”.

Today, I don't intend to answer this question, instead I'll post part of the discussion in Dagstuhl that included observations about "paradigm shifts" (thanks to my colleague, Peter Niblett, who documented the entire Dagstuhl seminar). I'll return to this topic again, with my (and maybe other) opinions about the answer, after the EPTS event processing symposium

Observations (from the Dagstuhl discussion):

• Paradigm shifts can’t happen if there are too many barriers; have the entry barriers for "event processing" already been removed? ;
• Paradigm shifts are more likely to happen when adopters decide they need a whole new avenue of applications; they are less likely to happen as a way of re-engineering existing systems. For example the German population will reach 1:2 old: young ratio by 2020 so this requires a paradigm shift of healthcare models. Can we identify new avenues of relevant applications?
• Paradigm shifts usually happen as a result of some external change, not just because of innate strengths of the technology itself. Can we identify such external changes?
• Standardization is not necessary for a paradigm shift, but good, appropriate standards (de facto or otherwise) certainly help

Another question is to where in essence is the "paradigm shift" - is it the decoupled "event-driven" paradigm ? is it the "complex event processing", i.e. ability to find patterns on multiple events? is it the entire processing framework as the Oracle's Blog claim?

More - Later