The Babylon tower and event formats

Still in the USA, after the OMG meeting in the Washington DC area, I got to the Boston area, and now I am in Burlington, MA, visiting the (former) Aptsoft guys. When driving abroad I am renting a car with GPS (see above to determine which one), and it typically gets me to where I want - however, it is still not totally reliable, in the last day it confused me twice, one yesterday night to find the hotel, it told me to turn left, and meant in the next turn, not in the current turn, but it did not say so, thus, I found myself back on the I95, and had to turn around at the next exit, and try again. Today it sent me to some shopping center instead of the Aptsoft site, and after I ignored it and found it - the local people told me that the GPS maps have the numbers of the street in the wrong directions (starting from the other end) - well, sometimes the technology is fine, and the weakest link is the data it uses.

Talking about data, one of the topics that were discussed in the OMG meeting about standards is the topic of -- semantic/structural standards for events. I have used the term "Babylon tower" in one of the earliest postings in this Blog and meant that we have Babylon tower of languages - like the original tower who separated the languages. However, there is another Babylon tower that relates to event format - syntax and semantics, and here the problem is even harder, since there are multiple formats in multiple domains, nobody even made an inventory survey. One of the presenters said (and it is true in some cases) that 80 percent of the cost of building EP application is to set up the events from the sources and transform them to a processable format by composing adapters (hand coded, or by using transformation engines). This is some of the domains that need more investigation, and perhaps we need a meta-data standard here.

What can we learn from paradigms that succeeded ?

Today I have taken a few hours off my generally busy schedule, and visited the "National Air and Space Museum" in Washington D.C.; the globe shown above is showing stars as they were known in the 16th century (if my memory does not mislead me), the museum is very impressive (I have seen it 14 years ago, which was my last time in the Washington area, somehow it is not on my path). Anyway, I got here to attend (parts of) the OMG Technical Meeting. I was invited to give a talk in the MARS session about EPTS and EP related standards, as a prelude to a "complex event processing standards" workshop that will occur tomorrow (I'll be able to be only in the first half, need to catch a flight - so apologies to the other presenters, but I am quite familiar with the different players...). I have already written in the past on event processing standards more than once, so I'll not repeat myself -- if the organizers will post my presentation on a website, I'll advertise the URL. However, I would like to show one of the concluding slides I've presented (actually reused it, I've presented this slide before...)

This slide shows an example of a successful paradigm - relational database. In order to get there we still need a lot of brain power, we are not really there, we are now, as I have written before, in a similar situation to databases in the sixties, currently we can do standards on the interoperability issues, and on modeling - however, we still need to get to the "heart of the matter" -- a relatively simple theory of event processing -- similar to what relational theory did to databases. Working on a single frame of thinking will also add the brain power of the research community as done for relational database with the massive work on query optimization - in EP the optimization issues is not easier (I think they are more difficult, and cannot be done incrementally from other disciplines' optimizations). Are we there -- not really ! Can we be there --- I believe that the answer is positive... More than that, in order to get to the main stream of computing we'll need to do the equivalent to ALL items on the slide...

Some people who heard my talk were skeptical - which is fine, I am surrounded by skeptical people all my life, got used to it. They have good reasons to be skeptic, and I have a good reason to be optimistic -I have always been a best-case-scenario person, and know it does not work, but in the attempt one lands close enough... more - later.

On Forward and Backward CEP

Paul Vincent from TIBCO has recently posted some thoughts on "goal oriented event processing" . As usual, Paul sees event processing during the " through the rule spectacles, so I'll try to predent it in a wider perspective. It is agreed that the role of CEP (which by itself is part of larger EP) is to detect patterns over multiple events. What is forward and backwards in this context ?

• Forward: For each event -- check if this event completes some pattern
• Backward: Given a pattern -- check if this pattern have been satisified (in some time context) in the past.

Both are useful for different cases, and sometimes we need a mix; looking deeper at the differences:

• Forward is always event driven (it is done as a reaction to event)
• Backward is request-response (it is done as part of explicit request), note that the request may by itself be trigerred by an event that makes it hybrid model.
• Forward is used when patterns need to be detected immediately, it is most cost-effective to optimize for all patterns that events may complete then to look for all patterns and see if they are completed. Example: detect arbitrage opporunity between two exchanges.
• Backwards is used when patterns are done periodically, or as ad-hoc queries. Example: At the end of each month, Find all stocks that during the last month have satisfied the following conditions: The stock closing values at the end of the day were strictly increasing over a period of five consecutive working days, anywhere during this month; The stock value in the beginning at the end of the five days value was at least 30 percent more than its value at the beginning of the five days period.
• Mixed is used when in some cases -- a pattern may need "reinforcement" in past patterns -- Example: A person that has deposited (in aggregate) more than $20,000 within a single working day is a SUSPECT in money laundering. To reinforce the suspicion the following retrospective patterns are sought:
  There has been a period of week within the last year in which the same person has deposited (in aggregate) $50,000 or more and has withdrawn (in aggregate) at least $50,000 within the same week.
  The same person has already been a "suspect" according to this definition within the last 30 business days.
  If any of these patterns are satisfied – the event "confirmed suspect" is derived.

Forward and backward pattern detection can, of course, be implemented in various ways - business rules, SQL (stream and regular queries as backward), Scripts and specialized patter-oriented implementation.

On the Event Processing Language Style

A few days ago, I got back my car that has been damaged in an accident
(I had a replacement car, did not really use public transport in this period), and on Saturday travelled with my family (well - most of it) to the nature reserve of Alonei Abba known for its flowers (see one in close-up). Anyway, I have been thinking lately about what should be the ideal language for event processing. I have written before about the fact that some of these who has approached this issue, took it as an extension to existing programming paradigms, be it SQL or business rules; some are using imperative script language. However, in previous posting, I have argued that a business user should control the behavior of the event driven applications ,

While the original idea of SQL (like Cobol) was to address the non-programmers community, they both did not achieve this goal. Imperative lanaguge of course is also not an answer - besides this fact, in all of them the EP is imposed on abstractions that were intended for other purposes - and don't include abstractions that are relevant to event processing -- thus, the style of language requires paradigm shift, similar to the paradigm shift that SQL has done to database query lanaguages... more thoughts about the language - later.

On Actions

Looking at the EP blog-land, I found that two of the current postings has talked about actions:
Matt Rathera from Apama and Marco from Rulecore both talking about actions as part of CEP functionality.

So - let me make some observations:

There are two types of what people refer as "actions" --

1. "actions" that actually process events (enrich, transform etc..);
2. "actions" that use events as trigerrs to do something outside the "event world" .

Thesese two types are semantically different:

• The first type is part of the "event processing network", in the sense that it gets one or more events as an input, process the input, and produces one or more events as an output. This may be - simple, mediated or complex event processing, but there are "internal" actions -- and I would not call them actions - but classify them according to what they are doing (e.g. XSLT is doing transformation).
• The second type is the real "actions" - they occur outside the event world -- and are trigerred by the events. It is true that they can be arbitrary complex, but then set of actions with relationship among them is not really something that is unique to event processing -- this is known by workflows (or BPM in their contemporary name).

As good software engineering practices talk about "seperation of concerns", I don't view actions of that type as part of "event processing", but as part of the activities of event consumers (and producers). Of course, they may interact with the event processing part by receiving and emitting events - thus, causality relations among events can go across such actions -- however, there is no need to re-invent the wheel here, as event processing is part of a bigger whole - the bigger whole also contain BPM.

Of course, there may be justification in some cases to have mini-workflow attached to EP engine - but architecturally this logical seperation will help the model and thinking about it to be cleaner.

More - Later.

From manual event processing to event-based programming

Do you remember typewritters ? It became obsolete before some of you were borne, but I have used it a lot in my youth, since my handwriting required some efforts to interpret, I preferred to submit projects from junior high upwards using typewriter. Not very convinient in "word processing" - changes sometimes required re-typing of the page, sometimes fixes could be done with Tipp-Ex which also became a forgotten artifact. Today's word processing is much more convinient.

Paul Vincent has written in a recent Blog about "manual event processing". The example brought is certainly an event processing, there are many examples of event processing -- everytime that somebody calls and I pick up the phone, it is event processing, or if I am in a middle of a meeting, looking at the screen of my cellphone and decides not to take the call - it is also an event processing. In fact, manually we are doing a lot of event processing, and this is something very typical to human behavior. However, the programming paradigms that have been constructed, did not really intend to model the human behavior, and programming was developed, starting from Assembly languages and upward, in the assumption that we are first put values into - registers, variables, buffers, databases etc - and later when we wish to do something specific we retrieve it from wherever it is and use these values. Events in old programming languages such as PL/1 (my programming mother tongue) where used to handle exceptions. We have discovered the usefulness of teaching information systems how to process events only at a later phase -- first, by using (or abusing) regular programming to process events, and recently by various COTS that help process events -- however, the tools used are still rooted in the "store and then process" world -- SQL, business rules, scripts -- are all part of the traditional programming paradigm.

This is similar, in a sense, to do word processing using a typewriter --- we still need to invent event processing tools whose primary programming paradigm will be based on event processing as a major set of abstractions - and not as implemented on top of programming paradigm that don't match ---- well, still a lot to do on this --- more later.

Revisiting the Blog ** 2 again

I have never been in Arkansas, and in return - nobody from Arkansas have ever read my Blog - not surprisingly. What is surprising is that in the rest of the USA states (with the exception of North Dakota) have people who live there, or at least visited there and have read this Blog.

I have done some investigation on - who reads this Blog, when the Blog has hit 1000 distinct readers. I have decided to re-visit the statistics if the Blog ever hits 3000 readers - when I checked it this morning I found out 3217 distinct readers, so it is a good time to look at some statistics, actually this number is quite surprising, since I don't really think that there are so many people interested in this, and indeed some are probably passing by in the Internet roads, however the statistics show that around 45% returned more than one time. Moreover 569 people has visited the Blog more than 50 times (172 of them more than 200 times, where the number of Blog postings did not reach 100 yet), so I guess that this is provide the size of the community who really reads the material.

From Geographical point of view, it seems that I still writing mostly for the USA readers - 1530 distinct readers (almost half of the readers). In number of accesses - USA is the first, UK second, Israel third followed by- Japan, Thailand, Canada, India, France, Spain and Germany. However, when looking at the number of "distinct readers" - it is somewhat different order -- USA is still first and UK still second, but here Canada is third followed by - India, Israel, Germany, France, Australia, Holland and Sweden. There is coverage of all continents - some of our neighbors - "the Palestinian Territory" that does not have map yet in Google analytics, so I don't know where exactly, Egypt, Lebanon, Lybia and Iran are all represented. from South America - almost all access are from Brazil, some sporadic ones from other countries. In Africa - South Africa and some countries in the north - but also - Sudan, Kenya and Uganda. Also coverage of almost all countries in Europe and Asia - total 92 countries. As far as cities - the three big cities, in number of accesses are: Petach-Tikwa (in Israel), London and Bangkok;

the big cities in number of distinct readers are: London, New York City and Bangalore.

The biggest source of reference is Google in various ways, followed by other Blogs like - the CEP Blog, TIBCO's Blog and Rulecore's Blog and from David Luckham's site

The most popular postings were:

1. Agnon, the dog, playing and downplaying in which I stated the opinion that Business Rules are a possible way to implement CEP, but not the only way - which lead to several more postings investigating the relationships.

2. On Event stream Processing in which I stated my controversial opinion that the term "event stream processing" does not represent anything interesting, and the person invented it, Mark Palmer, thinks it should go away.

3. On the mythical event per second in which I stated that "event per second" really means nothing outside the context of a specific benchmark. I have dedicated more postings to this issue.

4. On bitter pills , a recent posting that answered Tim Bass' criticism about the state of the EP market, provided more optimistic view (followed by some constructive assertions in the next posting).

It seems that the popular ones are the macro-level - but I'll continue from time to time also to write on micro-level things, since it seems to have its audience also. The most popular to those who got through search engines were related to XTP and context terms - so I need to get back to them also.

So - thanks to all the readers -- more event processing related postings - later.