My OMG talk

The picture is of Washington DC, as seen from the Arlington side. Two weeks ago in the OMG meeting in Arlington I gave a talk on EPTS and EP standards, the talk has been now posted on the OMG site, so you can download it from there. In the next few days we'll advertise the call for participation for EPTS, towards its official launch - stay tuned.

On Event Driven Marketing

Following my previous posting that talked about the effect of event processing on the life of some people, I came across another interesting article today that describes event-driven marketing, where event can be: "customer service call, a type of transaction, going through a spending level in a particular period, customer's birthday". This is different from traditional marketing, finding the right opportunity to issue personalized marketing is a capability enabled by processing events. More interesting applications - later.

Event Processing can change some people's life

After returning from a business trip, and a few days family vacation to celebrate the Purim holiday, I am retrurning to the Blog. One of my loyal readers, Eran Toch, has suggested the topic of this posting by sending me an article saying that the former NY governor, Mr. Spitzer, has started his way down of his high office, when caught by a "anti money laundering" program that looked for simple pattern of three events. Well - this is a classic event processing pattern, and it seems that it has changed the life of a few people (starting from Eliot Spitzer himself, and in a chain effect, many other people). This reminds me of a visit that a group of Eurpoean journalists visited the IBM Haifa Research Lab in Israel in 2003 or 2004, and I was asked to give them a talk about CEP (I am not sure even if we already used this name) - after giving them some example of use cases, all of a sudden one of the journalists had the association to Orwell's big brother, and after he mentioned it, there was some philosophical debate between some of the journalists to us, whether this technology has a potential to be a nightmare for human rights. My opinion is that many technologies can be abused, but can also used for good causes - criminals (and auditors) can use patterns to find flows in systems, and organizations can use similar patterns in order to trace the same criminals that are trying to take advantage of flows... No matter how used -- event patterns is very powerfull, and its impact on our life has not even scratched the surface of its potential.

Relativity and Event-oriented thinking

You may be familiar with the magnificent masterpiece of Escher called "Relativity". We see relativity everywhere -- on Saturday I've called home from my hotel in Burlington (since then I've moved south, and now I am in Hawthorne, NY) and told my daughter that I am sitting in the hotel waiting for the snow to end, in order to get out, and she said -- "how lucky you are to see snow", I have not felt lucky to see the snow, in fact, it kind of disrupted some pans I had, but in the world of a person that lives in Haifa, snow is a rare thing that occurs once every 20 years for a short time, so it is an attraction. At some point in my past I have been teaching a basic AI course, and the teaching assistant has taught the students Lisp. I have looked at the end of the semester on Lisp code produced by students - some got it right and wrote pure recursive program, and some wrote code that looked like Fortran thinking in Lisp.

The same relativity exist when dealing with event-based applications. Some people understand how to construct such applications, while other people think in other paradigms like request-respond, and are trying to do some blend here. It is perfectly OK to do an hybrid approach, but this should be a result of careful design, and not by chance. Bottom line - we need a methodology for constructing event-based applications and tools to support it - this will somewhat different kind of thinking... more about this topic - later,

The Babylon tower and event formats

Still in the USA, after the OMG meeting in the Washington DC area, I got to the Boston area, and now I am in Burlington, MA, visiting the (former) Aptsoft guys. When driving abroad I am renting a car with GPS (see above to determine which one), and it typically gets me to where I want - however, it is still not totally reliable, in the last day it confused me twice, one yesterday night to find the hotel, it told me to turn left, and meant in the next turn, not in the current turn, but it did not say so, thus, I found myself back on the I95, and had to turn around at the next exit, and try again. Today it sent me to some shopping center instead of the Aptsoft site, and after I ignored it and found it - the local people told me that the GPS maps have the numbers of the street in the wrong directions (starting from the other end) - well, sometimes the technology is fine, and the weakest link is the data it uses.

Talking about data, one of the topics that were discussed in the OMG meeting about standards is the topic of -- semantic/structural standards for events. I have used the term "Babylon tower" in one of the earliest postings in this Blog and meant that we have Babylon tower of languages - like the original tower who separated the languages. However, there is another Babylon tower that relates to event format - syntax and semantics, and here the problem is even harder, since there are multiple formats in multiple domains, nobody even made an inventory survey. One of the presenters said (and it is true in some cases) that 80 percent of the cost of building EP application is to set up the events from the sources and transform them to a processable format by composing adapters (hand coded, or by using transformation engines). This is some of the domains that need more investigation, and perhaps we need a meta-data standard here.

What can we learn from paradigms that succeeded ?

Today I have taken a few hours off my generally busy schedule, and visited the "National Air and Space Museum" in Washington D.C.; the globe shown above is showing stars as they were known in the 16th century (if my memory does not mislead me), the museum is very impressive (I have seen it 14 years ago, which was my last time in the Washington area, somehow it is not on my path). Anyway, I got here to attend (parts of) the OMG Technical Meeting. I was invited to give a talk in the MARS session about EPTS and EP related standards, as a prelude to a "complex event processing standards" workshop that will occur tomorrow (I'll be able to be only in the first half, need to catch a flight - so apologies to the other presenters, but I am quite familiar with the different players...). I have already written in the past on event processing standards more than once, so I'll not repeat myself -- if the organizers will post my presentation on a website, I'll advertise the URL. However, I would like to show one of the concluding slides I've presented (actually reused it, I've presented this slide before...)

This slide shows an example of a successful paradigm - relational database. In order to get there we still need a lot of brain power, we are not really there, we are now, as I have written before, in a similar situation to databases in the sixties, currently we can do standards on the interoperability issues, and on modeling - however, we still need to get to the "heart of the matter" -- a relatively simple theory of event processing -- similar to what relational theory did to databases. Working on a single frame of thinking will also add the brain power of the research community as done for relational database with the massive work on query optimization - in EP the optimization issues is not easier (I think they are more difficult, and cannot be done incrementally from other disciplines' optimizations). Are we there -- not really ! Can we be there --- I believe that the answer is positive... More than that, in order to get to the main stream of computing we'll need to do the equivalent to ALL items on the slide...

Some people who heard my talk were skeptical - which is fine, I am surrounded by skeptical people all my life, got used to it. They have good reasons to be skeptic, and I have a good reason to be optimistic -I have always been a best-case-scenario person, and know it does not work, but in the attempt one lands close enough... more - later.

On Forward and Backward CEP

Paul Vincent from TIBCO has recently posted some thoughts on "goal oriented event processing" . As usual, Paul sees event processing during the " through the rule spectacles, so I'll try to predent it in a wider perspective. It is agreed that the role of CEP (which by itself is part of larger EP) is to detect patterns over multiple events. What is forward and backwards in this context ?

• Forward: For each event -- check if this event completes some pattern
• Backward: Given a pattern -- check if this pattern have been satisified (in some time context) in the past.

Both are useful for different cases, and sometimes we need a mix; looking deeper at the differences:

• Forward is always event driven (it is done as a reaction to event)
• Backward is request-response (it is done as part of explicit request), note that the request may by itself be trigerred by an event that makes it hybrid model.
• Forward is used when patterns need to be detected immediately, it is most cost-effective to optimize for all patterns that events may complete then to look for all patterns and see if they are completed. Example: detect arbitrage opporunity between two exchanges.
• Backwards is used when patterns are done periodically, or as ad-hoc queries. Example: At the end of each month, Find all stocks that during the last month have satisfied the following conditions: The stock closing values at the end of the day were strictly increasing over a period of five consecutive working days, anywhere during this month; The stock value in the beginning at the end of the five days value was at least 30 percent more than its value at the beginning of the five days period.
• Mixed is used when in some cases -- a pattern may need "reinforcement" in past patterns -- Example: A person that has deposited (in aggregate) more than $20,000 within a single working day is a SUSPECT in money laundering. To reinforce the suspicion the following retrospective patterns are sought:
  There has been a period of week within the last year in which the same person has deposited (in aggregate) $50,000 or more and has withdrawn (in aggregate) at least $50,000 within the same week.
  The same person has already been a "suspect" according to this definition within the last 30 business days.
  If any of these patterns are satisfied – the event "confirmed suspect" is derived.

Forward and backward pattern detection can, of course, be implemented in various ways - business rules, SQL (stream and regular queries as backward), Scripts and specialized patter-oriented implementation.