Saturday, March 8, 2008

On Forward and Backward CEP

Paul Vincent from TIBCO has recently posted some thoughts on "goal oriented event processing" . As usual, Paul sees event processing during the " through the rule spectacles, so I'll try to predent it in a wider perspective. It is agreed that the role of CEP (which by itself is part of larger EP) is to detect patterns over multiple events. What is forward and backwards in this context ?
  • Forward: For each event -- check if this event completes some pattern
  • Backward: Given a pattern -- check if this pattern have been satisified (in some time context) in the past.

Both are useful for different cases, and sometimes we need a mix; looking deeper at the differences:

  • Forward is always event driven (it is done as a reaction to event)
  • Backward is request-response (it is done as part of explicit request), note that the request may by itself be trigerred by an event that makes it hybrid model.
  • Forward is used when patterns need to be detected immediately, it is most cost-effective to optimize for all patterns that events may complete then to look for all patterns and see if they are completed. Example: detect arbitrage opporunity between two exchanges.
  • Backwards is used when patterns are done periodically, or as ad-hoc queries. Example: At the end of each month, Find all stocks that during the last month have satisfied the following conditions: The stock closing values at the end of the day were strictly increasing over a period of five consecutive working days, anywhere during this month; The stock value in the beginning at the end of the five days value was at least 30 percent more than its value at the beginning of the five days period.
  • Mixed is used when in some cases -- a pattern may need "reinforcement" in past patterns -- Example: A person that has deposited (in aggregate) more than $20,000 within a single working day is a SUSPECT in money laundering. To reinforce the suspicion the following retrospective patterns are sought:
    There has been a period of week within the last year in which the same person has deposited (in aggregate) $50,000 or more and has withdrawn (in aggregate) at least $50,000 within the same week.
    The same person has already been a "suspect" according to this definition within the last 30 business days.
    If any of these patterns are satisfied – the event "confirmed suspect" is derived.

Forward and backward pattern detection can, of course, be implemented in various ways - business rules, SQL (stream and regular queries as backward), Scripts and specialized patter-oriented implementation.

Monday, March 3, 2008

On the Event Processing Language Style

A few days ago, I got back my car that has been damaged in an accident
(I had a replacement car, did not really use public transport in this period), and on Saturday travelled with my family (well - most of it) to the nature reserve of Alonei Abba known for its flowers (see one in close-up). Anyway, I have been thinking lately about what should be the ideal language for event processing. I have written before about the fact that some of these who has approached this issue, took it as an extension to existing programming paradigms, be it SQL or business rules; some are using imperative script language. However, in previous posting, I have argued that a business user should control the behavior of the event driven applications ,
While the original idea of SQL (like Cobol) was to address the non-programmers community, they both did not achieve this goal. Imperative lanaguge of course is also not an answer - besides this fact, in all of them the EP is imposed on abstractions that were intended for other purposes - and don't include abstractions that are relevant to event processing -- thus, the style of language requires paradigm shift, similar to the paradigm shift that SQL has done to database query lanaguages... more thoughts about the language - later.