On backwards windows

Continuing the series of posts about temporal windows:
Like the clock going backwards in this picture, one type of window is a window that is opened backwards.  The idea is that when event occurs, open a window backwards, to start either in some time offset, or by an occurrence of a specified event.   This is another type of window that is supported by some models, and emulated by using queries to databases that store past events in most models.   


An example is that an event processing system detects, using some pattern detection, that a person is a suspect in money laundering,  but substantiate the suspect is done by looking for another pattern on past event history.   This is known as "reinforcement" pattern use.  


The main difference between forward and backwards windows, is that in regular forward windows, events are being kept in the internal state only if they are required by some EPA which lives within this window, thus the events that should be kept in the system's state are well defined.   
In the backwards window case, since the window is not open when the event occur, then it should still be kept, thus we need to know of all possible backwards windows that might access this event in order to determine if an event should be kept,  or just keep all possible events (which in the most flexible case, since it allow unrestricted addition of backwards windows,  but may not be practical, as we may not know that an event even exists until some EPA that consumes it within a backwards window is defined.


This posting concludes the series of postings on temporal windows - for now.

On Forward and Backward CEP

Paul Vincent from TIBCO has recently posted some thoughts on "goal oriented event processing" . As usual, Paul sees event processing during the " through the rule spectacles, so I'll try to predent it in a wider perspective. It is agreed that the role of CEP (which by itself is part of larger EP) is to detect patterns over multiple events. What is forward and backwards in this context ?

• Forward: For each event -- check if this event completes some pattern
• Backward: Given a pattern -- check if this pattern have been satisified (in some time context) in the past.

Both are useful for different cases, and sometimes we need a mix; looking deeper at the differences:

• Forward is always event driven (it is done as a reaction to event)
• Backward is request-response (it is done as part of explicit request), note that the request may by itself be trigerred by an event that makes it hybrid model.
• Forward is used when patterns need to be detected immediately, it is most cost-effective to optimize for all patterns that events may complete then to look for all patterns and see if they are completed. Example: detect arbitrage opporunity between two exchanges.
• Backwards is used when patterns are done periodically, or as ad-hoc queries. Example: At the end of each month, Find all stocks that during the last month have satisfied the following conditions: The stock closing values at the end of the day were strictly increasing over a period of five consecutive working days, anywhere during this month; The stock value in the beginning at the end of the five days value was at least 30 percent more than its value at the beginning of the five days period.
• Mixed is used when in some cases -- a pattern may need "reinforcement" in past patterns -- Example: A person that has deposited (in aggregate) more than $20,000 within a single working day is a SUSPECT in money laundering. To reinforce the suspicion the following retrospective patterns are sought:
  There has been a period of week within the last year in which the same person has deposited (in aggregate) $50,000 or more and has withdrawn (in aggregate) at least $50,000 within the same week.
  The same person has already been a "suspect" according to this definition within the last 30 business days.
  If any of these patterns are satisfied – the event "confirmed suspect" is derived.

Forward and backward pattern detection can, of course, be implemented in various ways - business rules, SQL (stream and regular queries as backward), Scripts and specialized patter-oriented implementation.