Saturday, December 1, 2007

On CEP and IEP


Ambidexterity is a good property for a boxer, he can decide when is better to attack with his right hand, and when to attack with the left hand (I am part of the left-handed minority, should write sometimes a post about being left-handed in the Right-hand people's world). Likewise, there are problems in the event processing space that can be solved by deterministic means (rules, queries, scripts, patterns --- chose your favorite religion), and problems that are solved by stochastic means -- using probabilistic networks, machine learning etc.. (AKA IEP - Intelligent Event Processing). When there is a pattern that need to be traced , to check compliance with regulations, and the pattern is well-defined - then a deterministic approach should be used; when there is a need to dynamically change the traffic lights policies to have minimal waiting time of vehicle, there is a need to predict the traffic in the next few minutes - this is a non deterministic problem and require some stochastic tool (BTW - my student, Elad Margalit, is looking at the traffic lights issue as his M.Sc. thesis). Event Processing Platforms should include various types of functionality - which brings to another discussion on the "actor/agent" architecture - which I'll refer to in one of the next posts. more -later

Wednesday, November 28, 2007

Introducing DEBS 2008


DEBS 2008 is an international conference that is becoming the "flagship" conference of the Event Processing community. The Research discipline of "event processing" has its origin in various disciplines - databases (active databases, stream processing); verification; simulation; rules; programming languages and distributed computing. DEBS have been for several years a conference of the "pub/sub" community that came from distributed computing, they still control this conference (I have suggested to get some people out of this community to the steering committee but it has not happened yet !!). In the course of establishing the "event processing" community, a research annual conference is an important step, as you can see in the call for papers, the list of topics is quite comprehensive, in order to do that, there is an agreement for 2008 to hold other research workshops/conferences in this area, such as EDAPS and join forces
on DEBS 2008 (however, we shall still hold EPTS conference next year, as the goal is different).
In addition to the research program, DEBS 2008 will have:
  • Industrial session - where vendors/customers can present
  • Demo session - to demo products/prototypes
  • Tutorials - in-depth study of the state-of-the-art

There is plenty of time to prepare - deadline to submit in all these category is in March (the conference itself is in early July) -- last but not least -- an opportunity to visit Rome (I have never been there outside the airport).

Monday, November 26, 2007

Non events again - much ado about nothing

Shakespeare fans (and theater fans in general) love the wonderful play : Much Ado about Nothing.
I recalled the title of the play when seeing that different people this week have written various things about the issue of "non events". Tim Bass thinks that this is just a time-based event, there have been various answers to David Luckham's question on the CEP forum. I actually did not think that I'll write second posting on this topic, since I don't think that this topic is that important, it is just one of various basic patterns. However, I'll try to provide more formal discussion on what is called "non-event event".

Let's defer the name discussion for a while - and try to define what it is:

This pattern is a function of two things (as many patterns are): an event E, and a context C, for simplicity let's assume that this is a temporal context only - i.e. a time window.

The definition:
  • Forall t: t is a time-stamp and t is an element of C, the event e does not occur in C.
We can of course enable a condition - saying "event E that satisfies predicate P", but this does not change the principle.

The first question is - does the fact that an event E does not occur in a time-stamp t is an event - i.e. is a "negative event" an event by itself - the answer is NO - the fact that an event does not happen in a certain time-stamp is not an event, since an event is something that happens and not something that does not happen.

The second question is - is the definition of pattern shown above is an event ? the answer is - like any other pattern, the detection of a pattern may create a derived event. The semantics is that the detection of this pattern may be interpreted as a situation that requires some reaction, and in this case, we chose to derive an event that designates that this situation has been reported, this does not change the answer to the first question - the negative event at any time-stamp is still not an event!

Now - to the question of WHEN such a derived event is reported. In general, a derived event may be reported immediately when the pattern is detected, or in a deferred mode (this term is taken from active databases). There are patterns that only make sense in deferred mode - this pattern is one of them - since it is only make sense to talk about the fact that an event did not happen during a time-interval - at the end of this time-interval. Deferred report of derived events typically occurs at the end of a temporal context, but it can also have something like temporal coherence condition, such as: report within 2 hours. The reporting of the derived event is time-based, but this is true not only for this pattern, but to any derived event that is reported in deferred mode.

Last but not least -- the name : non-event event is a bad name, since it has a flavor of negative events as noted in question 1. The names - "absent pattern", "not pattern", and "time-out pattern" are used for it. Time-out can be a good name - but need to think if it general enough.

Back to other topics -- in the next post.

Saturday, November 24, 2007

Is there a non-event event ? on absence as a pattern


Recently, there were some discussions, related to the glossary, about the term "non-event event" , this seems to be a funny notion that seems to contain self contradiction, similar to "soapless soap" I don't think that this term is very good - I prefer to call it
"absent event" .
Now the questions - how is it really defined ? and is it really an event?
As for the first question: an absent event is a CEP pattern that is defined within a context. Taking some examples:
  • There have no been any major financial transaction during the working day for the customer John Galt.
  • The Pizza order has been received, but the delivery did not arrive within 40 minutes.

In these two examples we are looking for - things that did not happen within a certain context; the first example is the context of - "John Galt's financial activities within a single working day", the event that did not happen - "major financial transaction" is by itself a pattern that need to be defined and detected (say - a transaction with financial value of more than $10,000). The absence is on all time stamps that belong to the context - so the formal definition of the absence pattern of event e in context C is defined as : For context C, for all time-stamp t that belongs to the temporal extension of context C, event e does not occur in time-stamp t.

Let's check this definition in the Pizza order - the context here is a certain order, and its temporal extension starts when the Pizza order is confirmed, and expires after 40 minutes (which is the public commitment of the take-away Pizza shop for Pizza delivery).
This is also an example of the use of context as a major abstraction that gets the thinking about event processing closer to the way people think.
Now - the question whether the detection of an absent pattern is an event --- the answer is - a pattern detection by itself is not an event (although it may be an event from the point of view of the internal management system of the EP platform), however, it may create a derived event, which has some structure - for example - the derived event that is created in the wake of the lack of Pizza delivery consists of : date, time, pizza store id, value of order - this event may be enriched with summary of past late deliveries from the same store, and then notify or orchestrate something.

Wednesday, November 21, 2007

On Real-time, Right-time, latency, throughput and other time-oriented measurements

The illustration shows different types of "real time" types. This posting was inspired by a comment on a previous posting - trying to do some order in several notions of time. First, the term real-time is frequently used in conjunction with event processing. The popular belief is that real-time = very fast, but this is not really what real-time is about. real-time can be thought of a deadline accompanied by a utility function that designates the damage from missing the deadline. In this illustration there are four type of real-time:

(a). Soft Real-Time: there is a sense to react after the deadline, but the utility decreases (maybe fast) and at some point gets to zero - no use to do it at that point, but no damage.
(b). Firm Real-Time: The utility go immediately to zero when the deadline is missed - no use to do it after the deadline, but no damage.
(c). Hard Essential: Missing the deadline - the utility function goes to a constant negative value; there is a constant penalty.
(d). Hard Critical: Missing the deadline - the utility function goes immediately to "minus infinity", means: a catastrophe will happen.

One can, of course, define the real-time utility function differently, and create more variations.

So - Real-time is not about fast, but about missing the dead-line. The linkage is there, if the dead-line is very short (need to react within 1/1000 of a sec), but many dead-lines are longer than that -- seconds, minutes, hours or days - depends on what is needed to react to - e.g. the contract that we have with our local electricity company says that when a problem is reported, they should start fixing it within 2 hours; the deadline for delivery of Pizza (otherwise it is free of charge) in one of our local delivery centers is 40 minutes - so most the world typically does not work in milliseconds.

When talking about "quality of service" measurements they can be either statistical: in 90 percent of the cases, the deadline should be reached, or individual: for all cases the deadline should be reached. Typically there are different scheduling strategies to achieve each of them - for the individual case it is important to have a consistent deterministic reaction, and this is a source of the various implementations of Real-Time Java - since Java is non deterministic by its nature of garbage collection that happens in undetermined times and lasts for undetermined duration, which may work for the statistical case, but not for the individual case. The Real-Time Java does not stand for Java which runs much faster, but for Java in which the garbage collection differences are smoothed, thus its reaction is (more) deterministic.

What other measurements are there ? in event processing - latency can be measured by the end-to-end (from the time that the event happens in reality to the action being taken in reality), can be related only to the event processing part (from the producer sends the event, until the time that the consumer(s) receive the consequences of this event), and can relate to a specific function (agent) in the event processing network, so when latency is mentioned - it should be defined - what is really being measured. The deadline typically refers to time constraint of the latency.

The term throughput designates the amount of events that the system can get as an input in a given time interval (1 second, 1 minute etc...), throughput does not necessarily entails that all these events are handled within this time interval - it may be the case that the events are put into a buffer, and all of them are handled within the next 6 hours - so in this case, the throughput is mainly determine by the capacity of input channels and buffers and not in the speed of processing. Of course, there are applications that require high throughput together with individual hard real-time constraint on each event.

The "Right-time" to react is determined by these time constraints, and determine the scheduling and optimization requirements - more later.

Tuesday, November 20, 2007

"The only motivation to use EP COTS is to cope with high performance requirements" - true or false ?



Somehow, I find myself using my collection of analysts' presentations to help me make some points, this time, I am showing a slide from Roy Schulte's presentation in the Gartner EPS summit - I'll return to the content of this slide shortly, but will start with some discussion I had yesterday about the reasons that enterprises are using COTS for event processing. I have heard (and not in the first time) the assertion - the only reason that one will want to use EP software and not hard-coded solution is the ability to cope with high throughput / low latency - in short, to deal with high performance requirements. If there are no high performance requirements there are other solutions, e.g. the database guys think that in this case one can insert all events to a database and use simple SQL queries for CEP patterns, or just using the good old C/Java programming for this purpose. This is somewhat inconsistent with my own experience, where customers that did not have "high performance" requirements were eager to use CEP technologies. Indeed, high performance is a reason to use CEP COTS, however, as indicated in Roy Schulte's slide above - it is actually a minor reason. According to Gartner, the high end is somewhere between 5-10 percent of the candidate applications, while looking at the prediction for 2012 - the use in the high end will be 8% - out of the 27% total use; note also that Roy Schulte defines the high end as 250 events per second, which is really far from the definition of "high performance", so the numbers are even lower. It seems that the market for "non high performance CEP" is much larger, and will grow faster. If that's so - where does the misconception that EP equals high performance always ? I think there are two sources - the first, the early adopters were from the capital markets industry, where some (not all !) of the candidate applications has indeed high performance characteristics. However, with the growth of the market, and use of EP software in other applications and other industries, these type of applications, while continue to grow, will not match the higher growth of non high performance applications. The other reason is that some vendors make the high performance as their main message, and trying to get the market believe that this is indeed the most important property.
So - if high performance is not the only reason to use EP COTS what are the other reasons to use EP COTS ? this is a matter for investigation, but IMHO the main one is the "high level programming" and agility - in short - the ability to reduce the Total Cost of Ownership.
I'll provide more insights about the TCO issue in a future post.

Sunday, November 18, 2007

On CoDA - Context-Driven Architecture


Yefim Natis from Gartner, the same analyst who is bringing us the XTP vision
is also responsible for the brand new vision CoDA = Context-Driven Architecture; which is one or two steps further from the XTP era. In the previous posts about contexts: (1) ; (2)
I have argued about the use of context as a first-class-citizen in event processing; and indeed people act and think within contexts, and the use of explicit context can get the computing closer to the human thinking, and also has computational benefits (like indexed vs. sequential search in the early days of file systems). Both Yefim and myself did not provide formal definition of context when talking about it, and I still have a "to do" to think about it. One comment though about the "Architecture" - some people think that SOA and EDA are contradicting terms, since there can be only one "big A" - meaning architecture. As I have already mentioned - in the posting about EDA and SOA that they represent different aspects of computing, thus they are orthogonal (but can co-exist). The question is - whether context is yet another dimension that can co-exist with the other aspects ? again - still need to think about it. Thus, I'll return to the context concept soon.