Latency might be subjective

Last week I have been in a short visit to England, served as external examiner in a PhD exam (called by the locals: "Viva"). The exam was strange in the sense that it was the first PhD exam I have ever attended where the advisor was not present (actually he was not invited, I guess it is a matter of culture).   The dissertation that was submitted by Jenny Li dealt with performance metrics and benchmark framework for event processing system.  One of the ideas raised in this dissertation is that the measured latency may be subjective in what it means.   It took me some time to understand the idea, so I'll explain it through example.  

Let's assume that the pattern that we are looking for is a sequence of four events, of types E1, E2, E3, E4.

The two common metrics associated with latency are:

Measurement starts at E4, since the consumer expects to see results only when the last event that closes the loop occurs.

Measurement start at any event occurrence, and ends when the system finishes processing the individual event, which can be merely storing it in the buffer.

Note that the first metric is biased towards eager evaluation, doing the minimal work at the end, and preparing sub-patterns, while the second metric is more balanced. 

The proposed metric is -- start measuring in the event most significant to the consumer, and end at the end of the processing of the pattern.   Let's say that the most significant event to the consumer is E2, then the latency starts at the occurrence of E2, and ends either when there is a match of a pattern, or when the event E2 can be discarded since there is no match.   This can be applicable in cases that all events in the sequence typically happen (e.g. when they are time series events) in relatively fixed differences.   This is an interesting metric, we asked the student to define definite criteria for when it is an applicable metric and when it is not.   

On latency in event processing network

Packing, on my way to a family vacation in Anatalya, Turkey.

Following an interesting discussion yesterday about performance metrics -- it turns out that definition of latency in event processing network is quite tricky. The reason is that an event can move in multiple paths over the network, in some it is filterred out, in some it just getting into internal state of agent since it does not complete a pattern, and in some it may complete a pattern and trigger reaction, thus there are various ways to define the metrics here. This is important since optimization has to take into account the goal function - what is being optimized. More discussion on this area -- after I'll return. I'll be out of touch for a week.

On optimization criteria for EP applications

This picture shows optimization of sitting on chairs, I actually know a person who sits on a big ball when he works, claiming it is good to his back. I have read with interest Paul Vincent's report on the OMG Real-Time workshop (since I cannot be everywhere, it is good that other people are reporting on what's happening, and Paul is especially good on reporting on conferences), in this meeting there has been a discussion about metrics for metrics for how to measure event processing applications. We don't have a standard benchmark yet, and I don't believe in a single benchmark fits all - but on a collection of benchmarks based on classification of applications. I would like to go deeper into the issue of "runtime performance" mentions there -- interestingly "runtime performance" means different things to different people, and indeed different application have different requirements -- if we just look at the metrics of -- latency and throughput, then we have the following variations of goal functions (this is probably not a complete list):

• min (average e2e latency)
• min (max e2e latency)
• min (variance e2e latency)
• min (deviation from time constraints)
• max (input throughput)
• max (output throughput)

The metrics are not identical - in latency there is a difference if the metrics is to minimize average latency or minimize maximal latency. For example, in Java the maximal latency can suffer from garbage collection that will make it untypically high, while "real-time Java" implementations that smooth the garbage collection minimize the maximum latency, but the price is that the average latency may grow. Throughput can be measured by input or output events, which are not really identical. Each of these goal functions indicates different kind of optimization, and this is just by looking at two parameters of throughput and latency...

This poses two interesting questions: will there be partition of the market according to optimization capabilities, or will be able to generate adaptive software that will be able to be tuned to multiple optimization ? more about performance metrics - later.