On Real Time

Some thoughts about the term "Real-time"  in my weekly column in the Real-time Insights.

On "CEP and Big Data 2" - comments on Philip Howard's observations.

Philip Howard from Bloor Research has posted some observations on his Blog entitled "CEP and Big Data 2".   Here are some comments (actually nothing new - just summarizing things I have written about before).
Philip deals with three issues:

• whether the name CEP is appropriate or should be changed? 
• who should be credited as the pioneer of this area?   
• whether CEP implies real-time processing?  
•  who are the CEP big data platforms?

Here are summary of my views on each of this topics.

The name "Complex Event Processing"

Exactly four years ago I posted on this Blog an explanation about - "why I prefer to use the name event processing without any prefix, infix or suffix".   My particular dislike of the term "complex event processing" stems from the ambiguity in the name - some people (including David Luckham who coined this term) view it as processing of complex events, some interpret it as complex processing of events, and then debate of when something is complex enough, and what type of complexity is needed  to qualify as CEP.  Moreover some of the vendors use this term for products that are neither of the two options.   I think that two words is enough for the name of a discipline, examples: information retrieval, machine learning, image processing and much more....  Thus, from my point of view the term "event processing" subsumes all other terms like complex event processing, business event processing, event stream processing and more.

Who gets the pioneering credit

Philip as a good UK patriot wonders why the Wikipedia value about Wikipedia and other sources gives credit to David Luckham and forget the Apama work that came from Cambridge UK.    Looking at Wikipedia, it has one mention of David, as well as other references (like our EPIA book). It indeed does not mention Apama or any paper by John Bates, but being a Wikipedia, anybody can suggest additions.   

David Luckham had major influence on this area, since he was the first one who published a full book and exposed the young area to the general public.    An article in IEEE Computer, published in 2009,  made some investigation of the history of that area and determined that in the 1990-ies there were four parallel projects that can be classified as starting points in this area:  David Luckham's project in Stanford,  John Bates' project in Cambridge (UK, not Boston), Mani Chandy in Cal Tech,  and our Amit project in IBM Haifa Research Lab.    I share Philip's view that John Bates should have full credit as one of the pioneers, and still view David Luckham as the "elder statesman" of the community.

Is CEP necessarily associated with real-time?

I have written several times about this topic, last time in response to Chris Carlson, to whom Philip also responds.   There is some abuse of the term real-time in the industry, while its meaning is "within time constraints", many people interpret it as "with very low latency".   This is not the same,  anyway, event processing is a functionality with applications that require very low latency, applications which require to react within real-time constraints (which can be: 2 hours), some require both, and some require none.

Who are the CEP big data platforms?

I have taken upon myself the limitation not to state opinions on commercial products within this Blog  - leaving  it to analysts.   Thus will make one comment.  There is distinction between two types of software entities - 

which is sometimes confused in the language used by people.

• Event Processing Platform is a software that enables the creation of event processing network, handle the routing of events among agents, management, and other common infrastructure issues.
• Event Processing Engine is a software that enables the creation of the actual function - in the EPN term implementing agents.

This is similar to the difference between an application server and a single component (programming in the small vs. programming in the large).    Some of the available platforms for "event processing for big data" provide the first one -- it gives infrastructure, but not implementing any type of functionality, but enabling developers to create their own functionality, thus they don't do full-fledged event processing.   Seems that many people classify both under the same classification  (of course there are products that do both). 

On event processing for real-time and non real-time applications

Chris Carlson has written in the Informatica Blog about the fact that there is a growing segment of applications that are using event processing and are not really real-time, and stating the fact that their share in the event processing market is growing.   I have written four years ago in this Blog about the term real-time and the abuse in its daily usage.  When some people (and marketing messages of vendors) are talking about real-time, they mean "very fast", while real-time really means "within time constraints", the time constraint can be micro-second, second, 5 minutes, or 2 hours.    Indeed, the early adopters in event processing, trading applications in capital markets, are based on low latency and fast reaction.   Many other types of applications use event processing for the functionality of filtering-transformation-pattern matching (or continuous queries in the stream oriented programming style), and the non-functional aspects are secondary.   The area which I am working on these days , proactive computing, has some applications in which there are real-time constraints, but typically not in magnitude of micro-second, but in seconds to minutes.  This is the case where there is a forecast for a future problem (e.g. a traffic jam will occur in 5 minutes), there is a time constraint on activating an action (e.g. within 30 second there is a need to change the traffic lights policies to mitigate the traffic jam).  This is a real-time application, but it has to react within 30 second, to impact in 5 minutes.    The interesting thing is that low latency applications may be "best effort" and not have real-time constraints.  Thus - there are low latency applications, real-time applications, those who have both, and those who have none.  Interestingly, event processing applications can be found in all four groups.  

On San Francisco and Storm

Today I had a single day to play a tourist in San Francisco.  Arrived yesterday afternoon, and is planned to stay until Thursday I am attending AAAI 2011, giving a tutorial there tomorrow, and plan some side trips and meetings, meanwhile spent all day out.  Since I have been in SFO many times before (although last time was in 2007!),  I decided to pick up a place that I have never visited before, looking at the brochure that I found at the hotel I chose the Walt Disney Museum, that resides in the Presidio which was a good opportunity to take a look at the Presidio (never been here before).  The Disney museum was interesting and surprisingly I spent there more than four hours (including the Disney old film of the day - the "Absent Minded Professor" in black and white).    Any visit in SFO is not perfect without walking near the bay, and eating sushi in Pier 39 looking at the waves.  San Francisco is one of my favorite cities, although driving here require patience, as the center city is totally congested.     And what about the storm in the title of this posting?  well - the weather here is nice, no storms!,  catching up on news I found a news item entitled "Twitter to open source complex event processing engine",  which of course attracted my interest, but drilling down to the details -- the title is somewhat over-statement.   Twitter has acquired BackType which created Storm, which is a kind of real-time Hadoop, on which one can implement event processing engine, but Storm itself is a platform and does not seem to have event processing language implemented, similar to Yahoo's S4.  It should be noted that there are some similar works that have implemented at least part of the event processing functionality over a kind of real-time Hadoop.  One of them is Colin Clark's DarkStar, the other is HStreaming. 
Nevertheless, it is interesting to see the Twitter is getting there.

 

Event processing = big data + real time ?

BIG DATA is a hot phrase, one can also be labelled as "BIG DATA NERD" by purchasing the shirt shown above.   The term BIG DATA refers to the explosion of data in the universe, especially methods to store and process huge amount of data.   


Paul Vincent poses the question:  CEP = real-time bid-data?     Actually he doesn't answer this question on his blog posting, but the spirit of the postings makes the impression that the "=" relations is evaluated to "True".    


The answer is that it is not really an equality, but more like an intersecting concepts.   


Event processing of course does neither necessarily works on big data nor it necessarily has to satisfy real-time constraints.


On the other hand,  real-time big data, is not necessarily event-driven and process events at all, as big data can be fairly static.    As an example for real-time big data, I'll return to an invited talk in DEBS 2011 on Watson, in which there has been a live demonstration of  the Jeopardy! game with the Watson computerized system, you can see in the picture that Paul Vincent is playing (very skilfully) the host.  

Watson is a "big data" crunching application, it has very strict real-time constraints, so it certainly can qualify as "real-time big-data", alas, it has nothing to do with event processing, in fact the data exists prior to the game  and no additional data is added during the game.


While the equality does not work,  there is an obvious relation;   in some cases there are substantial number of data inputs, which much of it can be reduced, or filtered out.   Event processing can be used to process streaming data on the fly, and not follow the paradigm of store now process later.   This is important especially one the processing is required in real-time.


More on event processing and big data -- later

Revisiting "Right Time"

This illustration (as indicated in the bottom) is taken from the "enterprise irregulars" site, posted by Ray Wang.
Ray Wang cites a relatively old posting of mine, talking about real-time, right-time and other time related concepts.   I admit that sometimes I abuse the term real-time (like other people do, but this is not a good excuse!), but I have not adopted the term "right-time".   In that posting I bring some classical definitions of various types of real-time.  Wang is making a somewhat different classification as a matrix with two axes:  the reactive/proactive axis, and the business value axis (low/high).  The high value proactive is called "anticipation", and the low value of proactive is called "nice things to do".   My interpretation is that both deal with notifications that may allow proactive behavior, but not necessarily automated proactive behavior of the type that we talk about (see my keynote talk last year in the OMG conference),  on the reactive front, the high value are mission critical reactions. and the low business value are called "timeless responses".  Here, I am not sure it is the best title, as there are reactions that have low value, but are time dependent, since they lose their relevance in time.  Example here is that getting an alert on available discounts in a nearby store may not be that important for me, but the discount is applicable only within the next hour, so if I would like to respond, there is time bound on this response.   Anyway - interesting classification. 

On optimization criteria for EP applications

This picture shows optimization of sitting on chairs, I actually know a person who sits on a big ball when he works, claiming it is good to his back. I have read with interest Paul Vincent's report on the OMG Real-Time workshop (since I cannot be everywhere, it is good that other people are reporting on what's happening, and Paul is especially good on reporting on conferences), in this meeting there has been a discussion about metrics for metrics for how to measure event processing applications. We don't have a standard benchmark yet, and I don't believe in a single benchmark fits all - but on a collection of benchmarks based on classification of applications. I would like to go deeper into the issue of "runtime performance" mentions there -- interestingly "runtime performance" means different things to different people, and indeed different application have different requirements -- if we just look at the metrics of -- latency and throughput, then we have the following variations of goal functions (this is probably not a complete list):

• min (average e2e latency)
• min (max e2e latency)
• min (variance e2e latency)
• min (deviation from time constraints)
• max (input throughput)
• max (output throughput)

The metrics are not identical - in latency there is a difference if the metrics is to minimize average latency or minimize maximal latency. For example, in Java the maximal latency can suffer from garbage collection that will make it untypically high, while "real-time Java" implementations that smooth the garbage collection minimize the maximum latency, but the price is that the average latency may grow. Throughput can be measured by input or output events, which are not really identical. Each of these goal functions indicates different kind of optimization, and this is just by looking at two parameters of throughput and latency...

This poses two interesting questions: will there be partition of the market according to optimization capabilities, or will be able to generate adaptive software that will be able to be tuned to multiple optimization ? more about performance metrics - later.

On Event Processing and Now.

After some more technical postings, back to macro-level issue. David Luckham has written recently in his website about the history of CEP (I'll refer to the content of this article in another posting). All indications are the event processing is not a new thing, however, as some people indicate, there are a lot of interest, events, maybe hype around it NOW - the question is what is new ?



The first observation is that unlike the past, today there are commercial "on the shelf" products whose main purpose is to provide event processing capabilities, while in the past there were event processing capabilities in other types of products (simulation, databases, network and system management, middleware, real-time systems etc..), there is also a start of event processing as a discipline. What are the reasons for the interest now - some of what happened in the last few years that supported the shift from hard-coded event processing functionality to COTS are (based on discussions with people in multiple industries):

• Some contemporary applications which are event-driven by nature - such as: compliance with regulations, the need to detect frauds as two examples - have become pervasive in multiple industries
• The increasing complexity of inter-process integration that is simplified with event-driven interaction
• The need for flexibility and agility to gain market advantage in different areas - thus, move away from hard-coded solutions.
• The substantial growth in the number of available events - e.g. since RFID technology became pervasive.
• Some industry trends like - BAM, RTE, "on demand" - that are also based on responding to events.
• The drive to save expenses in back offices by automating exception handling - trends like STP.

I am sure that there are more of these.

Thus, while event processing functionality is not new - "event processing" as a first class citizen in the computing world - with its own dedicated products, community and emerging discipline is new. More - Later

Unplanned events - again

This (more or less) how my MPV car looked like at the beginning of this day, unfortuantely, it does not look like this now -- while driving in a highway, there was some traffic congestion, and the traffic slowed down, however, the driver behind me has not detected this event and proceeded to drive full speed - as a result he crashed his small car into my car, totally destroyed his car, but made also substantial damage to the back of my car. So - I had an absent event, did not make the meeting I was driving to, had to wait for police, and then almost two hours for a replacement car from the leasing company -- so wasted much of the day. Luckily for me, I have detected that with the momentum of the crash, I am approaching very quickly a track before me -- and succeeded to stop the car before getting into the track -- otherwise, I may not be sitting at home and typing now -- so people also need to process events in real-time and not in batch...

Event Processing for Business Intelligence

From the various descriptions of Business Intelligence found on the Web, I have chosen the one by Hakki Erbug as a starting point.



As noted in several previous postings, IMHO, event processing is a set of technologies that have multiple usages, and are not really strongly associated with a single type of application. Business Intelligence is similar in the fact that it combines various technologies, but different in the fact that it is focused around area of using data for decision making in various ways. In the event processing case, decision making is one of the possible usages, but not the only. This posting will briefly survey how event processing can enrich business intelligence, and in fact, in the Gartner EPS summit I have seen several BI vendors that are looking at EP as a natural capability to complement their products.



Going on Erburg's illustration clockwise:



1. "Active Data Warehouse" - While traditional warehouses are being updated in batch, the notion of active data warehouse makes the warehouse update itself an event-driven action. The rationale is: when a certain event (raw or derived) occurs, a decision has to be taken, however, the decision relies on a data warehouse, thus, an update of the data warehouse should occur before making this decision. The update can be of the same event that happens or of some collection of data that has still not been updated in the data warehouse and is needed for the decision making. There can be some time constraints associated with the decision (and in turn with the warehouse updates). The time constraints are not necessarily micro-seconds, the constraints can be minutes or hours, but they are typically well-defined.

2. ETL and mediated event processing: ETL has some functional similarity with mediated event processing, it is also about transformation. We see mediated event processing and ETL getting closer to one another, where difference may be in quality of service. In the future there may be a case that ETL will become a specific case of mediated event processing (of course ETL folks may say the same from the opposite direction).

3. Real-time analytics: While analytics (simulation, optimization, mining...) has been used for a while in the decision making part of BI, in the event-driven world, the reaction to an event in some cases is temporally-bound, which means that there is a real-time constraint, or upper limit on the requested reaction time. This provides new way of thinking about analytics - while without time constraints an optimization should strive to get the "best result" (or if heuristics satisfy some approximation condition), in real-time analytics the optimization strives to get "the best result that can be obtained in T time-units as specified (e.g. 18 seconds)". How does event processing play in real-time analytics? it may play in a simulation mode - scenarios are created and simulated events are emitted - they in turn may create simulated derived events which determine the situations of this simulation. This is, of course, in addition to the fact that in an event-driven universe, the entire BI cycle is event-driven and relates to the event and its context.

To conclude -- event processing is a natural step in the BI capabilities, and thus I expect BI suites to support the event-driven flavor... This - again, does not say that BI is the ONLY use of event processing. I still need also to refer to the issue - "has BAM failed because it was not based on BI techniques" as claimed in the article that triggered my discussion on the BI topic - stay tuned.

On Real-time, Right-time, latency, throughput and other time-oriented measurements

The illustration shows different types of "real time" types. This posting was inspired by a comment on a previous posting - trying to do some order in several notions of time. First, the term real-time is frequently used in conjunction with event processing. The popular belief is that real-time = very fast, but this is not really what real-time is about. real-time can be thought of a deadline accompanied by a utility function that designates the damage from missing the deadline. In this illustration there are four type of real-time:

(a). Soft Real-Time: there is a sense to react after the deadline, but the utility decreases (maybe fast) and at some point gets to zero - no use to do it at that point, but no damage.
(b). Firm Real-Time: The utility go immediately to zero when the deadline is missed - no use to do it after the deadline, but no damage.
(c). Hard Essential: Missing the deadline - the utility function goes to a constant negative value; there is a constant penalty.
(d). Hard Critical: Missing the deadline - the utility function goes immediately to "minus infinity", means: a catastrophe will happen.

One can, of course, define the real-time utility function differently, and create more variations.

So - Real-time is not about fast, but about missing the dead-line. The linkage is there, if the dead-line is very short (need to react within 1/1000 of a sec), but many dead-lines are longer than that -- seconds, minutes, hours or days - depends on what is needed to react to - e.g. the contract that we have with our local electricity company says that when a problem is reported, they should start fixing it within 2 hours; the deadline for delivery of Pizza (otherwise it is free of charge) in one of our local delivery centers is 40 minutes - so most the world typically does not work in milliseconds.

When talking about "quality of service" measurements they can be either statistical: in 90 percent of the cases, the deadline should be reached, or individual: for all cases the deadline should be reached. Typically there are different scheduling strategies to achieve each of them - for the individual case it is important to have a consistent deterministic reaction, and this is a source of the various implementations of Real-Time Java - since Java is non deterministic by its nature of garbage collection that happens in undetermined times and lasts for undetermined duration, which may work for the statistical case, but not for the individual case. The Real-Time Java does not stand for Java which runs much faster, but for Java in which the garbage collection differences are smoothed, thus its reaction is (more) deterministic.

What other measurements are there ? in event processing - latency can be measured by the end-to-end (from the time that the event happens in reality to the action being taken in reality), can be related only to the event processing part (from the producer sends the event, until the time that the consumer(s) receive the consequences of this event), and can relate to a specific function (agent) in the event processing network, so when latency is mentioned - it should be defined - what is really being measured. The deadline typically refers to time constraint of the latency.

The term throughput designates the amount of events that the system can get as an input in a given time interval (1 second, 1 minute etc...), throughput does not necessarily entails that all these events are handled within this time interval - it may be the case that the events are put into a buffer, and all of them are handled within the next 6 hours - so in this case, the throughput is mainly determine by the capacity of input channels and buffers and not in the speed of processing. Of course, there are applications that require high throughput together with individual hard real-time constraint on each event.

The "Right-time" to react is determined by these time constraints, and determine the scheduling and optimization requirements - more later.