On the straw and the waterfall

Tarrytown, NY. I have left Nashville this morning and arrived to Westchester county, NY, to do some IBM meetings on the way back from DEBS. I think that the nicest analogy I got from the conference came from Alex Buchmann keynote address, he used some other pictures, but provided the analogy that while regular programming is like drinking with a straw, this is good when the data is standing, while the data is moving, like in event processing, using the same kind of thinking is similar to use a straw to drink from a waterfall. I like this example. It requires some education to understand that event driven is a different paradigm of thinking, and this impact the way we should think about the programming model as well as the execution model of these types of applications. This also links to the discussion on education that will create a critical mass of developers who understand this paradigm. It is somewhat different, but it is not rocket science, with some education, I believe that it is doable.

I'll end with some travel anecdotes: Today I rented a car in the airport, got to the car and did not find a car key, just a remote control; it took me two minutes to understand that the remote control serves also as a car key, there is a hole in which it has to be inserted to, and when it is there, there is a on/off button which turns the engine on or off. I showed it to one of the people I've talked with in the place I visited and he told me that he has even more sophisticated electronic key that he does not need to insert it anywhere, it is enough that it is in his wallet that the car identifies it and enables to turn on the engine.

Another anecdote that always puzzles me is hotels' superstitions regarding the number 13. My Nashville hotel was semi superstitious, it did not have 13th floor, jumped from 12 to 14, however I have stayed in room 413. Now I am in a different hotel, this time in room 412. For curiosity I have looked for room 413, and did not find it, it seems that the level of superstition here is higher... I never understood this superstition, but this is not the only thing that I don't understand... Go figure.

Context and Situation - are they synonyms?

I have spent the last couple of days in Eilat a resort town near the Red Sea, the Las Vegas of Israel, if one counts hotels, and an interesting combination of dessert and beaches in a small proximity, in the picture you can see the hotel in which we stayed. Anyway - back to context. A comment that received to the previous posting made me realize that there is a need to explain what is the difference between - "context" and "situation", both of them are semantic terms. In a way - it is similar (but not identical) to the difference between "state" and "transition". "situation" is something that happens, and that has some meaning in the consumer's terms, and it is the event consumed by the consumer(and as a result may trigger an action), in syntactic terms, situation is an event, and may be either raw or derived event (I'll not get into complex / composite events - it is sufficient to say it can be any kind of event). Context is a state -- a context can be created by the occurrence of event and destroyed by the occurrence of another event. Let's take a simple context: "during red alert". There is an event that declares the "red alert" state, and then there is an event that declares any other color, and it is considered to be "converse event" to the starting event, thus, it end the context. While the declaration of "red alert" is an event and can also be a situation, the fact that another event happened "during red alert" is not an event (semantically). Context may also be spatial - for fleet management application - a context may be a geographical area - and the events are that a vehicle enters and exits a context, but the fact that a "vehicle is within the area" is not an event.

Thus "context" has a distinct semantic existence. As far as implementation goes - there are various relations between events and contexts. After writing my first blog about context - somebody attracted my attentions that analysts have been recently talking about "context-aware delivery architecture" - I'll refer to this four letter acronym (are we out of all combinations of three letter acronyms?) in one of my next blog.

XTP, Distributed Event Processing, and Distruptive Technologies

Hello everybody. Another experience of novice blogger - I had to start moderating comments, since I got spam attack of comments - so comments will not be seen in real-time anymore - but keep sending them anyway :-)

I have heard couple of times in the last few months talks from Yefim Natis
from Gartner who promotes now the concept of XTP - "Extreme Transaction Processing"
Yefim claims that XTP is a disruptive technology, in the way that it will be a new wave in enterprise application servers, he mentions in particular two disruptive technologies: dynamic grid architectures and event-based programming models. My friend Alan Lundberg asks in a recent blog what is the difference between CEP and XTP -- the short answer is that XTP is a framework that should have CEP capabilities but some other capabilities such as: transaction management, highly distributed environment etc... It is reasonable to think that some CEP applications will run on XTP, and maybe for some it will be overkill, but this depends on the ability of XTP to scale down as well as scale up. Another friend, Tim Bass (the person who talked me into writing this blog), has also recent blog about distributed event processing in which he said that "CEP engines should operate as cooperating intelligent agents" - and this assertion is the essence of the "event-based programming model" for XTP. I'll drill down into it in later blogs, but just point out some of the principles:

• In the event-based programming model - events and their flow are the most basic primitive
• Events are flowing among agents - agent may be intelligent agents that are doing "smart" things, or just software agents doing predetermined computations.
• The wisdom of the system is both in the agents and in the network - the network itself is intelligent and getting decisions.

This is another view of the world than the current programming models that are based around objects (and recently services), and may act as a disruptive programming model towards them, since having event as a first class citizen changes things.... These are preliminary ideas, but I find this idea intriguing and fascinating, so I'll probably dig into it deeper.

More - Later.

Agnon, the dog, playing and downplaying

My literature teacher in the 7th grade has told the class one day a story about her days in as a university student; one of the literature classes in the university has been dedicated to Agnon, the only Hebrew writer who received Nobel Prize in Literature (Agnon's picture is in the left hand side). One of the famous stories of Agnon has a dog (such as the one in the right-hand side) as its hero; thus, the class of students at the university had discussed this story and had a debate about what does the dog represents, one group of students thought that dog is a symbol for Agnon itself, and one thought that the dog is a symbol for the entire Jewish people. Since Agnon was still alive those days, they have decided to send a couple of students to meet Agnon, set up a meeting in his house (well - Email questions were not on the radar at that time), and explained the dilemma - what does the dog represent? Agnon was somewhat amused to hear about the debate, and told them briefly - "I don't know what you are talking about, the dog represents a dog".

Going back for the childhood Reminiscences to the contemporary issues - I have read with interest the recent blog of Paul Vincent about "CEP vs. Business Rules" in which he mentioned me by reference (and not by name) saying: "Others seem to downplay the idea of rules for CEP". I have a substantial respect for Paul, and generally agree with most of the assertions he made in that blog. Let me add some more insights into this issue - I am using this blog to reflect about event processing, and make the point the "event processing" is a discipline that stands on its own feet, and is not a footnote to anything else - be it databases or business rules.

CEP functionality can indeed be implemented in either SQL or in an extension to business rule language as some of the vendors successfully demonstrated. It can also be implemented in half a dozen different ways (at least). Since I am trying to investigate "event processing" as a discipline, and not a specific implementation, thus is the play, and indeed I downplay specific implementations for now, and try to think on the fundamental issues. I may return in future blogs to talk about what type of implementations fit different types of functions/applications - which is a difficult question, and I don't really see that there is an issue of "CEP vs. Business rules" as these are conceptually orthogonal issues (and I agree that in practice there are various relations), and Paul's blog does a tremendous job of highlighting the differences and relations between the two.

For now - my dog is "event processing" and it represents "event processing" and not other concepts that are not fundamentally necessary to understand what "event processing" is.

More - Later.

on the various temporal dimensions of an event

This is the Winchester cathedral in England, and I am finding myself in additional business trip, but this time to the country-side of England, one hour drive from London, and seems like several centuries distance in time. Talking about time - I'll dedicate this blog to temporal issues in event processing. Well, this is a big topic, so I'll start with the topic of the temporal dimensions of an event. The first issue is - does event happen over a time-point or over a time-interval. A couple of years ago I had a discussion about it with David Luckham

who claims that "nothing happens over a time point", which means - everything happens over an interval. This may be true, but for practical reasons, there are cases in which we want to look at an interval as a time-point, since we are not really interested to decompose the interval further, the abstraction of chronon as atom of time is very useful - saying that we can look at the "smallest significant unit of time" for a certain application, examples: minute for delivery of packages, 1/100 second for athletics, day for personnel applications, month for salaries etc...

Using this we can say that an event occurs over a time-interval, but for some (maybe most) applications it is sufficient to approximate the interval to a time point in the right granularity (the application's chronon). However, this is talking about the time that the event happened in reality - indeed there maybe several temporal dimensions associated with a single events:

• The "event occurrence time" - the time that the event happened in the reality
• The "event detection time" - the time that the event was emitted / signalled / detected by the event processing system (typically a single time-point).
• The "event valid time" - the time-interval in which the event may effect the event processing system - typically starts at the time it starts to effecting the processing, and ends at the time it stops effecting the processing. This is less obvious then the other two, but important for some types of processing.

In some cases the order of events do not matter, however, when it matters - e.g. a pattern is depends on the fact that event e2 occurs after event e1 (such as: the order of the events "printer is offline" and "printer is online" of course matters, since it indicates whether the problem has been resolved or is still there. The order in this case is of course according to the "event occurrence time" (note that if the event occurrence time is an interval, there may not be a total order, but interval relations).

This is only a start of the temporal saga in event processing, there are many more questions that I'll try to include in future blogs:

• What happens if events are getting out of order ? (short and long delays) ?
• What is the "occurrence time" and "detection time" of derived events ?
• What is retrospective event processing, and how it fits in ?
• What is temporal context and how it relates to time window ?

More on all of these - later.

Who should define event processing applications ?

This picture is taken from the recent ebizQ's event processing survey and shows an interesting picture, the survey done shows that the customers prefer that rules will be defined in 84% of the cases by non-programmers. This is not the current situation, but this is the way that the customers prefer it. Indeed, it is not possible to have the entire event processing end-to-end story done by non-technical people in this phase (maybe with applying some autonomic computing techniques we can come closer to it, but this is a topic for another Blog), but let's concentrate now in the CEP (Complex Event Processing) part - detection of patterns. Given the right level of abstraction both of the "programming in the small" - the individual pattern, and the "programming in the large" - the event flow, pattern composition may be delegated to business people (well - with some understanding of basic terms like the distinction between "OR" and "AND", but with no need to know how to program in Java, C, SQL and other languages geared for programmers). Bear in mind that the bigger expense in the application's life-cycle is not the cost of development, but the cost of maintenance, and this is especially true when changes are frequent, or there is a need to create new patterns to detect unexpected events within a short time. In fact, the agility, ability to quickly adapt, and ease-of-use for non-technical people is a success factor for the entire concept of event processing COTS, and the ground for competition among vendors. Investments in tools for developers address only 16% of the market, according to this survey. This reminds me that once in my academic past, I attended a teaching workshop given by an external experts, and they started their session saying that - most teachers base their teaching on the students' sense of hearing, while for most people the visual sense is much stronger, so there is a mismatch.... More later.

Route to markets for event processing technology

I am back today to more macro oriented blog (will try to flip-flop between macro and micro oriented issues, since it seems that each has its audience of interest). One of the interesting questions is -- in what form event processing technologies will materialize in the market ?

Looking at Bill Gassman's presentation in the Gartner EPS on BAM - he talks about the BAM market and partition it to:

• 65% - embedded inside vertical solutions
• 20% - embedded inside BPM (+ ESB) middleware/products
• 10% - embedded inside BI products
• 3% - embedded inside IT operations (e.g. BSM)
• 2% - general purpose BAM products

Gassman's interpretation of BAM is quite wide (I have somewhat narrower interpretation), and covers most of the EP types of applications, so let's take it as a starting point. Without commitment to the exact numbers, the order is consistent with my observation on this market. While the early adopters used stand-along engine, and built applications on top of it, this will become a relatively small route to market, and the segment that will grow most is that of EP technology embedded inside vertical solutions, we see signs of this is multiple industries now. The second largest segment is EP technology embedded inside middleware, we see that the big players in this area are taking this approach, the rational behind it is twofold - from the middleware point of view - EP capabilities is now becoming a must, due to competitive pressures, and from the ROI to customers POV - EP applications are typically not isolated, and the biggest investment is to connect them with the consumers/producers of events, thus application integration middlewares with adapters to multiple systems may assist. There will be always a market for a stand-alone event processing technologies, and this market can be segmented to "general purpose" engines, and optimized for special purpose ones..... I am not sure that what I am writing now reflects the current reality, but it certainly reflect the trend....

More on the role of general and specific frameworks - later.

More on EDA is EDA and SOA is SOA

Hello from Haifa again. My good friend Tim Bass has written a blog entitle EDA is EDA and SOA is SOA. I totally agree that EDA and SOA are not the same, more than that, I don't even think that they are talking about the same thing. Many people equate SOA with the synchronous "request-reply" style of interaction, but this is only one way to implement SOA, since SOA is not about interaction style, but about modularization or componentization of the enterprise, and hence it IT systems, introducing them as "services", the interaction style is a secondary issue, services can interact both in synchronous and asynchronous ways, and there is nothing inherent in the concept of services that say anything about the interaction type. EDA is indeed about asynchronous push interaction style, and this can be interaction between services, where we get both SOA and EDA, or interaction between other software artifacts, in enterprises that don't implement SOA. Implementation of SOA and EDA both make sense for other reasons and to solve other problems, and they are completely orthogonal. In enterprises that implement SOA there is a sense to use a combination of both, thus, it is important to position them in a way that they are complementary and not contradictory, since there are some misconceptions about it (exactly since some people interpret SOA = request / response). I may dedicate another blog to in-depth discussion about this two concepts. A side point - since this blog deals with "event processing", event processing is not implemented on top of EDA -- there are some cases in which event processing has "request-response" functions such as: getting events in PULL from the producer, which may be done periodically or on-demand, or event processing network that is implemented as part of transaction, and thus does not fulfill the loosely coupled principle of SOA. Thus "event processing" is not a pure EDA.... confused ?
more - later.

In-Line vs. Observations

I am back in business after spilling water on my laptop (total-loss)... Fortunately, the hard-disk was not damaged, and I after getting the excellent IBM support to work in holiday, I have (another) laptop, not recommended to spill - water, coffee or anything else on computers...

Anyway, last post I have discussed RTE vs. BAM, and succeeded to confuse some people, so it needs some clarification. First - as said before, event processing is a collection of functions, that derive, detect pattern, enrich, transform, and some other functions that take events as input, processes the events, and produces (possibly other) events as output.

The distinction is - where, relative to running transactions, the event processing is done. In-Line event processing means that the event processing is done inside a transaction, not observing the transaction from the outside. In this case, the event creation may be part of the transaction (e.g. an application emits event, but the result actions belong to the same transaction), or a starting-point for a transaction. To give a concrete example (repeating the same example from the previous post) - some application emits an event saying "cancel order" (this by itself may or may not be a result of event processing function), this should be atomic with the execution of the cancellation, if cancellation is not possible, then some compensation is needed in the producer application. Likewise, an event can start a transaction that spans all its descendants and actions, if an action in the consumer cannot execute, or there is some validation issue, the entire thread of the "event processing network" should abort/compensate, this is in contrast to "observation mode" -- observe if there is some anomaly, and if there is - notify somebody. In the In-Line case, the transaction can change its course due to event processing, in the observation case - the impact is indirect. Of course, in some cases we have combinations of all. There are two issues that should be discussed: what is the relationship between "observation" mode of event processing to what the market knows as BAM ? another question --- what is the semantic meaning of rollback or compensation when talking about events? more discussion on each of these issues - later. Happy New Year.

Event Processing - A paradigm shift ?

In the Dagstuhl seminar we held earlier this year Roy Schulte has raised the question: "Will Event Processing (EDA) become a paradigm shift in the next few years or not?”. If my memory does not mislead me we have discussed it in one of the evenings in the wine cellar, that's the reason that I don't remember much from that discussion. Anyway, thinking back about this issue and looking at a recent article claiming that "complex event processing is still on the launch pad"
we can realize that the paradigm shift has not happened yet (maybe it has happened for us which live and breeze this topic, but not in a large scale anyway). Looking at one paradigm shift that has succeeded - relational databases - we can analyze some of the reasons for that success :

• We need an underlying theory in two areas one - semantic (like relational algebra) and the second engineering-oriented theory (like query optimization) that is built on top of the first one;
• We need vendors that understand the theory to generate products that implement it
• We need to be able to explain the developers community (with the various types of developers - a topic for another article) how to use it --- a good textbook like Date's book about relational database is a good step, but development of use patterns, methodologies etc.. will be helpful too...
• Standards (topic for another forthcoming article) are complementary --- but relational databases have been paradigm shift before the SQL standard was published.

The state of the practice in event processing is similar to the database area in the pre-relational time, there is several approaches, all of them grown up from implementations. This does not at all undermine the importance of the first generation of event processing products, without more experience in the field, we cannot get anywhere...

While the various vendors continue to incrementally advance the products, Some of the effort (perhaps a community effort) should go now towards bridging the gap between the first generation and the "paradigm shift"...

Bottom line: Event processing has a big potential to make the paradigm shift that Roy Shculte is talking about, making event processing a major paradigm in enterprise computing. It can happen, it should happen, and we should make it happen - but there are mountains to climb and oceans to cross.

More on the challenges and obstacles as well as the futuristic vision - later