On terminology again

While the EPTS glossary team is about to produce additional version of the glossary,  I received by Email a question about terminology from a student investigating this area, whether "event driven architecture" and "event processing"  (both titles of books) are indeed the same thing  


The answer is -- not really, but the difference is simple:   


Event Driven Architecture is an architecture in which there are event producers, event processors, and event consumers, with the following principles:  1). All players are decoupled;  2). All the communication between players is that they send events to one another:  3). All communication between players is asynchronous. 


Event Processing is a type of processing that takes as input one or more events and produce as output one or more event applying functions such as: filtering, aggregation, transformation and pattern matching.   


The concepts are related in the sense that it is common to implement event processing on top of event driven architecture, but it is not tightly coupled:
Event Driven Architecture can implement  pub/sub systems that are doing routing only and do not do any processing of events;
Event Processing can be be implemented in synchronous mode and not on top of  EDA.

On the Event Processing book by Chandy and Schulte

Today I got a package of books from Amazon that included two new event processing related books. I'll review the first of them today. This is the book by Mani Chandy and Roy Schulte, called "event processing - designing IT systems for agile companies". The title itself (agile companies) indicates that the book is business related, and indeed it is primarily answers the questions: why use event processing, and how it is related to other concepts in enterprise architecture concepts. The book is non-technical and fits the level of managers/CIOs/ business analysts. The book starts with overview and business context of event processing, talks about business patterns of event processing (another type of patterns, besides all other types of event processing patterns), talks about costs and benefits of event-processing applications, and types of event processing applications. After doing the ROI part, it goes to more architectural discussion -- getting top-down approach: EDA, events, and employing the architecture. Next there are two chapters about positioning event processing against the rest of the universe: SOA, BPM, BAM, BI, rule engines (I'll write about this positioning attempts in later postings). Towards the end there is a chapter of advices how to handle event processing applications (and this chapter reads like analysts report). Last chapter talks about the future of event processing, again from business perspective, future applications, barriers and dangers (again a topic for which I should dedicate a complete discussion), and drivers for adoption.

In conclusion: good book to everybody who wants to know what event processing is and what is its business value. Things that I thought such a book might also include --- some reference to what currently exists in the industry, how the state-of-the-practice relates to these theoretical concepts presented in the book, when COTS event processing should be used vs. hard-coded, which are practical considerations of event processing applications

(maybe in the second edition?)

For those who asked me what is the relationships between the book Peter Niblett and myself are writing and this book, the answer is that our book has a totally different focus, explaining step-by-step, what is needed to build an event processing technology, providing the reader an opportunity to experience the various approaches in the state-of-the-practice by providing a free downloadable versions of various products and open source. The target population is also different - we aim for designers, architects, developers and CS students, while The book by Mani and Roy is aimed at managers, business analysts and MBA students. The review of the second related book - later.

On the Gartner 2009 application architecture hype cycle

Here is a revised version of my Blog entry that relates to the Gartner Application architecture hype cycle report (Gartner Report ID number G00168300 from July 16,2009) , the revision was done at the request of Gartner who asked that I'll make exact citations in their report, and make clear distinction between what is quoted from the Gartner report, and my own remarks.

Here are a collection of citations from the report that are of interest from the Event Processing perspective:

1. "Event-driven architecture (EDA) is an architectural style in which a component (or several components) in a software system executes in response to receiving one or more event notifications". In the report EDA is positioned under the hype cycle phase "Climbing the slope of enlightenment" which according to Gartner's terminology is defined as " Focused experimentation and solid hard work by an increasingly diverse range of organizations lead to a true understanding of the technology's applicability, risks and benefits. Commercial off-the-shelf methodologies and tools ease the development process"
2. CEP is positioned under the hype cycle phase of "Technology Trigger" which according to Gartner's terminology is defined as "A breakthrough, public demonstration, product launch or other event generates significant press and industry interest", and is the phase that precedes the "peak of inflated expectations" phase.
3. For CEP: "market penetration is 1% to 5% of target audience"
4. CEP use is expected to grow at approximately 25% per year from 2009 to 2014, but the use of COTS CEP products is expected to grow more than 40% per year in this time frame
5. For CEP COTS products: " Most of these products are immature and incomplete"
6. "Most business analysts do not know how to identify business situations that could be addressed through CEP, and that is limiting the rate at which CEP use can expand. Most software engineers are not familiar with CEP development"
7. "The Event Processing Technical Society (EPTS) was launched in June 2008, and it is expected to facilitate the adoption of CEP".
   

Here are my own comments:

• Note that EDA and CEP are positioned in different phases of the hype cycle.
• The fact that the market penetration is low indicates that there is still a substantial growth potential, if we can overcome the adoption challenges
• The adoption challenges consist of product maturity and market awareness. We are now still in the first generation of products in this area and maturity is typically achieved in later generation. Awareness and understanding of value and positioning are indeed a challenge.
• EPTS indeed has been formed to facilitate the adoption of the event processing area. Both challenges mentioned here – advancing the state of the art to accelerate the next generations, and educate the general community about the value and positioning of event processing within the enterprise computing.
  

On the head and the tail of EDA

Somehow the discussion around "EDA" and "CEP" continues in the Blogsphere - Giles Nilson from Apama has published seven points , out of which I quite agree to the first five. Jack van Hoof, who started this whole thread of discussion, argues that "CEP is not the beginning but finishing of EDA". So what is the answer? head or tail ? the right answer is --- it depends. Some customers work with methodic architecture way, in which first the EDA architecture is being set up, and then CEP tools are invoked on top of this architecture -- this is the case that Jack is talking about. However, in some cases, customers don't apply architectural thinking, but just acquiring some application that is implemented with CEP tool, and thus it introduces a kind of EDA ad-hoc for a specific application, this is the case that Jiles is talking about.

My guess is that most early adopters have applied CEP in ad-hoc way, so it serves as a "head" for more EDA in the future, while recently we see more customers looking at EDA first, since they are taking it from enterprise architecture perspective.

If we'll take the CEP functionality -- finding patterns on multiple events, it may not be implemented on EDA at all -- since the same technique may fit to "non event" environment.

More - Later.

On EDA, CEP and disruptive technology

This is the second time in the last few months that the term "disruptive technology" is being used, this time by Mark Palmer who adds his voice to the EDA vs. CEP discussion. Mark acknowledges the fact that EDA and CEP are not synonyms, but asserting that CEP is the disruptive part of EDA.

Recall that in the Oracle Blog the "disruptive" word was used recently and I have discussed it at that time; Richard Veryard has answered my response claiming that "disruptive" is often not better, at least not in the short run.

Anyway, I have noticed in the recent few weeks, two IBM customers who made plans to move to EDA (neither will use CEP, at least not in the short run). The shift to EDA is a fundamental change in the architecture thinking with the introduction of the decoupled event-driven thinking. Is it new ? - not really. Is it new for these customers - yes, not only new, but a significant change in thinking, it is an indication that there is a beef in EDA, even without introducing CEP, that the enterprise should digest. While thinking in events is natural in the daily life, it is still not natural for enterprise architecture and programming paradigm.

Bottom line -- while CEP certainly has its merits (if the customer has mature enough to digest it), EDA seems to be a more fundamental change in thinking relative to the alternative, and has its own beef. More - Later.

On EDA and maturity model

This is the first postings I am writing on my new Laptop, Lenovo T61p. New laptops are a trauma, since with the new laptop one gets new versions of all software, that somehow not always behave in a familiar way. I still did not succeed to convince the new laptop work with my home wireless network (it succeeds to connect, and then disconnects after a few seconds), so I have arranged workplace near the router and works wired, catching up in reading Blogs.

Jack Wan Hoof in his Blog claims that the market does not understand EDA, since people when talking about EDA are really talking about CEP, in a previous posting he makes a distinction between EDA that is about business events and CEP which processes messages.

It is true that people are often mix the terms of EDA and CEP (both TLAs) which are not quite the same, however, the "message" vs. "event" is something the needs some discussion, as well as the relations between all these, and why should a customer of these technology care anyway.

First of all - I agree that EDA is an architectural pattern, where various services/components/agents/artifacts are communicating in a decoupled way by sending events to some mediator which routes it based on subscription or other type of routing decision, such that the source and sink may not know each other, while CEP is a functionality of detecting patterns among multiple events and deriving new (derived/synthetic) events as a result.

To the question of "messages" vs. "events" -- CEP is called CEP since its inputs and outputs are events according to the definition of event as "something that happened" (with all its variations - see the EPTS glossary), message is in this case a way to represent event (not the only one - CEP can also process events represented as objects that are not messages).

The claim that messages in general may not represent events is valid, I can send my picture as a message and of course some smart guy can find event in it, but it is not intended to represent an event. However, if we apply pattern detection on messages that not represent events, then it is not CEP, since CEP assumes, by definition, that it inputs are events. This can be pattern detection on messages, and one can invent some name for it, if it makes sense.

Interestingly, people also realized that the same type of patterns may be useful for data and not only for events, which makes is detection of patterns on data and not CEP again, and we have started to see some works on patterns in the database community, although they are still quite complex.

EDA by itself has a value to the architectural thinking, recently I came across a plan made by an IBM big customer looking at their future architecture as a combination of SOA and EDA, in the sense that services will communicate among themselves in an event-driven way, but they have not yet discovered CEP at all. In fact, observation on various customers that are in different phase of thinking about the "event" issue leads me to think on a kind of maturity model in this area :

• phase I -- simple event processing: determine what events are important, how they can be obtained, who should receive these events, how these event should be transmitted etc.. This in some cases is a major effort, teaching the customer to "think in events", and building the way to emit and consume events, in some cases that by itself can be a big investment, since it can affect running systems, and invest in development of instrumentation/adapters/periodic pull mechanisms. The type of "event processing" that is being provided by basic EDA is typically routing (including subscriptions) and filtering. This phase should have a business benefit by its own right to justify the investment.
• phase II - mediated event processing: to bridge semantic and syntactic gaps between the event producer and consumer, there are additional mediators that provide ESB type functionality: transformation (including split and aggregation), enrichment (from external sources) and validation - this is often needed, and since many of these features are pretty much common, it may make sense to use COTS for it, although in simple cases, hard-coding may be cost-effective enough, this phase is optional, it may make sense to unify it with phase III.
• phase III - complex event processing -- this is a step-function towards looking at "event clouds" instead of events one-by-one, and typically built on top of the phase I, possibly I +II. I have seen cases in which phase III has been implemented as first step and included step I and II -- this makes step III much longer, but makes sense if the customer has a business problem for which it needs step III, but really did not think in event before.
• phase IV (optional) -- This extends the functionality towards intelligent event processing - which I already discussed in the past.

More about maturity models and event processing - later.

On the classification of events

I am returning to the area of "classification of events", although it has been discussed in this blog several times regarding certain aspects, the reason is - some discussion in which I realized that the confusion is still alive and kicking.

Events can be classified according to their structure, semantics, or pragmatics. This classification is orthogonal.

Based on Structure:

• Simple or atomic event - an event that reports on a single occurrence.

The event structure may be flat (like relational database), or semi-structured (XML) - which also can express non-flat structure (e.g. hierarchical). In some cases events are represented as objects. Typically events contain "header" - information about this event (event class, time stamp, source...), and "payload" - the content of the event.

• Complex event - an event that is composed of multiple events

There is some difference between this and the term "composite event" from active databases which mixes structure and semantic aspect, but I don't see it useful to use this term any more.

Based on source

• Raw event: Event that has been reported to the event processing system by an external event producer.

Note that a raw event can be simple or complex. The term "raw" is relative, since it can be result of some computation process in the event producer that is transparent to the event processing system.

• Derived event: Event that is created by the event processing system as a result of some processing.

Again, the derived event can be simple/atomic - where the attributes are computed as functions of raw and derived events, or complex - in which the derived event is a concatenation of events.

Based on ontology:

• Real event: An event that designates something that occurs in reality
• Virtual event: Hypothetical, simulated or uncertain event.

Again, this is a separate dimension - can be in any structure and by any source.

Based on pragmatics

• Reactionable event (AKA situation): An event that is consumed by an external consumer at the end of the event processing network (either notify or orchestrate).

We used the term "situation" for a phenomenon in the consumer's domain of discourse that the consumer wishes to react to. This can be raw event (and in this case the event processing system is reduced to "simple event processing" - i.e. filtering and routing) or derived event, it can also be real or virtual, and in any structure.

BTW - ECA - "Event-Condition-Action" rules are rules in which the events are situations, and they are executed within the consumer domain, but I'll continue to discuss the conceptual model in the next posting.

On the Event Processing Conceptual Model

The event processing conceptual model shown above is a result of a work by an IBM internal team lead by my colleague Mamdouh Ibrahim. I have presented it in the "reference architectures" session of the last EPTS meeting, we have decided not to use the term "reference architecture" since it has multiple implementations, but this represents the different parts of an event-based applications. The three major building blocks are - event prodcuers, event processing network, and event consumers.

• Event producers - emit events in push or pull (periodic or on-demand).
• Event Processing Networks - process the events, create derived events where applicable, route the event to event consumers at the edge of the network
• Event consumers - consume the processed events and orchestrate/notify.

As you can also see the event processing network has two bridges - one to the producers and another one to the consumers, in which several of the functions of the EPN may reside.

We believe that this is a general enough conceptual model that can contain various variations.

Note that the "event processing network" requires an "event bus", this is a platform-independent term - in SOA environment - as noted in a previous posting the "event bus" converges with the "enterprise service bus". More deep description of the different ingredients - in later posts

On Trifecta and Event Processing

Reading the educated blog of my friend Tim Bass
I had to admit that I don't have any clue what a "Trifecta" is, and rushed to wikipedia

for help to find out that in horse races - it is a bet on the first three horses. I must confess to my ignorance - I have never watched a horse race, neither have I gambled on horses. I have made a major technological gamble though, around 10 years ago, to bet around "complex event processing" (without using this name and without knowing on the three others who have worked in parallel on the same topics - will have at some point postings about the history), and after suffering some frustration periods, it seems that the reality also slowly getting there, and I still hope it will be the "next big thing" - but we have some work to do in order to get there -- again, a topic for another posting. Now the question posed both by Tim Bass, and by Joe McKendrick

is - whether CEP is depended on SOA, and in turn - does it require the coupling of SOA and BPM - which is by itself a problematic concept.

My answer is -- CEP is an horse that is indepdent in any other horse, donkey, mule or tiger. As I noted in earlier postings, SOA and EDA are orthogonal - SOA is about modularization or componentization of the enterprise, and hence it IT systems, introducing modules as "services". Now let's talk about the relations between SOA, CEP and BPM.

(1). Services in SOA can be producers and consumers for "Event Processing Agents"; as a producer - the service is instrumented to emit events, as a consumer it can be notified or triggered by an event; Services are one type of consumers and producers but not the only type - every application can be consumer or producer, thus there is a loose coupling.

(2). In SOA environment there is an ESB (Enterprise Service Bus), whose functions have some overlap with event processing (especially mediated event processing) - can be a natural host for the "event processing network" (I'll write more on ESBs in another posting). However, again Event Processing Agents can also run outside ESB.

(3). Workflows (BPM) are special type of consumers and producers. In the SOA world, BPM orchestrate services, and thus, BPM can emit events about the status of the workflow, while as a consumer it can add/modify/delete one or more workflow instances. Some of the Event Processing applications (e.g. some types of BAM applications) are based on business processing - however, even in SOA environment this is only one type of consumer and producer, and there are other types (e.g. RTE applications which process events in-line and not in observation mode).

Bottom line --- Event Processing can have different interactions with SOA, and when IBM's announcements in this area will be available you'll realize that there are different entry points. Event processing can also work in legacy and non-SOA environment.

So today I have learned about horse races and promised some more postings (not promising when) -- more, later.

On Event processing and Event Driven Architecture

As those who read this blog can notice, I use the term - "event processing" as the name of the discipline that this blog deals with. Other people are using the term EDA - "event driven architecture" as their key terms? one of the questions I was recently asked - are they the same, or similar like the two Arizona Moths that you can see in the picture above ? this, of course, depends on the exact meaning, and people do use them interchangeably.

In my opinions these terms are different.

EDA deals with the way components communicate, and in this case, unlike the request/response style, EDA is loosely coupled, asynchronous, and delivered via push.

Event Processing deals with the end-to-end processing of events, it may or may not be based on EDA. There are some cases that are EP, and are not EDA, examples:

• Event can be obtained in pull (periodically or on demand)
• Event can be part of transaction -and thus there is a dependency
• Event can be communicated to an event processor using request/response protocol, but the actual functionality is event processing.

(may be other reason).

So EP and EDA are not really identical...

On CoDA - Context-Driven Architecture

Yefim Natis from Gartner, the same analyst who is bringing us the XTP vision
is also responsible for the brand new vision CoDA = Context-Driven Architecture; which is one or two steps further from the XTP era. In the previous posts about contexts: (1) ; (2)
I have argued about the use of context as a first-class-citizen in event processing; and indeed people act and think within contexts, and the use of explicit context can get the computing closer to the human thinking, and also has computational benefits (like indexed vs. sequential search in the early days of file systems). Both Yefim and myself did not provide formal definition of context when talking about it, and I still have a "to do" to think about it. One comment though about the "Architecture" - some people think that SOA and EDA are contradicting terms, since there can be only one "big A" - meaning architecture. As I have already mentioned - in the posting about EDA and SOA that they represent different aspects of computing, thus they are orthogonal (but can co-exist). The question is - whether context is yet another dimension that can co-exist with the other aspects ? again - still need to think about it. Thus, I'll return to the context concept soon.

More on EDA is EDA and SOA is SOA

Hello from Haifa again. My good friend Tim Bass has written a blog entitle EDA is EDA and SOA is SOA. I totally agree that EDA and SOA are not the same, more than that, I don't even think that they are talking about the same thing. Many people equate SOA with the synchronous "request-reply" style of interaction, but this is only one way to implement SOA, since SOA is not about interaction style, but about modularization or componentization of the enterprise, and hence it IT systems, introducing them as "services", the interaction style is a secondary issue, services can interact both in synchronous and asynchronous ways, and there is nothing inherent in the concept of services that say anything about the interaction type. EDA is indeed about asynchronous push interaction style, and this can be interaction between services, where we get both SOA and EDA, or interaction between other software artifacts, in enterprises that don't implement SOA. Implementation of SOA and EDA both make sense for other reasons and to solve other problems, and they are completely orthogonal. In enterprises that implement SOA there is a sense to use a combination of both, thus, it is important to position them in a way that they are complementary and not contradictory, since there are some misconceptions about it (exactly since some people interpret SOA = request / response). I may dedicate another blog to in-depth discussion about this two concepts. A side point - since this blog deals with "event processing", event processing is not implemented on top of EDA -- there are some cases in which event processing has "request-response" functions such as: getting events in PULL from the producer, which may be done periodically or on-demand, or event processing network that is implemented as part of transaction, and thus does not fulfill the loosely coupled principle of SOA. Thus "event processing" is not a pure EDA.... confused ?
more - later.