Web services triggered by SAP ESP

More from the event processing vendors Blogs.  This time from SAP.  Its ESP product.   SAP event processing Blog notifies on a new feature in the product that enables a derived event to trigger WSDL that prompts the user for requesting a predefined web service.  In the EPN terminology, this WSDL is an event consumer.  There are also other products that cross the borders and provide actions as part of the event processing product.  As integration with adjacent systems is a major issue, there are already several solutions which extends event processing to the consumer side, with built-in actions. It is also useful to extend to the producer side -- the instrumentation process.  I'll write about the producer side in another post.    

Another type of proactive systems

Today I had a meeting with PhD student and her adviser to help them do sanity checks for a research idea related to event processing.    One of the ideas coming from them is actually looking at another type of proactive systems relative to what we have been looking at.   The difference stems from the fact that event producer can be of two types:  observer and creator.   An observer, e.g. sensor, observes that an event has happened and reports it, but it does not take any part in the event creation.   A creator is a producer that creates the event, and can decide whether an event is created, or in some cases can retract events.  Such a creator can be an instrumented program, BPM,  transaction system etc...

The proactive systems we have been looking at assumed that the producer is observer, thus we cannot control the creation of events but we can try to predict it, and eliminate it, or mitigate the damage it might inflict.   

Contrary, when the event producer is a creator, we can control the event.   There might be a few cases here.

One case can be that we again predict that event is going to happen since there is some causality relations with events that have already been detected, and within a certain context we don't want this event to happen,  we can then tell the producer -- if this event is being produced (again, within certain context, e.g. time window) -- don't produce the event.    Another case can be that the event is being created by a transaction system,  this event, within some context, and maybe in conjunction with other events, derives a situation that we want to eliminate.  The fact that the event is part of transaction that has not committed, gives the possibility to retract the event and eliminate it from being produced to the outside world when the transaction commits.  

The pure view of event processing puts producers and consumers outside the boundaries of the event processing system,  however, in reality there are cases where either the producers or the consumers (maybe even both) are controlled by the same logic, and thus detection of situation may impact the producers and consumers.   I think it gives rise to interesting applications.  I'll get back to this topic later.

On events versus data

The word "data" always reminds me of the android from Star Trek The Next Generation whose name was data. The word data (in computing) typically is very general and refers to anything the is represented on digital media, the picture of data above is also a piece of data, like many other things. The word "event" also has a broad term which means something that happened.

Recently Paul Vincent wondered in his Blog about the difference between event and data, as some people think that events are footnotes to data. Since by the definitions above, obviously event and data are not really the same, I'll try to talk about the touch points among them, since those are the reason of misconceptions.

There are various touch points between events and data:

1. Event representation contains data. Event is represented in the computing domain by "event object" or "event message" which usually is also is called "event" as a short name. This event representation includes some information about -- what is the event type, where it happened, when it happened, what happened, who were the players etc... Example: the event is "enter to the building", the event's payload contains information that answer questions such as: what building? who entered? when ? and maybe more. The payload of the event is data, it may be stored (see event store), or just pass by the system.
2. Data store can store historical events. Event representations can be accumulated and stored in a data store, for further usage. There are large data stores that collect weather events. Note that in order to navigate in historical events, these events may be stored in a temporal database an area that I've dealt with in the past, sometimes if the events are spatial then it have to be stored in spatiotemporal database.
3. Database can be event producer. In active databases the event were database operations; insert, modify, delete and retrieve, in this case the fact that some data-element has been updated or accessed is the "something that happens" (which may or may not reflect something that happens in reality), and the database acts as event producer and emits event for processing by an event processing network. Note that actually all event producer contains some data that is turned into event, for example transaction instrumentation like what IBM has done in CICS as event producer.
4. Derived events as database updates. An event processing application take events from somewhere as input, does something, and creates derived events, and send them somewhere, this is all event processing is in one sentence, a derived event created in this process may go to an event consumer, the event consumer may be a DBMS or another type of consumer whose action is to update some data store.
5. Event enrichment by data during the event processing. During the event processing operations, sometimes enrichments of events is requested, let's return to the event of a person enters a building, the event processing application deals with security access control, and needs to know what is the person security clearance, this information is not provided with the event which provides only identification of the person, and there need to be some enrichment process in which an enrichment event processing agent accesses some global store, in this case reference data, to extract the clearance value and put it inside the event for further processing.

Thus the main issue is not the "versus" issue but the various relationships between the two terms.

More on the ecosystem of event processing systems

The one week school break for children ends tomorrow, and today I went with my two younger daughters to see Avatar, a very ambitious movie with a lot of advanced 3D graphics. The downside is that the film spans over 3 hours without a break -- could cut 1 hour easily, but nevertheless the graphics is very impressive, the plot is kind of paraphrase on other movies.

In my previous posting I've written about event processing functions, one of the frequently asked questions is whether event processing has a value by its own right, or as part of a larger ecosystem. Let's look at another question: DBMS software deal with update, retrieve, store, organize, backup and restore data, it also supports concurrency control, transaction management and other stuff. DBMS itself does not have value, the value is in the use of data.

In a similar way, event processing software deals with transferring, filtering, transforming, pattern matching, and situation discovering, it also provides execution control and support of some non functional properties, but the aim is to receive event process them and derive more events. The value is not in derivation of events, but in the way it is used.

In order to complete the cycle we need two other types of software: event producers that produce the raw events, and event consumers that consume the derived events and execute the actions triggered by these events -- the consumers are those who are driving the value.

The producer can be any type of software or sensors. One example of event producer I've written about in the past is the "EDA extension of CICS" that I have written about before. It is an example in which a software is instrumented to produce events. We are seeing more software of this type in the sensor area, and other areas as well.

Consumers are the part of ecosystem that utilize the events flowing from the event processing systems. I've posted before about the consumer chapter in EPIA, There are many ways to consume events, they can also be connected with other types of enterprise software: Events can drive decisions, and can serve as input to business rules; events can drive KPI (Key Performance Indicators) metrics and can be input to BAM systems, it can have various actions related to BPM systems, events can drive real-time analytics and be input to BI systems, it can post on Twitter, and update Ambient Orbs (see the posting about the EPIA chapter), or if it is part of a "smart house" it can turn off lights or control the temperature using some actuators.

Like DBMS that requires somebody to feed the data, and somebody else to use it, event processing requires its ecosystem, we'll see more of the ecosystem software both in the producer side (like the CICS example) or in the consumer side (like BAM software). More on event processing ecosystem later.

On the EPIA chapter about event consumers

Back to writing about the EPIA, the next chapter out, will be the chapter dealing with event consumers. As anyone who has developed an event processing application may realize, the event processing, is just part, and sometimes the small part of the picture, and in order to be useful, the consumers and producers have to be set. A consumer consumes the events (either raw events or derived events created by the event processing system), in this chapter (written by my partner Peter Niblett) there is a classification of event consumers, and plenty of examples. Here is some classification

And here are some cool consumers:


This, believe it or not, is an application that consumes Twitter events. This map following a bus that notifies Twitter anytime that it gets to a station, it reports where the bus is, and how many passengers are on board. You can read more details and even follow this service on Twitter (169 followers, when this Blog is written). The map is an event consumer that consumes the raw Twitter event and displays the bus route and location within a certain time interval.

This is another cool event consumer called "Ambient Orb"; this gadget is actually an event consumer, an event processing application determines the color of the ambient orb, the application can be anything from the state of your favorite stock, the traffic load outside, the security risk, or some key performance indicator - how many purchases have been done through the company's website today. The output is a color (in the picture it is green, meaning that whatever the indication is -- the status is OK).

More about the EPIA book - later.

On Gartner's EPN Reference Architecture

Today is a holiday (for children, no vacation for adults..) called Lag Baomer, the highlight (besides not going to school) is that last night all children have gathered around bonfires, as seen in the picture. Fun.

Recently Gartner has published a report called "A Gartner Reference Architecture for Event Processing Networks".

On the positive side, it seems that the concept of EPN, as an underlying model for event processing is catching. The readers of the Blog may realize that I am in the opinion that we need an agreed upon conceptual and execution model for event processing (the same role that the relational model assumes in relational database, however, I never believed that the relational model per se, is appropriate also as the model behind event processing). The book I am writing now "Event Processing in Action" concentrates around the notion of EPN, and a deep dive into construction of EPN-based application.

Reading Gartner's report I found some slight differences between the way they describe EPN, and my own description. In the Gartner report they define a term called "dissemination network" that consists of event processing agents, channels and event flow among them, and then they define EPN to be a dissemination network + producers + consumers. I actually could not find any compelling reason to introduce the notion of dissemination network. According to the definition we are using, event processing network is a directed graph that has nodes for producers, channels, EPAs and consumers, and edges that determine the event flow among them. Another difference is that the Gartner report views event consumers and event producers as type of event processing agents. I have a slightly different opinions, I think that both event producers and consumers are not really event processing agents, since event processing agent is some software module that function events and may generate more events. Event consumer and producer have nodes representing them in the EPN in order to make the event flow from and to them explicitly, however, they are only proxies of the actual producer and consumer, for the event processing network, they are sources and sinks. The main difference is that EPA functionality is explicitly specified in the EPN definition, while what the producer and consumer do is "black box". We don't want to include their functionality, since we don't want to extend the event processing language ad infinitum,

Mentioning the EPIA book -- Chapter 3 is now on the Web, and can be obtained through the MEAP program, this is the last chapter in the introductory part, and deals with principles of programming with events. Chapter 4, the first in the deep dive will be sent to the publisher soon. It has been much more challenging to write, deals about what information we need to store about events -- I'll Blog about it soon.

On Event Processing Building Blocks

Back to work for one day in the office, with five conference calls (one with Germany, one with France, one with UK, and two with USA...) and then back to home for the weekend. When I have free time I like to read books, the current book I am reading is "A Lion Among Men", 0ne of the books of Gregory Macguire, who writes stories that take as background famous children stories (in this case - the Wizard of Oz), actually this is the third one behind the scene of the Wizard, now taking the Lion as its main character. I have another book of the same author still waiting...

We also submitted the draft of chapter 3 of the "Event Processing in Action" book to the publisher, which hopefully be posted on the MEAP site soon.

The approach we have taken in the book, as I have written before, is to use the "building block" approach, describing event processing principles, and the use case whose construction demonstrates the application, using building blocks, which are like the chemical elements. The application itself is being built by using "definition elements" which are like atoms (my partner for writing this book, Peter Niblett, has come with the analogy from the world of chemistry). we believe that this is the right approach to teach what event processing is -- in the "deep dive" part of the book we dedicate a chapter for each of the major seven building blocks and then dive deeper into the types of event processing agents (which deserves a different discussion). We'll also provide samples of how each building blocks is realized in different models.

The seven building blocks are:

• Event type: defines the event schema
• Even producer: the projection of the event producer over the event processing network (note that the event producer itself is outside the scope)
• Event consumer: same -- the projection of the event consumer over the EPN.
• Event channel: the glue that holds the EPN together
• Event processing agent: the brain that does the entire work; each agent is doing a specific task of processing.
• Context: the semantic partition of events and agents
• Event derivation: A building block that is possibly part of each EPA that specifies the derived event.

There are some more building blocks that are used to support these ones, but our claim is that this set of building block is what needed to build an event processing application.

Chapter 4 which is in advanced phases of being written starts the deep dive by discussing the event type building block, and in one of the next posts I'll say more about it.