EPTS event processing glossary 2.0 is now available

The event processing area keeps evolving and there are more terms being used.  The EPTS glossary workgroup, co-chaired by Roy Schulte and David Luckham has published a new version of the EPTS glossary.  It  contains both old and new terms.    

Is my cat CEP or not?

One of my favorite cinema artists is Mel Brooks, and one of the great films of Mel Brooks is known as "silent movie" (see flyer above). To those who don't know or don't remember, silent movie is a film about doing a silent movie, and most of the plot involves around Mel Brooks and his gang going to famous actors asking them to participate in a silent movie, they agree and this sums up their participation, since their voice is not heard; there is one exception - when asking the world famous mime Marcel Marceau to be in the film he replies, "Non", the only spoken word in the entire film, since he talked, he did not participate in the silent movie.

So here is my dilemma -- recently the blog-land is full of postings on the topic "is X CEP?" - personally I don't find that this debate leads anywhere, so my inclination is not to participate in this discussion, so I'll contribute my part by explaining why I don't find this discussion interesting.

There are different reasons why I don't find it interesting to spend time on these discussions are:

1. I don't see that the results of this debate are important to the reality.


2. One can debate terms forever - but this is not useful, there is a better way.


3. This debate will not converge anyway, the same arguments are being repeated and it does not enrich the reader's intellect to read them again.

Let's start with the reality -- what the current vendors attempt to do is to build generic products that will enable to develop various sorts of applications in the area of event processing. This is different from the past, where event processing has been done in hard-wired, hand-coded, ad-hoc fashion to build a single application. While any single application may be challenging, the challenges in building generic products that will be a cost-effective alternative to hand-coding, are somewhat different from building ad-hoc applications. The content of these products is determined by several factors - not by the boundaries between "simple" and "complex" - there are multiple dimensions of complexity, and different applications may exhibit different type of complexity (functionality, topology, scalability, event types)... No matter where we'll put the border between the "simple" and "complex", a generic product will have to support both sides of the border. Bottom line -- the importance is ROI for the customers and not whether something is called "complex" or not.

Debate over terms: when we have done the first event processing symposium, in March 2006, the conference started by nine presentations about the "state of the art", and a very quick observation has been that each of us has invented his own set of terms. Consequently we have decided that the first community effort will be to produce a glossary edited by our non-vendor colleagues, to reduce bias. They have collected input from many sources with different opinions, sometimes strong opinions (it is amazing that people are so caring about terms). The glossary has been published recently and its announcement was endorsed by many people in the community. I am sure that nobody is happy about all terms, but as a compromise among all opinions, the editors have done a pretty good job.

Personally I don't have strong feeling towards terminology, so the glossary definition is a possible one, there are others as well, there is no absolute truth here, but - I think that is good to have a precise definition of terms that are accepted by large community vs. continue debating on terminology forever, even if I personally would have defined it otherwise. Among the possible interpretations, the editors chose to define CEP: Computing that performs operations on complex events, where complex event is defined as:An event that is an abstraction of other events called its members. As far as I am concerned, this is what CEP is.

Last but not least -- If somebody decides to insist on his own (different) definition of a term T, and judge everything according to this own definition, then obviously, if a term T have two different defintions, then X may be T according to one definition, and not T according to the other definition, when the two sides stick to their definitions, then no one will convince the other anyway, and arguments will still repeat themselves. Actually, I have not learned any new insight from this discussion. Have I mentioned that the arguments repeat themselves? in case that I have not done it , I'll mention that the arguments repeat themselves.

Bottom line -- is my cat CEP ? well - I don't have a cat(I wonder if Roger King who has written the famous article "my cat is object oriented" had a cat).

There are plenty of more useful topics to Blog about - so no more on that issue.

On the EPTS Glossary

Today, EPTS has announced on the first version of the EPTS event processing glossary. The glossary has been edited by Roy Schulte and David Luckham, with contribution of many people throughout the community. The editors had quite a big challenge, as different vendors have been using different terminology, and this confuses the customers and sometimes owrselves. As you can see in the press release, there are support quotes from the entire community -- vendors, analysts, academic people and customers; this is an indication that the community prefers to have a common terms that everybody will use, instead on constant debates on terms.

The glossary can be downloaded from the EPTS website.

This glossary as one of the editors noted is a "living document" and will be updated periodically, since the area of event processing is still evolving. It is a starting point only, however, since it has been available in various draft forms on David Luckham's CEP portal, the glossary only received traction, and I am meeting people in various places who say that they adopted the terminology. Would I personally define all terms the way it has been defined in the glossary -- not really, and I guess this is true for many people, but since the glossary is a consolidation of various opinions and proposals, the editors have done a pretty good job of making everybody relatively happy, and the EPTS steering committee that consist of people who had initially used different terms, has unanimously approved it - as good enough to serve as first version.

What next ? - if you wish to propose to add more terms, or to modify the definition of existing terms - please use the Wiki on the EPTS website to make comments.

Citing my Irish friend, Brian Connel : friends, romans, countrymen - this is your glossary, help shape its future ! (I am the last year in Israel that has learned Shakespeare's Julius Caeser - so I can identify the source).

On EPTS again - calling for customers to join EPTS

After the first excitement of the launch (first press release is out, press briefing is out, some more press articles are coming soon, some of the members have referred in blogs or news, second press release is on its way) this is the time to extend the community. We have very good coverage of vendors, good coverage of academic people, some coverage of analysts. As I have written in the last blog, the target now is to add more customers. EPTS will issue a call for customers - in various places and opportunities (e.g. the Gartner EPS). What is the main motivation of customers to join:

• Impact: A lot of activities are forthcoming that will help shaping technologies, their positioning, studies about ROI, terminology, and interoperability issues. EPTS members will be able to impact all these activities - either directly (participating in working groups) or indirectly (reviewing and commenting). Some of the customers community have a lot to contribute - they give talks about EP, they blog about it, and they express opinions on forums. This will be an organized way to translate the opinions and knowledge to impact.
• Learning: EPTS will provide various community activities that will able its members to learn both on the technology and the business impact side of event processing, as well as sharing and learning best practices.
• Networking: The various conferences, the work-group conference calls, and other activities will enable active networking, besides networking facilities already exists (user groups, forums and social networks - that will all still exist).
• Help influence the establishment of "event processing" as an academic direction, impact research directions, and university level courses.

To remind you - there is no cost associated with joining EPTS, directions about joining can be found here: http://www.ep-ts.com/content/view/59/93/

Stay tuned for call for participation and contributions in the EPTS annual meeting in September 17-19, coming soon.

On the EPTS Press Briefing

When the USA president is talking with reporters he is standing on podium and facing the camera lights; yesterday I had participated in a press briefing in a much more modest setting, through the phone, sitting in the office I have borrowed for this week in IBM Hursley Lab, in UK (picture of Hursley can be found in another Blog entry ). The occasion has been - announcing EPTS, here is a transcript of my short briefing:

"Today We are announcing today EPTS which is a result of discussions between vendors, academic people analysts and customers that started two years ago in the "first event processing symposium", and continued in two subsequent meetings. One of the conclusions of these meeting was to form a permanent forum and working groups to promote the understanding of what event processing is. The list of founding members include around 30 organizational members – vendors, analysts, and customers, and 30 individual members – academic people, customers, independent consultants. You can see the list in our website. The steering committee consists of the group of initiators.

Why do we need another consortium? - I have watched event processing area evolving in the last 10 years , it is already making noticeable traction in the industry - However I believe that the big challenges are still ahead of us, since the industry barely scratched the surface of the potential use – and our mission is to help raise the level of wisdom in business decisions, business observations by correct utilization of such technologies.

Event processing is somewhat different computing paradigm and there is a need to help the customers understand this thinking and the value they can gain by using it. Hence, Our First goal is to document usage scenarios where event processing brings business benefits – most importantly, the business world needs to understand where this can be used, and the different ways it can provide benefits.

Event processing has been introduced in parallel by various vendors each using slightly different terminology, and there is a need to normalize the terminology, thus our goal is to develop a common glossary for its members and the community-at-large to use when dealing with event processing: A draft already exists and will be publicized soon.

We also believe that the academic community can help coping with the challenges, thus, one of our goals is to help establish "event processing" as a proper academic discipline by involving the research community and get them to focus on the right set of challenges.
¨ Last but not least - looking forward, help accelerate use and incubation of standards. This is similar to steps that other disciplines, e.g. databases, moved towards maturity.
¨

In conclusion – today we are announcing EPTS, a consortium of vendors, academic people, analysts and customers intended to promote the understanding and the development of the event processing area. "

The founding members (the list shown below has been sent to the reporters) represent vendors, academic people, analysts, customers, and consultants that are affiliated with event processing.

I hope that this list will grow in the future, we especially target customers that are interested to contribute to the community and gain from the collective insights to join, we shall issue soon a call for customers detail their value add from participating, the two first customers that have joined are - Betfair and Mitre.

Last but not least -- Mark Palmer has reminded me of our first meeting three years ago which took place in the lobby of the Spring Hill hotel in Tarrytown, NY. This meeting has indeed was the first step that started to roll the ball, and indeed we had very vague idea of where this ball should go, so Mark desrve much credit for his immediate enthusiasm from the vague idea.

Many people have been involved so far, and more people hopefully will be involved from now on; We should keep the drum beating, by a series of activities, with community efforts. Some such efforts are already in the way -- the glossary and the use-cases team, stay tuned for more details.

List of EPTS Founding Members:

Organizational Members

Active Endpoints
Aleri Inc.
Betfair Ltd.
Cabahu Pty Ltd
CAC Corporation
CITT GmbH
Coral8
Cordys
Elemental Links
Event Zero Pty Ltd
Forrester
Gartner
IBM
IDS Scheer
Leahy Consulting Group
MITRE
Nastel Technologies
OpenConnect Systems
Oracle
Progress Software
RuleCore
RuleML Inc.
SENACTIVE
Streambase
Streametics
Systar
TIBCO
WareLite Ltd
West Global

Individual Members

Individual Member
Alezandre Alves
Andre Bolles
Avigdor Gal
Beth Plale
Carlo Zaniolo
Christian Wolff
David Luckham
David O'Reilly
Francisco Gomez
Francois Bry
Gary Hale
Gero Muehl
Hans-Arno Jacobsen
Jeffrey Adkins
Jonas Jacobi
K. Mani Chandy
Michael Eckert
Mikael Berndtsson
Nenad Stojanovic
Pedro Bizarro
Philip Howard
Rahunandan Kandasamy
Simon Courtenage
Susan Urban
Themis Palpanas
Tim Bass
Udi Dahan

On the three meanings of CEP

There is an old Jewish story on two people who had some dispute, and decided to go to the Rabbi and ask his opinion. The Rabbi listened to the first person and told him: you are right, then he listened to the second person and also told him: you are right. The Rabbi's assistant who has been present asked him: Rabbi, how can they both be right ? and received the obvious answer: you are also right.

Recently, a dispute between two opinionated persons - Hans Glide and Tim Bass has stormed the network, and somehow got also to my Blog. Somehow both of them understood that what I've written is consistent with their view - which encourages me to change career and become a Rabbi, but I think that there are some skills I lack - so anyway, I'll stay on event processing thinking.

While the dispute started around POSETS, the last posting by Tim Bass in the CEP forum re-focuses the discussion around - what is CEP ? so I'll stay with this topic for a while, since I think that there are (at least) three different interpretations of what CEP is - and this is a source of a lot of confusion. Thus I'll try to explain the three interpretations.

Interpretation One: the glossary interpretation -- complex event is the processing of complex events, where complex event is an abstraction or aggregation of events. According to this interpretation, a software function can be defined as CEP if it involves the processing of multiple events - typically collecting or matching some pattern among multiple events; The test for CEP according to this interpretation is typically support of a state that accumulate the relevant events - since they typically arrive over time. This definition does not say anything about the number of events, number of event types, whether they are totally ordered or not, or whether causality relation is supported or not - these are all attributes of specific implementations.

Interpretation Two: Event Processing = Complex Event Processing. This is a common practice in the commercial work to label all event processing functions as CEP. This spans from support of CEP (according to the interpretation one) and other event processing functions (such as: routing, enrichment, transformation) that don't satisfy the CEP test (since they are stateless and deal with a single event at a time). It can get to an absurd that some product that does not support CEP at all according to interpretation one, calls itself CEP. This is the most confusing interpretation, IMHO.

Interpretation Three: Complex event processing is event processing that has some complexity associated with it. According to Tim Bass - hidden causality and Markov Processes are vital for something to be defined as CEP. This really says that it CEP must involve uncertain events, causality that need to be discovered (by mining and other techniques), and the general usage of CEP is to predict something according to analysis of recent (past) events. According to Interpretation three, indeed the products that current products that call themselves CEP, do not satisfy this criteria, and thus are not CEP.

My opinion: as stated in the past, the term CEP has some inherent ambiguity, therefore I always thought it is confusing term. As far as my own taste in terminology - I prefer Interpretation One, saying that CEP is a subset of EP functions that deal with "complex events", it also seems that this is the closest to glossary definition. Interpretation two is confusing, as it turns CEP from a well-defined term to a marketing buzzword, and thus there is no test for what it is. Interpretation three is interesting, there are certainly applications that require prediction and various usages of stochastic processing and use of AI techniques (machine learning and others) in event processing. Hidden causalities is an important term, and I'll refer to it in another posting, since it has some pragmatic difficulty to obtain. However, I prefer to stick with the concept that CEP is processing of complex events, and not complex processing of events, and for interpretation one, we don't really need (necessarily) to apply AI techniques, this is just one type of application, there are a variety of applications that does not require it, and that detect predefined patterns on events is sufficient.

So the terminology that I personally prefer is:

Interpretation One = Complex Event Processing

Interpretation Two = Event Processing

Interpretation Three = Intelligent Event Processing.

On Event Processing Agents

There are different type of agents -- double agents, as seen above which is a series of sweets, insuracne agents, travel agents, and some computerized agents - in my past I have dealt with mobile agents, and there are the intelligent agents in AI, and our own event processing agents (EPA).

David Luckham has written an amusing piece that follows Paul Vincent's "CEP and Agents" in TIBCO's Blog. The amusing part is that Paul has written about AI agents, which uses somewhat different terminology then the event processing terminology, and putting it in a "CEP Blog" is somewhat confusing. I am a "product" of the databases community, and have done some work that was on the AI border in the past, alas, the AI folks are using different terminology to talk about the same thing, and I thought at that time that they are doing it on purpose to confuse me. So, while there are many types of agents, I'll concentrate on the concept of "event processing agent" that has been coined by David Luckham. I like this term and adopted it in the following way: EPA (Event Processing Agents) is a software artifact that receives an event cloud or stream or collection of events or a single event (depends on the agent type and capabilities), does some computation on these event, and produce one or more events as an output. That's it. EPA is also a node in the EPN (Event Processing Network). There are different types of EPAs :

• "simple event processing" EPAs - filter and routing,
• "mediated event processing" EPAs - enrichment, transformation, validation
• "Complex event processing" EPAs - pattern detection
• "intelligent event processing" EPAs - prediction, decisions...

The common denominator: each of them receives events as input, emits events as output and does a single type of function.

I find this type of abstraction both very easy to explain people how EP systems work, and also basis for architecture. The EPN routing can be done by standard middleware, or in a stand-alone mode. Other terminology issues raised by David Luckham is the relationships to the "actor model" and to "engines".

The actor model is a model that helps reasoning about concurrency, while agents in AI are autonomous goal-driven artifacts. These are orthogonal terms, of course. In the context of EPA - when looking at EPAs as an executable network, we can look at each EPA as an actor and apply actor models.

Last but not least -- relationships of EPAs to engines -- an EPA is a software artifcat, it can be an instance of an engine, it can be some software that contains an engine, and it can be hard-coded program, as long as it complies with the EPA definition. In a future world, with inter-operability (and perhaps also language) standards, we'll be able to run (and maybe to self-select) multiple engines for the same EPN, residing in different EPAs.

More about EPA types -- later.

On the classification of events

I am returning to the area of "classification of events", although it has been discussed in this blog several times regarding certain aspects, the reason is - some discussion in which I realized that the confusion is still alive and kicking.

Events can be classified according to their structure, semantics, or pragmatics. This classification is orthogonal.

Based on Structure:

• Simple or atomic event - an event that reports on a single occurrence.

The event structure may be flat (like relational database), or semi-structured (XML) - which also can express non-flat structure (e.g. hierarchical). In some cases events are represented as objects. Typically events contain "header" - information about this event (event class, time stamp, source...), and "payload" - the content of the event.

• Complex event - an event that is composed of multiple events

There is some difference between this and the term "composite event" from active databases which mixes structure and semantic aspect, but I don't see it useful to use this term any more.

Based on source

• Raw event: Event that has been reported to the event processing system by an external event producer.

Note that a raw event can be simple or complex. The term "raw" is relative, since it can be result of some computation process in the event producer that is transparent to the event processing system.

• Derived event: Event that is created by the event processing system as a result of some processing.

Again, the derived event can be simple/atomic - where the attributes are computed as functions of raw and derived events, or complex - in which the derived event is a concatenation of events.

Based on ontology:

• Real event: An event that designates something that occurs in reality
• Virtual event: Hypothetical, simulated or uncertain event.

Again, this is a separate dimension - can be in any structure and by any source.

Based on pragmatics

• Reactionable event (AKA situation): An event that is consumed by an external consumer at the end of the event processing network (either notify or orchestrate).

We used the term "situation" for a phenomenon in the consumer's domain of discourse that the consumer wishes to react to. This can be raw event (and in this case the event processing system is reduced to "simple event processing" - i.e. filtering and routing) or derived event, it can also be real or virtual, and in any structure.

BTW - ECA - "Event-Condition-Action" rules are rules in which the events are situations, and they are executed within the consumer domain, but I'll continue to discuss the conceptual model in the next posting.

EPTS plans for 2008

This picture has been taken in the EPTS meeting in Orlando in September 2007, the meeting itself has been described by Paul Vincent in his blog : first day, second and third day.

As written before the EPTS is a consortium (under construction) intended to promote the understanding and incubate standards in the event processing area. Getting into 2008 - here is the work plan that has been recently approved by the steering committee:

Quarter 1: Launch - as expected when some of the players are big companies, the formalization process take some time, but it will hopefully converge soon. After getting approval of the steering committee - the idea is to invite vendors, customers, academic people and other individuals to join the list of founding members. The launch will also include - introduction of EPTS website (a prototype already exists - thanks to Brian Connell from WestGlobal and Serge Mankovskii from CA) which will support the work of work groups and other EPTS related information (leaving news, blog pointers, general discussion forum for David Luckham's site ). In the launch opportunity we'll also sign and seal the first version of the consensus glossary edited by David Luckham and Roy Schulte, with many contributions provided so far (a draft of this glossary is available)

Quarter 2: "State of the practice" white paper based on a collection of use cases. This is a workgroup launched in the last meeting, with a diversified team of volunteers. The first phase carried out by Tao Lin from SAP, Dieter Gawlick from Oracle and Pedro Bizzaro from University of Coimbra in Portugal to define a template to describe and compare use cases, where the use cases are those presented in the three EPTS meetings, in the Dagstuhl seminar on event processing and maybe open for others. After the glossary, this will be the next community effort in order to understand the state of the practice (there has also been some work last year on "reference architecture" lead by Tim Bass that we'll return to after completing the use cases survey).

Quarter 3: In this quarter we'll hold the fourth EPTS meeting (place and time are not determined yet), participate in the DEBS conference, which we support of becoming the research "flagship" of the event processing community. We'll also launch a portal of teaching material on event processing for the next school year.

Quarter 4: Our first contribution to standard -- we'll launch in early year a workgroup to provide the EPTS input for the OMG RFP on meta-modelling.

Many people are contributing their time and energy for these community activities, and I am looking forward to the momentum that will be created after the EPTS formal launch... stay tuned, or even better - join. Instructions about joining -- soon.

CEP and the story of the captured traveller

Reading the recent posting of my friend Tim Bass entitled "CEP and the story of the Fish" I decided to answer with another story (from the other side of Asia) :

A traveller went in the jungle somewhere on the globe and unfortunately was captured by a tribe that is still using ancient weapons. He is brought to the chief, and the chief says - " You have trespassed into the tribe's territory, which is punishable by death, however, I am a very curious person, if you'll show me something I haven't seen before I'll let you go"; our unlucky traveller started to look in his pockets and the only meaningful thing he found was a lighter, so he took his chance, showing it to the chief saying: "this thing makes fire", however, since he was in under a big pressure, he pressed once - no fire, pressed twice - no fire, in the third time the lighter indeed has produced the promised fire, the chief did not hesitate and said "let him go", so our relieved traveller muttered to himself - "I knew that they have not seen a lighter", but surprisingly to him the chief said - "oh, I have seen many lighter, but a Zippo lighter that does not light in the first time I have never seen".

When someone disagrees with somebody else, it is very easy to assume that my point of view is right since I am smarter / knows more / more qualified / older / more experienced / generally always right etc... My preference is not to doubt the wisdom, experience or qualification of anybody that I am arguing / discussing / debating with, but make the arguments on the issue and not on the person who makes the arguments....

Enough introduction -- now for the main message of this posting, the term CEP (Complex Event Processing) has more or less agreed now in the industry to denote "computing that performs operations on complex events", where complex event is an "abstraction or aggregation of events". The term complex does not say that the processing is complex, but that it deals with complex events, as defined. Complex event processing is typically detecting predefined patterns that can be expressed by queries/rules/patterns/scripts and are deterministic in nature. Regardless if I think that this is the best term, I think that it is important to have common agreed terminology, otherwise we are confusing the industry, the customers (and sometimes ourselves). Now, Tim Bass claims that since event processing with stochastic/probabilistic/uncertain nature is more complex than what we call "complex event processing", we have to call this one - "complex event processing", and rename what we call "complex event processing" to be "simple event processing". Unfortunately, it is too late for that - and also not justified, again, since the "complex" in the "complex event processing" does not say that this is "complex processing of events" but that this is "processing of complex events" (very common misconception !). Bottom line: yes - there is another class of event processing capabilities that requires techniques from AI, machine learning, OR etc.. and that is not deterministic in nature; no - I don't think we should call it "complex event processing", we have suggested the term "intelligent event processing" which I have already referred to in previous posting , there are a variety of other postings that I have dedicated to terminology.

More - later

On Virtual Events

The glossary defines event as something that happens, it also has some escapte to talk about virtual events that is: an event that does not happen in the physical world but appears to signify a real world event; an event that is imagined or modeled or simulated

Which leads to the question- when we are processing events, is there a place to process events that don't really happen in the physical world - well, today that people have second life in a virtual reality, and there are some events in this reality that worth processing, this is an example of a virtual event, but there are some other examples -- when event processing is used in a simulated mode, the simulated event also does not happen in the physical world, if an event is predicted it has not happened (yet?) in the physical world and there are some other examples. The question is -- what is the difference between event and virtual event ? are the boundaries even clear ? - in the meta-physical world the answer is YES, an even either occurs or does not occur (following the law of excluded middle), however, in reality we may have uncertain events, thus, when processing this event we don't really know if it is a real event or a virtual event. In previous posts I have started to talk about the importance of context and this is the key to handle these events - an event happens within a context. The context may be physical deterministic world, virtual world, physical stochastic world, simulated world, predicted world etc - and the event happen within this context. The type of processing for these events is the same, so something that is real in one context may be virtual in another context and vice-versa.

Furthermore, something that may be considered as an important event in one context For example:

the event that the Dow Jones is up by 5 percent is an exciting event for some people, and a non-event to the fisherman on the Pacific island beach who does not even know what Dow Jones is, but for him the capture of a fish that weighs 100 KG is a big event, and the broker in Wall Street is not excited from such an event, thus - the question whether something deserves to be called "event" is also a question of context. I'll get to a more formal definition of context in the next posting.

On Event Representation

Back to micro-oriented issue, and today I'll start discussion about -what's behind the definition of the event processing glossary and get to the issue of event representation. As the glossary says - event is something that happens in reality. We also tend to call "event" to the representation of this reality for the purpose of processing by a computer. This notion in event has in the glossary several aliases: event object, event message and event tuple. The various aliases are indications that the space of event representation is not uniform, some think about event as a message that moves around, some thinks of it as a tuple, which is part of a stream, and really the twin brother of a tuple in relational database, some think of it as an object with arbitrary structure (which may also be hidden). Obviously, there is no "universal event", and unfortunately, since in many cases, events are already given from the sources with their given formats, and the event processing designer has little to say about it, then a generic event processing system has to support multiple type of events, or have adapters that translate all types of event to some cannonic type of event (and typically both -- supporting some cannonic type of events and having adapters translating other types of events to the cannonoic type). Event can be structured, semi-structured (XML), and unstructured (the area of unstructured events processing deserves more focused attention). One of the questions is - whether there are common attributes that each event should have to enable event processing. In the data world - the answer is no - there is not a single attribute that must exist in all relations (besides the fact that each tuple should be a member of some relation - no floating tuples). For event processing -- there are some attributes that have been proposed as common attributes:

• Event-type
• Source
• Time-Stamp (or Time-Interval)

Let's look about the question - are they mandatory or not:

• The first question is whether each event is an instance of an event-type (or event-class). The glossary says - yes ! "all events must be instances of event-type". This seems reasonable, however, we may think of some exceptions - such as rare events that have not been classified.. I need to drill down on rare events in some other post.
• The second question is whether the source should be mandatory - again, this is desirable if we want to have lineage or tracing back actions/decisions, but there may be cases in which the source is indefinite, or we wish to hide the source (e.g. leaking of information).
• The third question is whether each event must have a time-stamp (or time-interval in case it happens over an interval - another area that needs more discussion) - the answer is that many event processing patterns are time related, and if we want to know which event occurs first, or if two events occurred within 5 minutes of each other, we need to know WHEN this event occurred in reality. However - in some cases it is not known, in other cases it is not really needed.

It seems that all common attributes are useful, but may be optional in some cases.

There are attributes that are common for types - such as probability for uncertain events, spatial coordinates for spatial events etc -- this is before relating to the content.

The content is determined according to domain related ontologies - and there is a lot of work today in different application domain or industry to define such ontologies. XML is the ontology language, and it has its own benefits, it also carries overhead relative to "flat" events in which the attributes are positional oriented and not keyword oriented.

Events also carry semantic information - such as: reference to entities in certain roles. In fact, event can be thought of a transition between one state to another and the information included in the event refers to a change in the universe such as:

what was changed ? what entities are affected? when it was change ? where did the change take place ? what other information is important about the change ?

This short discussion raised already several open issues that deserve further discussion - so I'll put these topics on the queue for further postings.... more - later.

Is there a non-event event ? on absence as a pattern

Recently, there were some discussions, related to the glossary, about the term "non-event event" , this seems to be a funny notion that seems to contain self contradiction, similar to "soapless soap" I don't think that this term is very good - I prefer to call it
"absent event" .

Now the questions - how is it really defined ? and is it really an event?

As for the first question: an absent event is a CEP pattern that is defined within a context. Taking some examples:

• There have no been any major financial transaction during the working day for the customer John Galt.
• The Pizza order has been received, but the delivery did not arrive within 40 minutes.

In these two examples we are looking for - things that did not happen within a certain context; the first example is the context of - "John Galt's financial activities within a single working day", the event that did not happen - "major financial transaction" is by itself a pattern that need to be defined and detected (say - a transaction with financial value of more than $10,000). The absence is on all time stamps that belong to the context - so the formal definition of the absence pattern of event e in context C is defined as : For context C, for all time-stamp t that belongs to the temporal extension of context C, event e does not occur in time-stamp t.

Let's check this definition in the Pizza order - the context here is a certain order, and its temporal extension starts when the Pizza order is confirmed, and expires after 40 minutes (which is the public commitment of the take-away Pizza shop for Pizza delivery).

This is also an example of the use of context as a major abstraction that gets the thinking about event processing closer to the way people think.

Now - the question whether the detection of an absent pattern is an event --- the answer is - a pattern detection by itself is not an event (although it may be an event from the point of view of the internal management system of the EP platform), however, it may create a derived event, which has some structure - for example - the derived event that is created in the wake of the lack of Pizza delivery consists of : date, time, pizza store id, value of order - this event may be enriched with summary of past late deliveries from the same store, and then notify or orchestrate something.

Context and Situation - are they synonyms?

I have spent the last couple of days in Eilat a resort town near the Red Sea, the Las Vegas of Israel, if one counts hotels, and an interesting combination of dessert and beaches in a small proximity, in the picture you can see the hotel in which we stayed. Anyway - back to context. A comment that received to the previous posting made me realize that there is a need to explain what is the difference between - "context" and "situation", both of them are semantic terms. In a way - it is similar (but not identical) to the difference between "state" and "transition". "situation" is something that happens, and that has some meaning in the consumer's terms, and it is the event consumed by the consumer(and as a result may trigger an action), in syntactic terms, situation is an event, and may be either raw or derived event (I'll not get into complex / composite events - it is sufficient to say it can be any kind of event). Context is a state -- a context can be created by the occurrence of event and destroyed by the occurrence of another event. Let's take a simple context: "during red alert". There is an event that declares the "red alert" state, and then there is an event that declares any other color, and it is considered to be "converse event" to the starting event, thus, it end the context. While the declaration of "red alert" is an event and can also be a situation, the fact that another event happened "during red alert" is not an event (semantically). Context may also be spatial - for fleet management application - a context may be a geographical area - and the events are that a vehicle enters and exits a context, but the fact that a "vehicle is within the area" is not an event.

Thus "context" has a distinct semantic existence. As far as implementation goes - there are various relations between events and contexts. After writing my first blog about context - somebody attracted my attentions that analysts have been recently talking about "context-aware delivery architecture" - I'll refer to this four letter acronym (are we out of all combinations of three letter acronyms?) in one of my next blog.

Revisiting the "classification of event processing applications - part I"

The picture about is an attempt to classify styles of beards, my first beard has been "goatee and mustache", but after a short period I moved to full beard and persists for more than 30 years, unlike people who grow and remove beards, for me it is an integral part of my personality. Anyway, talking about pictures - for some strange reason the almighty Google has chosen the dog picture I've posted in one of the previous blog as the first answer to the query about "dog" in Google image points at this blog... the wonders of search technology :-)

Now - back to the event processing business, following a comment by PatternStorm (my dear colleague Claudi), I have checked the current definition "complex event processing" in the glossary is somewhat wider than I have remembered, thus, according to it - derived events are subset of complex events, where I thought that some derived events (like enrichment of a single event) are not complex event. Anyway, I am revisiting the classification done in the previous blog, and avoiding the terms "simple event processing" and "complex event processing" - since indeed it has nothing to do with the intuitive meaning of the terms : simple and complex. However, the three classes in terms of type of processing still exists - so I'll repeat them and try different names (but names are always a problem - so any help will be appreciated !).

• raw event processing: operations on events that do not change the events - just filter and route them.
• transformed event processing: aggregation and enrichment - this is a type of processing that transform the event. Examples: enrichment it is transformed by adding attributes taken from an external store (e.g. database), translation - create an event with different format / contain translated attributes...
• Pattern-based event processing -- detecting pattern - and derive events based on the events that participate in the detected pattern.

As said, this is just one of the classifications. More discussions about other classifications - later.

event processing and business rules

New week is starting after all holidays (in Israel the working week is Sunday - Thursday), so let's start it with writing something in the Blog. I have stated that SQL is not the ideal way to express event processing functionality, so, people ask me what is the alternative - business rules ? the answer is - not really. The term business rules is somewhat overloaded, however, we can group the rule types into two types:

• knowledge-creation rules - rules that create more 'knowledge" - facts, data-elements etc... These rules can use various inference techniques - deduction, induction, abduction...
• behavioral rules - rules that cause something to be done, or prevent something from happen.

The EPL should include constructs with some similarity to these two types, but this does not say that existing rule languages can be taken as a basis, this is because the domain in event processing is events (not facts or data), and that event has a distinct semantics.

• The first type of construct in EPL is the "event derivation rules" which includes transformations (filtering, translation, aggregation, split, and enrichment) - these typically transform an event (aggregation is a special case, we'll dedicate a blog to it), and "pattern based derivation" which derives events, such that the instances of the derivers are selected as part of a detected pattern. We can see that all these types require specialized semantics, thus while there is similarity in ideas - it is not really rule semantics.
• The second is somewhat similar to the "orchestration" constructs of EPL - this is being performed in the leaf of the network - after all derivations occurred, the result event may (conditionally) perform some action related to a consumer (notify, orchestrate etc...), which is also known as ECA rules (event-condition-action). Note, that from architecture point of view, ECA rules are not done in the event processing network (i.e. they are not MEP or CEP), but they occur in the territory of the consumer - at the border of the EPN.

An interesting observation id that Complex Event Processing (CEP) as defined in the glossary, deals with derivation of new events, it is neither an instance nor an extension of ECA rules, since CEP deals with "knowledge creation" and ECA deals with behavior.

Some more discussion on this topics, and relation with "reaction rules" - later.

What is an edge in event processing network ? - a terminology wondering

Today I'll leave the macro questions about SOA and positioning of event processing discipline, and deal with a micro issue, an issue of terminology, that came up when I have looked at the current draft of the event processing glossary is the following issue: one of the main abstractions of event processing is event processing network. The network is in essence a directed (maybe cyclic) graph, as any graph it consists of nodes and edges, the definition in the glossary is: Event Processing Network (EPN): A set of event processing agents (EPAs) and a set of event channels connecting them. Clearly, EPA is a node - one can also say that producers and consumer are type of nodes (to show the entire picture), the question is what is an edge in this network -- from the definition one may assume that channel is an edge, however unlike an edge in graph, that connect two nodes, a channel may have multiple sources and multiple sinks, thus it is more comfortable to look at the channel as another type of node (and indeed channel has also some functions associated with it), thus channel is not the edge, we can say is that edge is the "pipe" through which a events flow from one node (producer, channel, agent) to another node (channel, agent, consumer). Now we need to determine what it is ? my suggestion is to call such a collection an "event stream", however, the glossary defines it as:
Event stream: a linearly ordered sequence of events. Here we have two issues - one to distinguish between the "pipe" and the collection of events flowing on this "pipe", which is the same type of ambiguity we have when talking about "event" vs. "event message", however, we can tolerate such ambiguity, the second problem is in general the collection of such events may not be totally ordered, thus it does not conform with the definition of stream.

There are two possible solutions --- one to modify the definition of stream to be more general (I am not sure that the word stream inherently mean sequence, it is just happened to be the way it was implemented in a certain academic project), the other possibility is to invent a new name for the edge - like event pipe (or some other creative name). What do you think ?

I am also putting it as a comment on the glossary website.

I am taking this opportunity to suggest that anybody who wishes to make a terminology comment will do it soon, since we would like to close a first version in the next month or so.

More terminology issues - later.

The trap of ambiguity - case of "event correlation"

One of the terms that is missing from the event processing glossary is "event correlation", yet, I don't miss this term at all, in facts, I am trying to lose it for years. The main reason is the homonym effect which the same term is used in different meaning. Here is the collection of different meanings that I know of of this term:

1. A network and system management application: following the ComputerWorld article - "Event correlation simplifies and speeds the monitoring of network events by consolidating alerts and error logs into a short, easy-to-understand package" - in this case the idea is to reduce the number of symptoms and concentrate on problems.

2. Some call the pattern detection part of CEP in the name "event correlation"

3. Some call the matching between events and data (e.g. for enrichment) as "event correlation"

4. Some are using the "correlation identifier pattern" in the enterprise integration patterns - the act of matching two events based on shared attributes.

5. Last but not least, event correlation refers to statistical correlation among the occurrence of two events, used to mine patterns of causalities.

So - what is the problem in homonyms ? the problem is that since in general there is a confusion of terms in this area, homonyms tend to intensify the confusion, especially among decision makers who don't really have technical depth, but get decision based on perceived impressions -- this may have undesired results.

Thus, I prefer to use non-ambiguous terms as much as possible, and avoid confusing slang.

This is just one example of confusing terminology, in a recent incident I noticed people spending much energy trying to understand what is the meaning of BAM (which I may discuss in another time).... The consensus glossary is not just a "nice to have", it is a business tool to save a lot of time on miscommunication.... More, later.