On the right event processing language

Recently, the discussion about the "right" event processing language has been resumed in couple of Blog entries. First Mark Tsimelzon, the energetic CTO of Coral8, claims that the most important property is that it is familiar and thus similarity to SQL is a benefit. Louis Lovas from Apama argues that familiar syntax is indeed important, but readability is even more important, and argues for imperative language - since most programming in the universe is done in Java / C style; from the comments it seems that David Luckham consistently thinks that Rapide is the right answer. This discussion reminds me of the famous story illustrated in the picture above where a collection of blind people are touching an elephant in different places, and consequently each of them describes the elephant differently. Different people have arrived to the Event Processing area from different backgrounds, and have in mind different type of usages and applications, they also may see different types of users in mind - e.g. verification engineers can work comfortably with temporal logic language, C/Java programmers may have some inclination towards script languages, business users need somewhat higher level language etc... In some applications the main natural abstraction is "stream" as a set of events in a certain time window, and most patterns are set oriented (like: aggregation, threshold, trend seeking etc...), while for other applications the main natural abstraction is "event", since events are processed individually, and the type of patterns deal with individual events (like: conjunction, disjunction, time-out etc..). The challenge with the maturing of the area is to build a language with all the right abstractions, but we need to understand them well first. A comment about the language syntax seems familiar --- yes, it has a benefit, however following this type of reasoning, we would still be in Assembly languages which was what programmers were most familiar with when I started my career as a programmer (once upon a time...), we are advancing since then to provide abstractions. SQL is by itself an abstraction over imperative languages that has started someday, I have still written imperative queries with loops in the pre-SQL era, likewise, spreadsheet programming (which started with Lotus 1-2-3) does not look familiar, but became the most pervasive programming style that exists today (for non-programmers). Thus, IMHO, abstractions that enable to think naturally about a certain type of functions is more important than familiar syntax.

On Event Proessing Platforms and Engines

Here are pictures of some engine and some platform, and they don't seem to be compatible, however if a car will replace the horse in the 2nd picture, all of a sudden, the engine (of the car) will have a role in the platform. In event processing we hear more and more about platforms, and even about event-based middleware that will be a basis for XTP and other stuff. The platform is providing some services for agents to run, like a road system that can enable to get from place to place and provide services such as: signs, traffic lights etc... The way to go there can be by foot, riding an hoarse, and driving a car, among other options. Taking this analogy further -- in the event processing world, every agent can be implemented in ad-hoc fashion (going by foot), use C/Java code with some tools to help create EP applications, or use engines, which are COTS tools that do event processing. Implementation of heterogenous agents on the same platform requires interoperability standards, if we also wish to implement it in a seamless applications we'll need language standards. There are some starts of event processing platform, quite basic at that point, but the direction seems promising, and may, at some point in time we may see event processing platforms as a basis for the new generation of enterprise application servers.

On EPTS again - calling for customers to join EPTS

After the first excitement of the launch (first press release is out, press briefing is out, some more press articles are coming soon, some of the members have referred in blogs or news, second press release is on its way) this is the time to extend the community. We have very good coverage of vendors, good coverage of academic people, some coverage of analysts. As I have written in the last blog, the target now is to add more customers. EPTS will issue a call for customers - in various places and opportunities (e.g. the Gartner EPS). What is the main motivation of customers to join:

• Impact: A lot of activities are forthcoming that will help shaping technologies, their positioning, studies about ROI, terminology, and interoperability issues. EPTS members will be able to impact all these activities - either directly (participating in working groups) or indirectly (reviewing and commenting). Some of the customers community have a lot to contribute - they give talks about EP, they blog about it, and they express opinions on forums. This will be an organized way to translate the opinions and knowledge to impact.
• Learning: EPTS will provide various community activities that will able its members to learn both on the technology and the business impact side of event processing, as well as sharing and learning best practices.
• Networking: The various conferences, the work-group conference calls, and other activities will enable active networking, besides networking facilities already exists (user groups, forums and social networks - that will all still exist).
• Help influence the establishment of "event processing" as an academic direction, impact research directions, and university level courses.

To remind you - there is no cost associated with joining EPTS, directions about joining can be found here: http://www.ep-ts.com/content/view/59/93/

Stay tuned for call for participation and contributions in the EPTS annual meeting in September 17-19, coming soon.

On the EPTS Press Briefing

When the USA president is talking with reporters he is standing on podium and facing the camera lights; yesterday I had participated in a press briefing in a much more modest setting, through the phone, sitting in the office I have borrowed for this week in IBM Hursley Lab, in UK (picture of Hursley can be found in another Blog entry ). The occasion has been - announcing EPTS, here is a transcript of my short briefing:

"Today We are announcing today EPTS which is a result of discussions between vendors, academic people analysts and customers that started two years ago in the "first event processing symposium", and continued in two subsequent meetings. One of the conclusions of these meeting was to form a permanent forum and working groups to promote the understanding of what event processing is. The list of founding members include around 30 organizational members – vendors, analysts, and customers, and 30 individual members – academic people, customers, independent consultants. You can see the list in our website. The steering committee consists of the group of initiators.

Why do we need another consortium? - I have watched event processing area evolving in the last 10 years , it is already making noticeable traction in the industry - However I believe that the big challenges are still ahead of us, since the industry barely scratched the surface of the potential use – and our mission is to help raise the level of wisdom in business decisions, business observations by correct utilization of such technologies.

Event processing is somewhat different computing paradigm and there is a need to help the customers understand this thinking and the value they can gain by using it. Hence, Our First goal is to document usage scenarios where event processing brings business benefits – most importantly, the business world needs to understand where this can be used, and the different ways it can provide benefits.

Event processing has been introduced in parallel by various vendors each using slightly different terminology, and there is a need to normalize the terminology, thus our goal is to develop a common glossary for its members and the community-at-large to use when dealing with event processing: A draft already exists and will be publicized soon.

We also believe that the academic community can help coping with the challenges, thus, one of our goals is to help establish "event processing" as a proper academic discipline by involving the research community and get them to focus on the right set of challenges.
¨ Last but not least - looking forward, help accelerate use and incubation of standards. This is similar to steps that other disciplines, e.g. databases, moved towards maturity.
¨

In conclusion – today we are announcing EPTS, a consortium of vendors, academic people, analysts and customers intended to promote the understanding and the development of the event processing area. "

The founding members (the list shown below has been sent to the reporters) represent vendors, academic people, analysts, customers, and consultants that are affiliated with event processing.

I hope that this list will grow in the future, we especially target customers that are interested to contribute to the community and gain from the collective insights to join, we shall issue soon a call for customers detail their value add from participating, the two first customers that have joined are - Betfair and Mitre.

Last but not least -- Mark Palmer has reminded me of our first meeting three years ago which took place in the lobby of the Spring Hill hotel in Tarrytown, NY. This meeting has indeed was the first step that started to roll the ball, and indeed we had very vague idea of where this ball should go, so Mark desrve much credit for his immediate enthusiasm from the vague idea.

Many people have been involved so far, and more people hopefully will be involved from now on; We should keep the drum beating, by a series of activities, with community efforts. Some such efforts are already in the way -- the glossary and the use-cases team, stay tuned for more details.

List of EPTS Founding Members:

Organizational Members

Active Endpoints
Aleri Inc.
Betfair Ltd.
Cabahu Pty Ltd
CAC Corporation
CITT GmbH
Coral8
Cordys
Elemental Links
Event Zero Pty Ltd
Forrester
Gartner
IBM
IDS Scheer
Leahy Consulting Group
MITRE
Nastel Technologies
OpenConnect Systems
Oracle
Progress Software
RuleCore
RuleML Inc.
SENACTIVE
Streambase
Streametics
Systar
TIBCO
WareLite Ltd
West Global

Individual Members

Individual Member
Alezandre Alves
Andre Bolles
Avigdor Gal
Beth Plale
Carlo Zaniolo
Christian Wolff
David Luckham
David O'Reilly
Francisco Gomez
Francois Bry
Gary Hale
Gero Muehl
Hans-Arno Jacobsen
Jeffrey Adkins
Jonas Jacobi
K. Mani Chandy
Michael Eckert
Mikael Berndtsson
Nenad Stojanovic
Pedro Bizarro
Philip Howard
Rahunandan Kandasamy
Simon Courtenage
Susan Urban
Themis Palpanas
Tim Bass
Udi Dahan

On Maturity

After a week break -- lot of work, some taking care of family health issues - I am back catching up on the community Blogs. One thing that I saw has been a debate whether CEP is not mature enough. I think that both sides have a point, depending on what "maturity" means. I'll make a distinction between: maturity of an application, maturity of a product, and maturity of a technical area.

Maturity of an application is measured in the fact that an application is working with relatively low amount of faults, and satisfy its functional and non-functional goals. Typically there is some time for stabilizing the application followed by a maturity time, and in some point the application becomes an obsolete in something, which requires to initiate the next generation.

Maturity of a software product is materialized in the fact that the product is working, being used in applications, and has relative few bugs, and customers trust it to rely on it.

Maturity of a technical area has to do with the level of understanding of this area by customers, amount of utilization of the area relative to potential, clear concepts and standard support.

In the CEP area we certainly have mature applications, we also have some maturity in the products of the first generation, but we are somewhat far from the maturity of the entire area.

On CEP agent in EPN

This logo shows that we are not alone in the world of TLAs - there are other who use this acronyms, once I have made a list of acronyms of EDA which has shown that this is not a very good choice of acronym, but an acronym these days is context-sensitive. Anyway -- talking about "event processing networks", which we now try to clean its definitions - we have talked about SEP (simple event processing) agents, MEP (mediated event processing), CEP (complex event processing) and IEP (Intelligent event processing). The names can still change, I am not happy about any of them, but it seem to be relatively intuitive, and can be explained to non-technical people quite easily. Anyway, this time I'll concentrate on what is CEP agent. CEP agent have two basic phases:

1. Detect patterns over multiple events (these can be either multiple events of the same type, or multiple events from multiple event types - some are doing distinction here, but I don't see the rationale behind it) - this creates a collection of event-instances that satisfy the patterns.

2. Derive events as a function of the collection of events created in phase 1.

Some comments:

• There can be cases in which one of the two capabilities is used in a degenerated form. The pattern can be trivial (take all events without any pattern), but still a derivation (e.g. count, calculate the average of a certain attribute etc...) is required; on the other hand - the pattern can be complex, however the derivation is trivial (just a concatenation of all participating events).
• The term for the result of phase 1 - can be called "composite event" or "complex events", the two are not exactly synonyms, according to the glossary, as composite event is a syntactic term that is created by operators from primitive events, whereas complex event is a semantic term that includes aggregation of events that may not be obtained from patterns - however, the definition of phase 1 results is consistent with both.
• A CEP agent always lives within a context (temporal, spatial, semantic) -- most common type of context is a time-window, but the concept of context is much wider.
• The semantics of a CEP agent need to be fine-tuned in order to pick the instances of events participating in patterns relative to other possibilities which require to define policies - I'll talk more about semantics fine-tuning is a later postings.
• The result of a CEP agent is one or more derived events (consistent with other types of agents).

On Event Processing Research Challenges

This picture has been taken a year ago on the stairs to the old church in Schloss Dagstuhl, in that meeting we had people both from academia and from industry discussing the state of the art and future of event processing. I have returned to the conclusion of the Dagstuhl Seminar done by my colleague Peter Niblett from IBM recently, after my return to the IBM Research Division from spending several years in the product organization, I was asked about challenges to the research community, as seen by the product development community (or the industry in general), since we had a session about it in Dagstuhl, in which people from the industry expressed their opinions about the same questions, I just had to take it as a basis of a presentation in this area.

In Dagstuhl four major areas has been put as challenge to the Research community (adding my own interpretation and comments)

1. Event Processing Algebra and Meta-Language: Like the database area in the pre-relational era, the first (and probably second) generations of event processing are "engineering based", various vendors are building implementations based on the use cases they see and their innovative ideas, bringing to the table, not only a variety of languages, but also a variety of (typically - implicit) conceptual models behind this implementation. One of the indication for a maturity of an area is the existence of an agreed upon conceptual model. The relational model has done it for databases, the browser concept has done it for the internet, now we need the same for event processing. David Luckham's concept of EPN/EPA is a possible starting point, but more work is needed on this - the challenge is still there; after constructing the "relational model" for event processing we also need the "SQL" (not necessarily SQL extension - but this is a possibility) for it - in term of meta-language that describes the model and can be mapped to various implementation.

2. Software Engineering issues: This is a challenge from a different perspective. Event processing and Event Driven Architectures impose different thinking about computing. We are programmed to think in a certain way in all programming language, and the event based programming is somewhat different paradigm. Even the basic principle of decoupled asynchronous processing is something that is not easy to digest f0r people. We need software engineering tools, methodologies and best practices.

3. Implementation optimization: If we have done the analog of SQL, another analog come to mind - database tuning and query optimization. There are many optimization issues especially when the event processing applications are distributed and may have various QOS requirements - parallel processing, acceleration of hardware, inter-operability, support in illites in general - all are challenges. The goal function in this optimization is not unique, while in some systems it is maximal throughput, in other it may be scalability in number of rules/queries/patterns (there are applications in which the throughput is measured in millions, and others in which the number of rules, the number of producers, or the number of consumers is measured in millions...), the goal function can be a combination of multiple criteria.

4. Variety of "small issues" - examples: uncertain events, out-of-order events, retention and vacuuming of events etc....

It will be interested to look at recent research project to see how much progress has been done on these, and if the list should be modified after a year that has passed. The major research event of the EP community this year is DEBS 2008 , the program has not been published yet, and it will be interesting to compare the accepted papers with this list (I'll do it when surveying the conference itself in July). We also intend to do in the EPTS event processing symposium in September a session with academic people about it - stay tuned for further notice. I am also thinking about a follow-up Dagstuhl Seminar, may be in 2009 or 2010 - to re-discuss event processing in perspective. More - later.