On disciplines and marketing devices

Yesterday I participated in the "parents teaching" program in my third daughter's junior high (8th grade) and gave the children a short introduction to the issue - does a computer think ? I did not give them an answer for this question, but gave them several basic puzzles and explained them how we can teach a computer to solved them -- one of them has been the old good missionaries and cannibals problem.



From the question --- does a computer think, I will move to the Blog of Hand Glide who phrased his posting in a form of a question -- CEP is a marketing device, so what does it say about CEP products ?

The answer is --- not much.

Let's change the TLA from CEP to SOA and ask the same question, the answer is that there are good and bad products that are marketed under the TLA of SOA, some of them have been here before SOA, and maybe some of them will be here if another TLA will dominate.

I have blogged before about the various interpretation of CEP, and the observation about what is called "CEP products" is that there is a variety of implementations that call themselves CEP, this does not teach anything about the quality of these products, their benefits to the business etc...

While TLAs became the property of marketing people to position products, somehow disciplines consist of one or two words such as: data management, image processing, graphics, information retrieval and many more - that's why I consistently use "event processing" when talking about the discipline.

Disciplines normally start in multiple places that try to solve similar (but not necessarily identical) problems, first generation of product is developed, and sometimes also hype is created and this is consistent with the "hype cycle" concept of Gartner. In the EPTS conference Brenda Michelson has argued that if anything this area is under-hyped and not over-hyped. There are some other indications that support her observation.

The early phases of a discipline lacks standard, agreed upon theory, and coherent thinking.
In the OMG meeting, March 2008, I have used the following slide as an example of what are the indications/conditions for a discipline to succeed:

The fact that EP is not in the maturity level of relational databases or some other more mature discipline is obvious, however, while there are people who made a career out of criticizing and complaining that what other people are doing is not good enough, I think that our challenge is to advance ---- it took years until there was an agreement what a relational database is, during which all databases suddenly became relational (to anybody old enough to remember, there were some funny situations of products that claim to have relational extension, when they did not understand the term), we need an event processing manifesto, and a collection of standards, but they will not be constructed in a single day, so we also need patient and persistence... I believe that EP will be 10 years from now one of the major disciplines of computing, and that we have the challenge to get there...

BTW - I agree with Hans that if products have business value for customers, they will be used regardless of the fact if at the end they will be classified EP or not. more - later

On Event Processing Networks

Back in my office, with the machine-made coffee; starting the day by reading some stuff on the Web, and first I've seen David Luckham's request to write some prediction about the CEP market in 2009. It seems that I've misplaced my crystal ball, which probably means that I am not in the prophecy business recently. While there are things that are beyond our control, I think that the approach taken by Paul Vincent to talk about challenges is more constructive.

I agree that the one of the challenges is to define the borders of the area -- like other areas that have determined clear definition of their scope -- and maybe partition to sub-types. There are other challenges of interoperability -- how connectivity to many producers and many consumers of various types can be achieved, and also interoperability between event processors that can send events to each other. I view the EPTS work groups that will be launched hopefully later this month (and those who continue from the pre-EPTS era) as vehicles for the community effort to advance in these areas: the use-cases work group in defining the various sub-types, the language analysis one in working on required functions, the interoperability analysis one on interoperability issues, meta-modeling on the modeling perspective, and of course the glossary and the reference architecture as pivots in defining terms and relationships among them. We shall not finish all work in 2009, but my challenge to the community is to achieve significant progress in all of these during 2009, and make it the year in which much of the discipline will be defined.

In addition, I have also read with interest Philip Howard's short article on "Event Processing Networks" (Below is Philip's picture on the Web)

I have received it in direct Email from David Tucker, Event Zero's CEO, and later also found it on David Luckham's site. Anybody who reads my Blog may realize that I view the EPN as the
basis of the conceptual an execution model for event processing. Anybody who reads Philip's article may infer that EPN is a new concept invented by Event Zero, and this is not really true; Though Event Zero is indeed one of the first companies to implement an EPN based solution.
The glossary defines EPN as: set of event processing agents (EPAs) and a set of event channels connecting them. The glossary definition is very general and there can be many implementations that fit this definition. One view of EPN is as a conceptual model and implement it using existing tools, another view of EPN is as an execution architecture. With the few implementations of EPN right now we see the known phenomenon of the "Babylon tower" that I have written about in the past -- each implementation chooses its own set of primitives (in this case -- agent types).


The benefits of the EPN model is in its relative simplicity, generality, and its natural support in distributed environment and parallel processing (not for free, some more wisdom is required here!). My view is that the concept of EPN should be in the center of the event processing community efforts mentioned before --- from the fundamental theory to the execution optimizations. I'll write more on that in later Blogs.

On EPTS working groups

Towards the year 2009, EPTS will increase its activities. Currently six working groups has been approved by a series of meetings of the EPTS steering committee extended with all the people who proposed working groups. We are going to issue soon a call for -- comments, vote and participation for the EPTS community.

First - something about the process of EPTS work. The main work will be done in working groups, the steering committee serves as a facilitator, but each working group has two co-leaders (as the proposals go now), and help the proposers devise the charter, make sure it makes sense, and meet the legal requirements (one of the properties of making EPTS as a formal organization is that there are some legal agreements between the members that need to be kept). The sec0nd phase which we are now entering is -- putting the working groups proposals in the EPTS site, in a members only section of the site, for comments, vote, and call for participation - each organizational and individual member can participate in any working group they are interested in. However, participation also means commitment for active participation. We shall hold a members' call to present all the proposals, and then the members will vote. Each negative vote will have to be augmented, and the proposal leader will answer - both objections and answers will be made public. After the members' vote, there will be final discussion in the steering committee, especially for a working groups that had objections, and final decision will be made.
The idea is to finish all this process in early January and launch the working groups for 2009.
EPTS members will get further instructions; the participation in the working group is restricted to EPTS members only, for legal reasons; however, everybody can become EPTS member (organizational member or individual member - see instructions in the EPTS website.

The six working groups that will be presented are:

(1). Glossary: We have issued version 1.1 of the glossary, but the work has not ended; this is a living document and a moving target, as the event processing area is in a relatively young age as a discipline. An agreed upon glossary is important to have common language, and has been successfully done in other disciplines.

(2). Use Cases: This working group continues from 2008 and had devised a template for the analysis of use cases, the idea is to survey a significant amount of use cases in order to classify event processing applications.

(3). Meta-modelling: OMG has issue RFPs for meta-modeling standards that have relations to event processing, in specific: Event Metamodel and Agent Metamodeland Profile.
EPTS still needs to determine about its status of engagement with OMG, according to it this can be either official response to the RFP, or input to OMG. In any event, EPTS has been recognized by OMG (and referenced in the RFP itself), and was asked to provide input. The working group will attempt to provide unified response of the EPTS community. Note that this is the pattern we are pursuing in general - EPTS will not become a standard development organization, but will assist existing organizations to develop EP related standards.

(4). Reference Architecture: In the early days of the pre-EPTS, there has been some work to collect and compare reference architectures of various vendors. We are now returning to deal with refernce architectures, this time in the form of an EPTS Working Group. This working group will propose one or more reference architectures for various cases (consistent with the evolving classification in the use cases workgroup and the evolving glossary).

(5). Interoperability Analysis: This working group will engage in study of requirements and mechanisms for interoperability - both between event processing products of various vendors, and between event processing products and various producer and consumers of event processing.
After the study, the working group will recommend to the EPTS community about next phases
(e.g. creation of additional standards, revision of current standards etc...).

(6). Languages Analysis: This working group will engage in study of existing event processing languages (both from products and from the literature) to devise (in a semantic level) a set of functions that is being used. After the study, the working group will recommend to the EPTS community about next phases (e.g. creation of a single language standard or creation of N variations for various languages or creation of a meta-lanaguage standard...).

I am personally will co-chair the languages analysis one (an area that I spent a lot of time on recently), and will follow, all others.

More working groups may be launched, however, I surveyed only those approved so far to continue to the next phase.

I believe that at the end of 2009 with the results of these working groups report, we'll advance the understanding of the event processing discipline, and will have a clear road-map for related standards...

Enough for now -- more later.

On the scope of event processing as a discipline again

Back home... short work week due to the Jewish New Year holiday (tomorrow is the holiday eve).

One of the topics that were not discussed in the EPTS meeting is - "what is CEP?", an indeed EPTS is looking at "Event Processing" as a discipline, where "Complex Event Processing" - no matter how it is defined, is a subset of a larger whole. One of the discussion points is to define the scope of the "event processing" discipline (some people prefer to call it "event-based systems" but we are talking about the same thing), I have already written in this Blog about event processing as a discipline before, talking about some interesting subsets.

As one interesting source, let's look at the scope of DEBS 2009:

Event-based systems are rapidly gaining importance in many application domains ranging from real time monitoring systems in production, logistics and networking to complex event processing in finance and security. The event based paradigm has gathered momentum as witnessed by current efforts in areas including publish/subscribe systems, event-driven architectures, complex event processing, business process management and modelling, Grid computing, Web services notifications, information dissemination, event stream processing, and message-oriented middleware. The various communities dealing with event based systems have made progress in different aspects of the problem. The DEBS conference attempts to bring together researchers and practitioners active in the various sub communities to share their views and reach a common understanding.
The scope of the conference covers all topics relevant to event-based computing ranging from those discussed in related disciplines (e.g., coordination, software engineering, peer-to-peer systems, Grid computing, and streaming databases), over domain-specific topics of event-based computing (e.g., workflow management systems, mobile computing, pervasive and ubiquitous computing, sensor networks, user interfaces, component integration, Web services, and embedded systems), to enterprise related topics (e.g., complex event detection, enterprise application integration, real time enterprises, and Web services notifications).

While this is not a definition that I have phrased, it shows that the discipline is diverse, and has touch points with some other disciplines (software engineering, databases, sensor networks, embedded systems etc...). It is also interesting to note that the applications presented in the EPTS use cases group were also diversified: we have seen applications from - Finance and Defense (not surprising), but also from - Media and Entertainment, Chemical and Petroleum, Telco and emergency management.

The event processing discipline crosses several aspects - modeling, architectures, languages, engineering aspects, performance and optimization, user interfaces, intelligent components, and domain-specific additions - again, all of these in the context of creating specific platforms and tools for building event processing applications. In the next few postings I'll return to some micro-level issues I have faced in the last few weeks.

Happy New Year.

On event processing as a discipline and some subsets

Every parent knows that discipline is very important from very early age (e.g. make the child realize that "sleeping time" is not negotiable), the term "academic discipline" also derived from a body of knowledge related to a certain issue that a student should learn, and indeed one of the interpretation of this word in the Webster dictionary is a field of study. In the DEBS 2008 conference, a disscussion has started whether "event processing" is a discipline by its own right, or subset of another discipline. My answer is that "event processing" is beginning its steps as a discipline, but has to go some way in order to become a full-fledged discipline. One of the interesting questions is -- since event processing applications in one or other form have existed in the past, why it did not develop until now to become a discipline, what has changed recently? my own answer is that earlier work have done ad-hoc event processing for a special purpose, while recently there is an attempt to handle "event processing" as an area systematically. To give a couple of examples:

(1). Processing of time series (see example) have existed long time ago, and also used as inspiration to those in the database community who looked at data stream management. Time series processing assumes that events arrive in fixed intervals, and typically the processing is statistical operations - like aggregation, exponential smoothing, regression, trend analysis and other stuff. The people who has dealt with this area were not interested in the more general picture of event processing (e.g. event processing when event arrive in a sporadic way and not in fixed intervals, processing event at a time, and not set etc..).

(2). Event correlation in network and system management - this has been around in the last 15-20 years (see the Computerworld article) - here again, there is a very specific sub-case, aimed to cope with "event storm" - a network or system administrator is facing a lot of events which are symptoms for problems (e.g. time-out of device can be a symptom for the fact that a router is offline) and "correlate" the symptoms to their "root cause", the problem. There is a notion of looking at patterns - but typically very limited patterns (e.g. conjunction of events over a time interval), while this has been core to system and network management, people who dealt in this area have never investigated event processing in the larger sense (e.g. looking at additional patterns), and this area has also not spawned the event processing discipline. In a previous posting about the term event correlation I have discussed the fact that while in the network and system management this term is well-defined, people sometimes use it in different contexts in an ambiguous way, and thus this is one of the confusing terms (a common misconception is that event correlation is alias to CEP) . I prefer to leave it in its network and system management meaning, and use more precise term for other interpretations.

There are more subsets of event processing, these have been just examples; the difference in the recent years is that the fact that several works both in academia and industry started to look at the bigger picture of "event processing" - what is it, what are all possible utilizations, what are the required functions, what are the non-functional requirements, what techniques have been used in other disciplines (AI, databases, control theory, distributed computing, simulation...) that can help solving this issue. This has not been done before; as said, there is a long way to go on all these topics, but this has been true for each discipline in its beginning, and I see positive momentum.

Last but not least -- I was recently asked if the discipline is called "event processing" or "complex event processing" are they aliases ? -- my own preference is to use "event processing", discipline names typically consist of two words (information retrieval, data management, artificial intelligence, image processing, computer vision...). As noted before, I accept the glossary definition of "Complex Event Processing" according to which denotes a subset of the larger event processing picture. More later -- this issue will be discussed in the EPTS 4th event processing symposium, and it will be interesting to hear the various opinions on that topic.