For those who read the title and grinned -- not again, discussion about what is the meaning of the term CEP, relax --- I explained in a previous posting entitled: "Is my cat CEP" , why such a discussion is futile, and I am typically consistent. BTW - when I have written that posting I did not have a cat, since than my daughter has adopted one, and he does not seem to me complex.
However, I would like to answer a more interesting question that somebody asked me recently -- what are the sources of complexity in event processing ?
In high school we have learned about "complex numbers", we liked this topic, since it was one of the most simple topics in the matriculation exam in Mathematics... Complex number is just a combination of two numbers, thus the complexity is in the structure. David Luckham also coined the term "complex events", where the complexity is also in the structure. However, there are more levels of complexity that may serve as a motivation to use COTS instead of hand-coding this functionality. What type of complexities can we observe beside the structural
complexity ?
Complexity derived from uncertainty:
- The applications specification is not known a priori and has to be discovered, example: fraud detection. This is related to the "pattern discovery" I have discussed in the previous posting.
- There are no reliable sources to obtain the desired events, or the events achieved can have uncertainty associated with them. This is a distinct complexity, since there may be the case where the application specification is well defined but the events cannot be obtained, and vice versa-- the patterns are unknown, but once discovered, the required events are easily available.
Complexity derived from connectivity:
- Producer related complexities --- semantic differences among various sources, problems of time synchronization among various sources etc..
- Consumer related complexities --- similar to the producer ones, these two are, of course, orthogonal to each other, and to all other complexities.
- Interoperability complexity where various processing elements are involved.
Complexity derived from functionality:
- Complex functions requirements -- e.g. complex patterns that may involve temporal, spatial, statistical operators and combinations of them.
- Complex topology of the event processing graph, with a lot of dependencies among the various agents, which creates a complexity in validation and control.
- Complex subscription / routing decisions.
Complexity derived from quantities
- High throughput of input events.
- High throughput of output events.
- High number of producers
- High number of consumers
- High number of event processing agents (imagine 1M agents in a single application)
- Requirement to maintain high amount of space for processing state.
Complexity derived from quality of service requirements:
- Hard real-time latency constraints.
- Compliance with QOS measurements such as threshold on average latency, threshold on percentage of events that don't comply with some latency constraint etc...
- High availability requirements.
Complexity derived from agility requirements
- Dynamic, frequent changes in the logic of the event processing
- Need for programming by various types of "semi-technical" people among the business users community...
I am sure that this list is not complete, but it provides some indication...
Of course, a single application may be the ultimate complex application of event processing and need ALL of these complexities, finding this application is, for sure, the dream of every researcher --- getting a lifetime of research challenges, but in reality different applications have different combinations of complexities. An application can be simple in all metrics, but have hard real time constraints, it can have very complex functionality, but no quality of service, or quantities issues. Another applications may need pattern discovery, but again the rest is simple, another combination can be relatively simple application, with complexity in quantity of producers and consumers and in semantic integration with all of them, and with the wonder of combinatorics, one can get to many more combinations....
More on complexities - later.
This drawing, in various forms, has been used by us for many years to illustrate the notion of pattern, actually in the PowerPoint version it is animated, and the geometric shapes are keep moving. The term pattern is a bit overloaded in event processing, as noted my DEBS 2008 tutorial on this topic, but this illustration refers to the pattern which shows some combination of events, to be more accurate it is a predicate on the event history that if evaluated to the value of
"true" something should happen. This illustration was created by Tali Yazkar-Haham from IBM Haifa Research Lab as an exercise in a presentation course, and was used in dozens of presentations ever since (including presentations of some people outside IBM who typically forgot to give credit to the source).
Paul Vincent, in a continuous debate with Tim Bass, on the complex events forum, has written about "detection of new instance" and "detection of new type". While these terms make sense, I prefer not to overload the term detection and use the terms event pattern detection and event pattern discovery.
Pattern detection deals with detection that a predefined pattern has happened. This is what illustrated in the picture above. Some example can be: a patient is hooked up to a heartbeat monitor, and the physician is pre-setting a pattern "the heartbeat is monotonically increasing within 10 minutes, and the amount of increase is more than 30 during that period". This is actually a predicate over a part of the event history of a single source and type (other examples can involve multiple sources and types, but the principle is the same).
So event pattern detection is defined as detection that a predefined patterns has occurred.
This is equivalent to what Paul called: Pattern instance Detection.
In contrast, when we talk about event pattern discovery we mean that the pattern is not known in advance, and the pattern discovery function determines what is the pattern. The legend says that Archimedes discovered his famous laws about floating bodies when sitting in the bathtub and shouted: Eureka (this illustration was taken from the homepage of a company who has the word Eureka in its name, again animated in the source). 
A pattern can be discovered by machine learning techniques using decision trees, statistical modeling, Bayesian Networks and numerous other methods. At the end when a pattern is discovered then it also need to be detected in reality; there are also cases in which there is a continuous detection since the patterns are changing after a short time.
Getting back to the previous example about the heartbeat, it may be the case that this pattern has bot been set by a physician, instead it was detected by some method that has looked at past events and found out that this pattern has some significance.
Most people thinking about "complex event processing" are actually talking about pattern detection, regardless of whether the patterns were composed by a human or discovered by machine learning. The illustration at the top of this page illustrates what people typically mean. As stated in the past, I don't want to get into the meaning of TLAs, and leave it to my colleagues who are doing marketing. Thus the term that I used "event pattern detection" is the more accurate one.
Another observation about the difference is that event pattern detection can be applied as COTS -- a user can use such a product, compose some patterns, hook it up to event sources, and get the pattern to be detected.
On the other hand --- while there are many tools that can help in the event pattern discovery, we cannot hook it up to event source and tell it: discover all. There is a need to do some formal modeling of the system, kind of patterns that are sought etc... In other words, this is not something that a typical developer or business analyst can do, since it requires some expertise,
It is getting late - so I'll finish at this point and return to this topic at some later point.
This is past event that was held in Waterloo almost two years ago and dealt with "quilt hanging".
One of the interpretations of the term "event" in The Webster dictionary online is : something that happens. This one has been used by the EPTS glossary editors. But there are also some other interpretations, one of them is: social occasion or activity
Today in a meeting with some students they have attracted my attention to an interesting set of products that are processing events, however, the event they process are of the second type and not the first type. There are several companies that have products in this area, and the one name I remember is Eventful.
I Mention it just as an example, as I have not investigated this area, I understand that there are some others as well.
As a food for thought -- what is the difference between the two types of "event processing" ?
First Try:
Products in the first interpretation are typically geared towards the enterprise market, an event typically is assumed to occur within a single time-point (actually an interval reduced to a time-point), and the reason for doing it is typically one of the list that I have mentioned in a previous posting.
Products in the second interpretation are typically geared towards the consumer market through the Web, thus the business model is totally different; an event typically occurs in an interval, the participants in the events are often at the center, and the processing related to them. The motivations related to person's free time, and sometimes to social or other collaboration.
An interesting question is whether from technology point of view these two interpretations are totally distinct or have some commonality (e.g. both have producers, consumers, processing, events, subscription etc..), is this commonality interesting enough to try and generalize them, and whether there is any good reason to inter operate and mix event of the two types...
This picture, taken from the site of Nature Reviews as part of an article about "parallel processing in mammalian retina",
illustrates that structures like the human body distribute the functions it needs to perform, and performs many of them in parallel and by specialized systems.
Getting to event processing, the producers and consumers of event processing can be distributed, as events can come to many sources, and situations may be consumed by many sinks. The first generation of event processing was mostly centralized in processing, the centralization has been twofold: functional centralization using an monolithic engine that performs all processing fucntions, and location centralization, this engine runs on a single server.
Today I'll concentrate on the second aspect of centralization, there are various reasons to decentralize the processing, one is to do some of the activities closer to the producers or consumers, example: if a producer produces events, where only 1% is relevant to the defined event processing, and it can be done by independent filtering that does not depend on other events, then it will be more efficient that the filtering will take place at or close to the consumer site, and thus eliminate the unnecessary network traffic.
Another reason to distribute the functionality is the scalability aspect, which is really an old idea to "divide and conquer" problems. The challenge is how to do a "good" partition. First there is a need to define what a "good" partition is, i.e. looking at it as an optimization problem, what is the goal function, then solving it is a function of the topology, semantics and behavior of a particular application which can be dynamic.
IBM has recently released the first version of WBEXS (Websphere Business Event Exterme Scale) and made a statement of direction for another product: Infostreme Streams
both are aimed to handle scalability by distribution in different environments. While details about IBM products you can obtain from the appropriate people in IBM, we in the IBM Haifa Research lab are working on related topics, we have exposed initial results in DEBS 2008, in the fast abstract session introducing the statification approach.The project has substantially advanced since that time, and I'll discuss it further in future Blogs (well - I need to go over the Blog and list all the topics I promised to discuss later and have not done so yet...).
More - later
Recently, the acquisition of ILOG by the blue giant (IBM, the company who pays my salary) has been completed.
It was interesting to read the InformationWeek article which had event processing in the title of this article.
ILOG is, of course, well known for its business rules product ( I had some history in this area in my past...).
There are various relationships between business rules and event processing technologies, and there are also various opinions about the various relationships.... I have blogged a year ago about this issue.
While I looked at business rules from EP perspective, Some ILOG guys also investigated the opposite direction, in the last EPTS meeting we had met Pierre-Henry Clouin, who blogged about it , prior to the EPTS meeting. Now we can continue the short discussion we had in Stamford .
Another ILOG guy who had a guest posting about EP issues is Changhai Ke.
Joining a big corporate is certainly a cultural change (shock for some), but also opens some horizons, so I wish Pierre-Henry, Changhai and the rest of the ILOG team a smooth transition and we'll certainly meet in one of the blue corridors.... Welcome on board.
Today's experience relates to the remote control which opens the doors of my car (well - mine does not have a panic button), for some strange reason I have noticed that it resides on the floor of my office decomposed to basic components, one screw has broken or something... I picked up all parts, held them tightly together and went straight to the contact person of the leasing company in IBM Haifa Labs, he looked at the broken device and said -- these days we are not allowed to replace them if they work, so he took the following working tools (see below) - 
and now I have a remote control with cello tape around it, it turns out that the atmosphere of "reducing expenses" in our leasing company gets to the tiny details.... I also heard that some almost forgotten professions like shoemaker who knows how to fix shoes got alive these days...
Anyway today I am in reactive mode -- since several of the recent Blogs deserve some footnotes:
A footnote to the Blog with the provocative title: SOA is dead; long live services
by Anne Thomas Mannes (along with the many other responses)- I once heard a good talk about the many interpretations of SOA, which means different things to different persons, so it may be dead for some, and alive for others... anyway, good ideas live more than marketing TLAs that come and go with fashions. As an analog, The TLA CEP may survive or change with other fashionable term that will have some other blend of technologies, however, the more interesting thing is not the marketing term, but the substance behind it. If it solid and have value to customers, it will survive and prosper, and I believe that event processing (as a discipline - see: my previous posting on EP as a discipline) is one.
A footnote to Mark Palmer's Blog -
Speaking on event processing - it was interesting to read Streambase's report by Mark Palmer stating that Q4 was the best quarter ever for Streambase, which is a pure play event processing vendor. While this may or may not be an indication for a more general phenomenon, as I've started this posting, everybody is trying to reduce costs these days, and one of the ways to do it , automate processes that are event-driven by nature (e.g.automated exception detection and handling), getting alerts on cases where there is a potential expenses leakage (auditing), compliance with regulations that corporates see as "tax" on their operations, and would like to invest as little resources as they can, straight through processing and other activities associated with cost reduction are event-driven in nature, and thus can benefit from use event processing technologies, thus, the positive correlation between troubled times and growth in the use of event processing software may not be surprising, again, this may or may not explain Streambase's report, I am not familiar with the details.
A foontnote to Marc Adler's Blog - Marc cites a study about the influence of Blogs on purchasing. I am amazed every time to see the power of Blogs... from my personal experience, I am getting a lot of private communication based on my Blog, including some surprising offers, will write about it one day; one the RFPs that I got from our sales team to advice on, was traced by them to be copied from one of the area Blogs... customers started to see Blogs as authority, and this can be of course dangerous since not everybody who Blogs about something is really an authority on the area he or she Blogs on and get into the trap --- that's all for now -more alter.
Yesterday I participated in the "parents teaching" program in my third daughter's junior high (8th grade) and gave the children a short introduction to the issue - does a computer think ? I did not give them an answer for this question, but gave them several basic puzzles and explained them how we can teach a computer to solved them -- one of them has been the old good missionaries and cannibals problem. 
From the question --- does a computer think, I will move to the Blog of Hand Glide who phrased his posting in a form of a question -- CEP is a marketing device, so what does it say about CEP products ?The answer is --- not much.
Let's change the TLA from CEP to SOA and ask the same question, the answer is that there are good and bad products that are marketed under the TLA of SOA, some of them have been here before SOA, and maybe some of them will be here if another TLA will dominate.
I have blogged before about the various interpretation of CEP, and the observation about what is called "CEP products" is that there is a variety of implementations that call themselves CEP, this does not teach anything about the quality of these products, their benefits to the business etc...
While TLAs became the property of marketing people to position products, somehow disciplines consist of one or two words such as: data management, image processing, graphics, information retrieval and many more - that's why I consistently use "event processing" when talking about the discipline.
Disciplines normally start in multiple places that try to solve similar (but not necessarily identical) problems, first generation of product is developed, and sometimes also hype is created and this is consistent with the "hype cycle" concept of Gartner. In the EPTS conference Brenda Michelson has argued that if anything this area is under-hyped and not over-hyped. There are some other indications that support her observation.
The early phases of a discipline lacks standard, agreed upon theory, and coherent thinking.
In the OMG meeting, March 2008, I have used the following slide as an example of what are the indications/conditions for a discipline to succeed:
The fact that EP is not in the maturity level of relational databases or some other more mature discipline is obvious, however, while there are people who made a career out of criticizing and complaining that what other people are doing is not good enough, I think that our challenge is to advance ---- it took years until there was an agreement what a relational database is, during which all databases suddenly became relational (to anybody old enough to remember, there were some funny situations of products that claim to have relational extension, when they did not understand the term), we need an event processing manifesto, and a collection of standards, but they will not be constructed in a single day, so we also need patient and persistence... I believe that EP will be 10 years from now one of the major disciplines of computing, and that we have the challenge to get there...
BTW - I agree with Hans that if products have business value for customers, they will be used regardless of the fact if at the end they will be classified EP or not. more - later
