On embedded intelligence within event processing application

In the previous post I have referred to the term "Intelligent Event Processing" - one question that I have asked - is this a new term ? how does it related to the other "X event processing" terms ? -- I am not sure if the term "intelligent event processing" will stick around, I would say that a better way to explain what it is may be - "embedded intelligence in event processing".

If we look at event processing architectures

- there are: producers who produce events, consumers who consume the processing results, and the EPN (Event processing Network) in the middle, which really does the processing. So where are intelligent techniques can help - here are some (real) examples:

1. In the producer - the producer has a video stream of all cars that pass below the camera, an intelligent process (using image processing techniques) isolates the license plate number of the car, and send it for further processing (security, traffic violation, billing etc..).

2. In the "meta-data" composition -- a "pattern detection" node typically looks at pre-defined patterns and attempts to detect them in run-time. In current applications the patterns are entered by the developers or users. In some cases the patterns are "moving target" like in - fraud detection -- if the patterns for fraud are discovered they are of little value, and thus in the other side of the law - people are constantly looking for new loopholes, thus, intelligent techniques, such as machine learning are used to refresh the patterns that are looked for in run-time. The run-time does not change - same pattern-detection mechanism, just different sources of where these patterns come from.

3. Intelligent nodes within the EPN -- in some cases the process of derivation of new events cannot be expressed as derivation expression and need some intelligent derivation process - e.g. a heuristic algorithm to determine the traffic light policies based on traffic events.


There are many more examples - like creating predicted events and more -- but this was more to give some flavor. Is it useful -- yes, it is useful for a variety of applications. Does every CEP application need embedded intelligence -- not really. More - later.

On Intelligent Event Processing - AAAI symposium

This is a picture of the beautiful Stanford University, I am happy to inform today that the 2009 spring symposia of AAAI has accepted our proposal to hold a symposium about "Intelligent Event Processing" that I am organizing together with two young and energetic colleagues - Nenad Stojanovic and Adrian Paschke. This will be an attempt to get the AI community involved in event processing, and help realize the vision of "intelligent event processing". How can AI help event processing: there are several ways, and all of them are reflected in the symposium description, here are the important ones in my opinion:

Event Processing Modeling - using AI techniques: advanced logics, semantic

nets etc..

Event Pattern Discovery/mining

Event Prediction

Reasoning with uncertain events

Event-based reasoning under real-time constraints

The list of topics on the site contqins some additional topics as well. As one of our missions taken (within the EPTS) is to help accelerate the advancement of the event processing area, and using intelligent techniques may help some types of applications (again -- getting back to the elephant metaphor of my past postings - just a leg, not all applications !), the dialogue with the AI community and some projects already launched is a step in that direction. We'll post "call for papers" on the epts site, soon.

On Unicorn, Professor and Infant

What is common to - unicorn, professor and infant - and how they are related to event processing anyway? -- These where my associations while reading the amusing post of my dear friend Tim Bass when he writes "On elephants and analytics" in an amusing response to my blog posting from this morning .

I am glad that Tim likes my metaphor of the big elephant, since my visit in Thailand last year, I dream about elephants all the time -). Since I am in a metaphors mood today, I'll use some more.

It seems that both Tim and myself share the opinion that the domain of event processing is much larger than the applications that exist today. In fact, I have stated several times, including in my press briefing for the EPTS launch: I believe that the big challenges are still ahead of us, since the industry barely scratched the surface of the potential use

In my previous Blog I have used the metaphor of an infant to describe this state.

Tim is constantly writing about the hype around CEP, again, in the metaphor level, some people that claim that their infant is a professor, and this of course not really true.

Indeed, there are people who over-hype CEP (a perfectly normal phenomenon, there is a hype cycle for each technology, and CEP is getting closer to the top), no surprise here.

Where I respectfully disagree with Tim, is in his claim that what has been done until today is just hype and hence totally worthless, my experience shows otherwise, there are customers which use current products and get value from using these products, and for some strange reason, the number of such customers is even growing (as a comment - calling all of them "time-series stream data engines" is a gross generalization, there is a variety of products, and not all of them are based on time-series streams, on the other hand, those who are based on time-series streams also have value). This does not say that current technologies solve all potential applications, and satisfy all potential users, but treating all the existing technology as a unicorn,

saying that everything is a hype, and there is nothing concrete, like a mythological creature, seems to me as going to extreme. Which takes me back to the elephant that both Tim and myself like While I agree with Tim that those who describe their infant as a professor may be looking at a toy elephant, the other extreme of treating an infant as a unicorn seems also to miss some part of the elephant.

Getting back to issue that start the case analytics and EP -- in my mind - to say that all EP applications require analytic tools, is like saying that one cannot get value from relational databases without using analytic tool, this assertion is certainly true for some types of application, and certainly wrong for others --- and it seems that the same hold for event processing applications. I'll let every reader to quantify the ratio based on his/her experience.

And last comment: it seems to me that there are more than one person in this community who meet customers and care about the value to customer; nobody has a monopoly on understanding all types of customers, applications and business values, and surprise - some vendors actually

believe that bringing value to custotmers is a better way to do business than selling balloons.

On EP and Analytics

Recently, some of my fellow-bloggers have occupied themselves with the question whether analytics are integral necessary part from CEP, and without it CEP does not really exist. My good friend Tim Bass went further in his current Blog and called the current state of the practice in CEP as Snake Oil , well - here I return to my previous posting with the metaphor of a group of blind people touching an elephant and each of them touches a different side of an elephant and each is confident that he knows what an elephant is, and furthermore he is the only one who knows what an elephant is. One of the benefits (there also some shortcomings...) of working in a large company is the exposure to many types of applications that are very far apart, and yet, all of them can share the same infrastructure (with some variations), talking specifically about the issue of EP and analytics - there are some applications that you cannot really think of without using analytics like "fraud detection" - where we need always to look at patterns that were unknown before, and have been used when our software has blocked the previous set of gaps, thus if the blind person touches the right back leg of the elephant in the "fraud detection" side, he sees analytics as a must. However - likewise there are plenty of other application - probably on other legs, back or trunk that do not require analytics - some simple examples: A physician sets individual alerts based on combination of test results/monitors reading - when to alert the physician or nurse; exception handling in manufacturing process - where the exceptions types are well-defined (and translated again to some combination of events); monitoring security regulation of different persons that need to be accompanied to various places within a plant according to the tag type, and checking if an autorized escorting person exists within the required distance, which monitors regulation and does not require any analytics, this is a small sample of CEP applications I have noticed recently.

Personally I think that analytics are very important, and we'll see more and more applications in which they are required, like the fraud detection one I've already mentioned, smart auditing etc... As stated before, I think that the combination of EP and some intelligent techniques (like: machine learning, prediction, handling uncertain information), which I call "Intelligent Event Processing" is very important for going forward, and we intend to reach to the AI community by doing a first Intelligent Event Processing conference as one of the AAAI spring symposia - stay tunde to note on that,

Having said that, I cannot say that this is the majority of applications, currently most CEP applications do not require analytics, here I join the opinion of Hans Glide on this issue.



This assertion returns me to the picture on the top of this blog - showing the half-full glass syndrom. Some people look at the empty half glass and some look at the full half glass. First - let's take the optimistic "full half" approach -- indications show that there are CEP products that serve as platform to build applications that bring real value to real customers, these applications are in a wide variery of areas, in variety of products, some of the type of "time series" streams, and some are not (next week I'll give a tutorial in DEBS 2008 on patterns, so this will be an opportunity to blog about types of patterns). The fact that they exist, is an indication that the EP indusrty is not just promoting "snake oil"
It is more interesting, as a scientist, to look at the empty half glass - which I think is even more than half. I think that the EP discipline is in its infancny, it already walks some steps, can say a few words, but we need to invest more until it will run, dance, sing and write Blogs... However, as every parent can testify, huge progress happens since from being a newborn until getting to the state I've described, so I don't underestimate the work done so far, and the traction achieved in the market, and think the entire community believe that we just scratched the surface of the potential. More - later.

Call for contributions - EPTS F2F meeting

The EPTS fourth event processing symposium is planned for September 17-19 in Stamford, CT.
While the exact agenda is not available yet (has dependency on contributions) here is a call for contributions in the following topics:

1. Customer Panel -- Nominate a customer to participate in customers panel about the state of the practice in event processing.
2. Use Case Session -- call for use cases, the use cases will be presented under the template that will be prepared by the use cases workgroup.
3. Vendors Business Perspective Panel -- call for business executives of vendors to participate in a panel that will discuss the business trends of event processing products.
4. Technology challenges panel --- call for CTO/senior architects of vendors/customers to participate in a panel that will discuss the technology challenges of event processing.
5. Keynote Speaker on technical vision -- nominate a keynote speaker.
6. Research projects session --- call for presentations from the research community of projects that may be of interest to the general event processing community.
7. Standards -- Call for presentations on standards related issues
8. Glossary --- Call for participation in a panel about the EPTS glossary

Nominations should be sent to : info@ep-ts.com