Thursday, June 26, 2008

On EP and Analytics

Recently, some of my fellow-bloggers have occupied themselves with the question whether analytics are integral necessary part from CEP, and without it CEP does not really exist. My good friend Tim Bass went further in his current Blog and called the current state of the practice in CEP as Snake Oil , well - here I return to my previous posting with the metaphor of a group of blind people touching an elephant and each of them touches a different side of an elephant and each is confident that he knows what an elephant is, and furthermore he is the only one who knows what an elephant is. One of the benefits (there also some shortcomings...) of working in a large company is the exposure to many types of applications that are very far apart, and yet, all of them can share the same infrastructure (with some variations), talking specifically about the issue of EP and analytics - there are some applications that you cannot really think of without using analytics like "fraud detection" - where we need always to look at patterns that were unknown before, and have been used when our software has blocked the previous set of gaps, thus if the blind person touches the right back leg of the elephant in the "fraud detection" side, he sees analytics as a must. However - likewise there are plenty of other application - probably on other legs, back or trunk that do not require analytics - some simple examples: A physician sets individual alerts based on combination of test results/monitors reading - when to alert the physician or nurse; exception handling in manufacturing process - where the exceptions types are well-defined (and translated again to some combination of events); monitoring security regulation of different persons that need to be accompanied to various places within a plant according to the tag type, and checking if an autorized escorting person exists within the required distance, which monitors regulation and does not require any analytics, this is a small sample of CEP applications I have noticed recently.

Personally I think that analytics are very important, and we'll see more and more applications in which they are required, like the fraud detection one I've already mentioned, smart auditing etc... As stated before, I think that the combination of EP and some intelligent techniques (like: machine learning, prediction, handling uncertain information), which I call "Intelligent Event Processing" is very important for going forward, and we intend to reach to the AI community by doing a first Intelligent Event Processing conference as one of the AAAI spring symposia - stay tunde to note on that,

Having said that, I cannot say that this is the majority of applications, currently most CEP applications do not require analytics, here I join the opinion of Hans Glide on this issue.

This assertion returns me to the picture on the top of this blog - showing the half-full glass syndrom. Some people look at the empty half glass and some look at the full half glass. First - let's take the optimistic "full half" approach -- indications show that there are CEP products that serve as platform to build applications that bring real value to real customers, these applications are in a wide variery of areas, in variety of products, some of the type of "time series" streams, and some are not (next week I'll give a tutorial in DEBS 2008 on patterns, so this will be an opportunity to blog about types of patterns). The fact that they exist, is an indication that the EP indusrty is not just promoting "snake oil"
It is more interesting, as a scientist, to look at the empty half glass - which I think is even more than half. I think that the EP discipline is in its infancny, it already walks some steps, can say a few words, but we need to invest more until it will run, dance, sing and write Blogs... However, as every parent can testify, huge progress happens since from being a newborn until getting to the state I've described, so I don't underestimate the work done so far, and the traction achieved in the market, and think the entire community believe that we just scratched the surface of the potential. More - later.


Richard Veryard said...

There is an important difference between your elephant and CEP. The elephant exists independently - the problem is one of perception. Whereas CEP only exists as a social construct - as the result of some extremely clever people talking about and innovating around CEP. Ultimately CEP is whatever people agree it to be.

Once a technology matures, people stop discussing what it is or what it contains; people start taking it for granted (it becomes what Bruno Latour calls a "black box") and simply concentrate on refining it and making it more powerful and efficient. I agree that EP is currently immature, maturity will come not when EP becomes more powerful (after all, adolescents can be powerful but immature) but when it becomes stable. (Some extremely useful technological innovations never reach this stage.)

Opher Etzion said...

Hello Richard. My real-life to elephant is the "set of applications for which EP technology is useful/coss-effective". While this is somewhat more abstract term relative to an elephant, it is concrete, in the sense that this set is already non-empty set, since such applications exist, but we don't know its size - maybe we think it is an elephant and it is just a rabbit, and maybe it is a hugh whale. As for stability - I agreee, note that stability has several dimensions, but this is a topic for another blog.



Anonymous said...

Well said.