On Semantics and Race Conditions - introduction

In this Blog posting I'll touch upon an issue that requires some attention to the exact semantics.

I'll introduce the topic today -- wait a few days to see if there are comments - and then post the analysis of this case.

Given the simple application shown below:

Let's explain this simple example, since I would like to concentrate on a single issue, I'll simplify all other things to eliminate any noise.

• There is a single event source (so no clock synchronization issues) which generates events of three types e1, e2, e3.


• Let's also say that in our story there is a single events of each type that is published (so no synonyms issues), the table shows their occurrence time (when they occurred in reality) and detection time (when they have been reported to the system) - each of them has been reported 1 time unit after its occurrence, no re-ordering problem.


• Events e1, e2 serve as an input to an EPA of type "pattern detection" which detects a temporal sequence pattern "e1 before e2", and when this is detected, it derives an event e4 - some function of e1 and e2.


• Events e3 (raw event) and e4 (derived event) serve as input to another EPA of type "pattern detection" which again detects a temporal sequences pattern "e3 before e4", if this pattern is detected - create event e5 which triggers some action in the consumer.

The question is -- given the above - will the action triggered by e5 occur?, i.e. will the pattern - "e3 before e4" will be evaluated to true.

Before getting to the analysis -- I wonder what will be the results in current EP solutions:

1. The action will always be triggered.


2. The action will never be triggered.


3. The behavior is non-deterministic (sometimes yes and sometimes no)


4. Any other possibility (specify).

Please send it as a comment to this post, I'll publish an interesting analysis of this case next week.

Happy New Year.

On the scope of event processing as a discipline again

Back home... short work week due to the Jewish New Year holiday (tomorrow is the holiday eve).

One of the topics that were not discussed in the EPTS meeting is - "what is CEP?", an indeed EPTS is looking at "Event Processing" as a discipline, where "Complex Event Processing" - no matter how it is defined, is a subset of a larger whole. One of the discussion points is to define the scope of the "event processing" discipline (some people prefer to call it "event-based systems" but we are talking about the same thing), I have already written in this Blog about event processing as a discipline before, talking about some interesting subsets.

As one interesting source, let's look at the scope of DEBS 2009:

Event-based systems are rapidly gaining importance in many application domains ranging from real time monitoring systems in production, logistics and networking to complex event processing in finance and security. The event based paradigm has gathered momentum as witnessed by current efforts in areas including publish/subscribe systems, event-driven architectures, complex event processing, business process management and modelling, Grid computing, Web services notifications, information dissemination, event stream processing, and message-oriented middleware. The various communities dealing with event based systems have made progress in different aspects of the problem. The DEBS conference attempts to bring together researchers and practitioners active in the various sub communities to share their views and reach a common understanding.
The scope of the conference covers all topics relevant to event-based computing ranging from those discussed in related disciplines (e.g., coordination, software engineering, peer-to-peer systems, Grid computing, and streaming databases), over domain-specific topics of event-based computing (e.g., workflow management systems, mobile computing, pervasive and ubiquitous computing, sensor networks, user interfaces, component integration, Web services, and embedded systems), to enterprise related topics (e.g., complex event detection, enterprise application integration, real time enterprises, and Web services notifications).

While this is not a definition that I have phrased, it shows that the discipline is diverse, and has touch points with some other disciplines (software engineering, databases, sensor networks, embedded systems etc...). It is also interesting to note that the applications presented in the EPTS use cases group were also diversified: we have seen applications from - Finance and Defense (not surprising), but also from - Media and Entertainment, Chemical and Petroleum, Telco and emergency management.

The event processing discipline crosses several aspects - modeling, architectures, languages, engineering aspects, performance and optimization, user interfaces, intelligent components, and domain-specific additions - again, all of these in the context of creating specific platforms and tools for building event processing applications. In the next few postings I'll return to some micro-level issues I have faced in the last few weeks.

Happy New Year.

event processing meets artificial intelligence

Bedford, MA, USA.

In the EPTS symposium last week, Alan Lundberg from TIBCO, who moderated the "business panel" made the analogy to AI, especially to "Experts systems", saying that there was a hype in the beginning, and people believed it will solve many of the world problems, and in the reality, it did not recover from sliding down in the hype cycle, this triggered the (somewhat surprising to some) response of Brenda Michelson, that actually EP is under-hyped, and its place in the hype-cycle is much lower in the climbing phase than the Gartner analysts draw, this is the diagram that Brenda presented with "event processing" in orange, way below SOA (in blue), BPM (in red), and Web 2.0 (in green).

Anyway - this is not the topic of today's Blog, but going back to the AI issue. The term AI is interesting, in the sense that it has spawned several disciplines (e.g. robotics, image processing, information retrieval, data mining and more) which are based on AI principles, but when they mature they stop being AI and become disciplines of their own. This is the same phenomenon we have for philosophy - the mother of all arts and sciences - many disciplines has emerged from philosophy, but when they depart, they are not considered as philosophy anymore. Event processing as a young discipline, is a descendent of multi disciplines as stated in the past, AI is certainly one of them.

What are the current topics in which AI touches event processing?

1. Modeling: the basic term "situation" and "context" have been taken from AI (situation calculus), conceptual modeling is important for design of EP applications, AI techniques can help here

2. Discovery: Prediction of events, mining of patterns - these are all derivatives of machine learning in AI.

3. Reasoning: Defining precise semantics of both event processing languages and execution models. Evidently from the recent discussions in the community, this becomes an important topic - again, precise reasoning of both the regular case of event processing, and the extended case of handling uncertain events.

As my colleague Guy Sharon described in the research session of the EPTS meeting, we in IBM Haifa Research Lab (together with some colleagues in IBM Watson Research Center) are engaged in the "Intelligent Event Processing" project that concentrates now on the discovery aspects, however, the idea is to extend the activity probably through collaborative work with the academia, as part of this collaboration we are organizing the "Intelligent Event Processing" workshop which will take place as one of the AAAI spring symposium series that will take at Stanford University, March 2009. The idea is to have the EP community meet the AI community and create partnerships to deal with these issues... so - target this conference for paper submission and/or attendance. More - later.

On EPTS and coopetition

Hello from Bedford, MA, to where I have driven earlier today from Stamford (3.5 hours with one stop in the way).

Louis Lovas from Apama is a newcomer in the EPTS meetings (though other Progress/Apama persons participated in all the meetings). He summarizes his impressions in an amusing blog posting called "a truce in the CEP Front", where he finds out surprisingly that his competitors are persons and one can sit and drink wine with them (well - Louis drank beer, if my memory does not mislead me). I think that most of us have gone through this process earlier, when I have proposed that IBM will host the first symposium, I came to marketing and explained them the idea, they did not quite get it; The dialogue on the phone was something like:

- I give a 2 minutes speech explaining the idea of community building event

- The marketing guy says: you mean conference for customers, sure we can do it.

- I answer: there may be also customers, but also others - everybody that belongs to the community.

- The marketing guy says: Ah, now I understand you, you mean conference for IBM business partners.

- I answer: we can also have business partners, but we should also have companies like TIBCO, Oracle, Progress and some others...

- The marketing guy: I don't get it, you want to invite competitors? you must be kidding...

Well - I have been just a guy from a small province (IBM Haifa) and could not convince the almighty marketing to agree to fund this symposium, thus I had to wait a little bit more and find another sponsor (the IBM Academy of Technology eventually funded it)... in the first meeting, most people did not know each other, and there were low expectations about any concrete results.

However, now in the fourth meeting most people in the meeting know each other, and from the amount of work-groups that people suggested, it seems that there is now more confidence in the usefulness of EPTS as a community, and it seems that there will be step-function in the activities.

The co-opetition idea is interesting, the picture above illustrates, of course, co-opetition in the open source (the famous penguin), there are many co-opetitions in life, for example in Israel there is a central clearing-house to transfer money between the various competing banks, there are many other examples.

In the "Event Processing" area, we have barely scratched the surface of its potential, and if we'll succeed to help better understand the foundations, establish some standards, advance the awareness to this area, and better understand various ways to gain business value -- we'll all earn from that. I think that the activities done so far has already influence on individual vendors and customers.

Of course, in the daily life, the different vendors will continue to fight on deals and customers as usual - that is the beauty of co-opetition.

BTW - Louis, congratulations for your new role as Apama's representative in the EPTS steering committee, hope that you'll be as contributor to the cooperation side of the co-opetition as your two predecessors - Mark Palmer and John Bates.

On the EPTS 4th event processing symposium

Still in Connecticut, whose flag you can see above. The three days of the 4th event processing symposium are over - what a relief, took a lot of effort to organize it - but it seems that people were happy with the content; I'll let other blogs review the meeting, some already done it - Brenda Michelson had several postings (this one, for example), Paul Vincent also wrote some of his impressions. Needless to say that I am exhausted from these days, but I still need to summarize the (relatively big!) amount of action items and send the EPTS members, may be tomorrow morning. Anyway - some short impressions:

• It seems that people feel more mature now to discuss issues like: benchmarks and languages, that they did not really want to discuss before, we have action items on both areas.



• Brenda Michelson has given an interesting presentation in the business oriented panel, showing that EP is actually under-hyped relatively to some other hypes, the panel moderator, Alan Lundberg, asked the audience if anybody think that EP (or CEP) is over-hyped, and no one raised hand. It should be noted that audience consisted not only of vendors but from analysts, consultants, academic people and customers.



• The use cases workgroup demonstrated some of the varaince we have under event processing (not sure that everything is CEP according to the glossary definition, but EPTS is about EP, not just CEP). The six has different use cases were: Alex Kozlenkov from Betfair talked about fraud detection in the betting business, although the audience did not agree that all his examples represented fraud; Arkady Godin from MITRE talked about large dissemination information (sophisticated pub/sub in the area of weather), Brian Connell from WestGlobal talked about performance monitoring of mobile operators (a monitoring system), Dieter Gawlick from Oracle talked on "first responders" in case of emergency - again a dissemination system, Guy Sharon from IBM talked about facility safety - a kind of monitoring system, and Richard Tibbetts from Streambase talked about alternative trading venues (operational system).



• Susan Urban presented a list of research challenges, and there was report about various research projects, including two presentations from IBM Research which has very active research in this area; Fred Douglis talked about "system S", while my colleague from IBM Haifa Research Lab, Guy Sharon, talked about the history of AMiT, and about the current research projects we are conducting now, the one which attracted attention is the EPDL (Event Processing Description Language) project, and it seems that we may accelerate its exposure to the larger community.



• Chris Ferris, the IBM CTO of Industry standards, explained the value of standards in general started with the chinese king who unified China, following by a standard panel - it seems that the community is now more mature to start talking about standards than in the previous years - I wonder what is the reason.



• Today in the business meeting - quite a lot of activities and working groups have been proposed. We'll see how much of it will materialize, people are more enthusiastic to commit on investing time when they are out of the office, until we return to the grey reality... but it seems that the EPTS activity is accelerating.

Some Trivia:

• We had 70 participants (the upper limit we set) - 41 vendors, and the rest - academic people, customers, analysts and consultants.

We had participants from 11 countries - see chart below (since the conference has taken place in the USA, we miss much of the European academic community).

More - Later.

On the Gartner EPS 2008

Early morning in Stamford. The "hype cycle" has become Gartner's most known artifact, as well as their constant flow of TLAs - SOA, BAM, EDA, RTE, XTP - are all Gartners' words. The Gartner EPS has ended last night, and in less than two hours we'll start the EPTS 4th event processing symposium (so Roy Schulte will be able to relax and I'll start sweating).

Some short impressions from the conference (besides the networking and meeting again old friends).

• The Gartner analysts came with new slides, but very little new insights relative to their past messages.
• Mani Chandy had interesting talk, saying that EDA is a natural continuation of SOA and not a paradigm shift (good topic to discuss). Also said that one of the big benefits of EP is - saving time for people by filtering and aggregation of flowing information.
• David Luckham has talked about "holistic event processing" as the future -- which from technology point of view means - dynamic big event processing networks. Also talked about the need to have formally defined clear semantics of event processing language (well - that is what we are trying to do in EPDL).
• Marc Adler has a great talk (in my opinion, the best one) - about his experience in developing CEP application, which seems to be a success story. He talked about criteria to select a vednor, difficulties in the development itself, and shortages in the state-of-the-art. One of his insights is that unlike what the "stream SQL" fans are saying that since people know SQL anyway, it is a good basis, he claims that while the syntax looks like SQL, it is a totally different type of thinking, and the knowledge of SQL does not help on getting it.
• Richard Brown had a talk about collecting events from text - news, blogs etc... I think that getting events from unstructured data has a lot of potential, I also talked with somebody who told me about start-up that extracts events from video cameras - the scenario of tracing behavior of people that fit the shop-lifting pattern (was discussed recently in Blogs) - may become reality soon, it seems that the technology is getting there.

That is all my time permits me to write today -- more later.

On sporadic events

I have never been a student in Stamford high school, but Stamford, CT is my home away from home for the next seven days. Starting tomorrow, I'll provide some impressions from the Gartner meeting and EPTS symposium, but I rely on other people in the blog-land to have a better coverage (e.g. Paul Vincent, with endnotes and references). I've Arrived earlier today, and resting before the busy week.

One of the thoughts that came to mind when looking at some of the discussions around the Stream-SQL standards, is one more observation.

While the claim that the difference between a "stream" and a "cloud" is that a stream is totally ordered and a cloud is partially ordered, I think that there are also some more distinctions. I'll discuss one of them --- sporadic events vs. known events. When dealing with "time series" type of input, then the timing of events are known, for each time unit (whatever it is) there is an event (or set of events) that are reported. This is true when the events are stock quotes provided periodically, or signals from sensors provide periodically. There are events that are not naturally organize themselves in time-series fashion, for example: bids for an auction, complaints from customers, irregular deposit, coffee machine failure etc...

From the point of view of functionality, there is no much difference --- one can create time series which reports a possibly empty set of events for each time-unit, but if in most time-unit the reported set is empty, this will not be very efficient way to handle it. On the other hand a system that does not support time series as a primitive can view all events as sporadic events, but there may be some optimization merit to the knowledge that events are indeed expected at every time unit. This is just one dimension -- but this leads me to reinforce my conclusion that there are various ways to provide event processing functionality, and the most efficient way is probably hybrid of approaches based on the semantics of functions and characteristics of the input. So this is the observation of today; it will be interesting to see what will be the main discussion topics in the coming conferences.

More later - from Stamford Hilton.