On EPTS and coopetition

Hello from Bedford, MA, to where I have driven earlier today from Stamford (3.5 hours with one stop in the way).

Louis Lovas from Apama is a newcomer in the EPTS meetings (though other Progress/Apama persons participated in all the meetings). He summarizes his impressions in an amusing blog posting called "a truce in the CEP Front", where he finds out surprisingly that his competitors are persons and one can sit and drink wine with them (well - Louis drank beer, if my memory does not mislead me). I think that most of us have gone through this process earlier, when I have proposed that IBM will host the first symposium, I came to marketing and explained them the idea, they did not quite get it; The dialogue on the phone was something like:

- I give a 2 minutes speech explaining the idea of community building event

- The marketing guy says: you mean conference for customers, sure we can do it.

- I answer: there may be also customers, but also others - everybody that belongs to the community.

- The marketing guy says: Ah, now I understand you, you mean conference for IBM business partners.

- I answer: we can also have business partners, but we should also have companies like TIBCO, Oracle, Progress and some others...

- The marketing guy: I don't get it, you want to invite competitors? you must be kidding...

Well - I have been just a guy from a small province (IBM Haifa) and could not convince the almighty marketing to agree to fund this symposium, thus I had to wait a little bit more and find another sponsor (the IBM Academy of Technology eventually funded it)... in the first meeting, most people did not know each other, and there were low expectations about any concrete results.

However, now in the fourth meeting most people in the meeting know each other, and from the amount of work-groups that people suggested, it seems that there is now more confidence in the usefulness of EPTS as a community, and it seems that there will be step-function in the activities.

The co-opetition idea is interesting, the picture above illustrates, of course, co-opetition in the open source (the famous penguin), there are many co-opetitions in life, for example in Israel there is a central clearing-house to transfer money between the various competing banks, there are many other examples.

In the "Event Processing" area, we have barely scratched the surface of its potential, and if we'll succeed to help better understand the foundations, establish some standards, advance the awareness to this area, and better understand various ways to gain business value -- we'll all earn from that. I think that the activities done so far has already influence on individual vendors and customers.

Of course, in the daily life, the different vendors will continue to fight on deals and customers as usual - that is the beauty of co-opetition.

BTW - Louis, congratulations for your new role as Apama's representative in the EPTS steering committee, hope that you'll be as contributor to the cooperation side of the co-opetition as your two predecessors - Mark Palmer and John Bates.

On the EPTS 4th event processing symposium

Still in Connecticut, whose flag you can see above. The three days of the 4th event processing symposium are over - what a relief, took a lot of effort to organize it - but it seems that people were happy with the content; I'll let other blogs review the meeting, some already done it - Brenda Michelson had several postings (this one, for example), Paul Vincent also wrote some of his impressions. Needless to say that I am exhausted from these days, but I still need to summarize the (relatively big!) amount of action items and send the EPTS members, may be tomorrow morning. Anyway - some short impressions:

• It seems that people feel more mature now to discuss issues like: benchmarks and languages, that they did not really want to discuss before, we have action items on both areas.



• Brenda Michelson has given an interesting presentation in the business oriented panel, showing that EP is actually under-hyped relatively to some other hypes, the panel moderator, Alan Lundberg, asked the audience if anybody think that EP (or CEP) is over-hyped, and no one raised hand. It should be noted that audience consisted not only of vendors but from analysts, consultants, academic people and customers.



• The use cases workgroup demonstrated some of the varaince we have under event processing (not sure that everything is CEP according to the glossary definition, but EPTS is about EP, not just CEP). The six has different use cases were: Alex Kozlenkov from Betfair talked about fraud detection in the betting business, although the audience did not agree that all his examples represented fraud; Arkady Godin from MITRE talked about large dissemination information (sophisticated pub/sub in the area of weather), Brian Connell from WestGlobal talked about performance monitoring of mobile operators (a monitoring system), Dieter Gawlick from Oracle talked on "first responders" in case of emergency - again a dissemination system, Guy Sharon from IBM talked about facility safety - a kind of monitoring system, and Richard Tibbetts from Streambase talked about alternative trading venues (operational system).



• Susan Urban presented a list of research challenges, and there was report about various research projects, including two presentations from IBM Research which has very active research in this area; Fred Douglis talked about "system S", while my colleague from IBM Haifa Research Lab, Guy Sharon, talked about the history of AMiT, and about the current research projects we are conducting now, the one which attracted attention is the EPDL (Event Processing Description Language) project, and it seems that we may accelerate its exposure to the larger community.



• Chris Ferris, the IBM CTO of Industry standards, explained the value of standards in general started with the chinese king who unified China, following by a standard panel - it seems that the community is now more mature to start talking about standards than in the previous years - I wonder what is the reason.



• Today in the business meeting - quite a lot of activities and working groups have been proposed. We'll see how much of it will materialize, people are more enthusiastic to commit on investing time when they are out of the office, until we return to the grey reality... but it seems that the EPTS activity is accelerating.

Some Trivia:

• We had 70 participants (the upper limit we set) - 41 vendors, and the rest - academic people, customers, analysts and consultants.

We had participants from 11 countries - see chart below (since the conference has taken place in the USA, we miss much of the European academic community).

More - Later.

On the Gartner EPS 2008

Early morning in Stamford. The "hype cycle" has become Gartner's most known artifact, as well as their constant flow of TLAs - SOA, BAM, EDA, RTE, XTP - are all Gartners' words. The Gartner EPS has ended last night, and in less than two hours we'll start the EPTS 4th event processing symposium (so Roy Schulte will be able to relax and I'll start sweating).

Some short impressions from the conference (besides the networking and meeting again old friends).

• The Gartner analysts came with new slides, but very little new insights relative to their past messages.
• Mani Chandy had interesting talk, saying that EDA is a natural continuation of SOA and not a paradigm shift (good topic to discuss). Also said that one of the big benefits of EP is - saving time for people by filtering and aggregation of flowing information.
• David Luckham has talked about "holistic event processing" as the future -- which from technology point of view means - dynamic big event processing networks. Also talked about the need to have formally defined clear semantics of event processing language (well - that is what we are trying to do in EPDL).
• Marc Adler has a great talk (in my opinion, the best one) - about his experience in developing CEP application, which seems to be a success story. He talked about criteria to select a vednor, difficulties in the development itself, and shortages in the state-of-the-art. One of his insights is that unlike what the "stream SQL" fans are saying that since people know SQL anyway, it is a good basis, he claims that while the syntax looks like SQL, it is a totally different type of thinking, and the knowledge of SQL does not help on getting it.
• Richard Brown had a talk about collecting events from text - news, blogs etc... I think that getting events from unstructured data has a lot of potential, I also talked with somebody who told me about start-up that extracts events from video cameras - the scenario of tracing behavior of people that fit the shop-lifting pattern (was discussed recently in Blogs) - may become reality soon, it seems that the technology is getting there.

That is all my time permits me to write today -- more later.

On sporadic events

I have never been a student in Stamford high school, but Stamford, CT is my home away from home for the next seven days. Starting tomorrow, I'll provide some impressions from the Gartner meeting and EPTS symposium, but I rely on other people in the blog-land to have a better coverage (e.g. Paul Vincent, with endnotes and references). I've Arrived earlier today, and resting before the busy week.

One of the thoughts that came to mind when looking at some of the discussions around the Stream-SQL standards, is one more observation.

While the claim that the difference between a "stream" and a "cloud" is that a stream is totally ordered and a cloud is partially ordered, I think that there are also some more distinctions. I'll discuss one of them --- sporadic events vs. known events. When dealing with "time series" type of input, then the timing of events are known, for each time unit (whatever it is) there is an event (or set of events) that are reported. This is true when the events are stock quotes provided periodically, or signals from sensors provide periodically. There are events that are not naturally organize themselves in time-series fashion, for example: bids for an auction, complaints from customers, irregular deposit, coffee machine failure etc...

From the point of view of functionality, there is no much difference --- one can create time series which reports a possibly empty set of events for each time-unit, but if in most time-unit the reported set is empty, this will not be very efficient way to handle it. On the other hand a system that does not support time series as a primitive can view all events as sporadic events, but there may be some optimization merit to the knowledge that events are indeed expected at every time unit. This is just one dimension -- but this leads me to reinforce my conclusion that there are various ways to provide event processing functionality, and the most efficient way is probably hybrid of approaches based on the semantics of functions and characteristics of the input. So this is the observation of today; it will be interesting to see what will be the main discussion topics in the coming conferences.

More later - from Stamford Hilton.