Stratification is one of the terms that computer scientists borrowed from Geology. In 2007, Ayelet Biger, my former M.Sc. student has done her thesis about Complex Event Processing Scalability by Partition which looked at semantic partition of an EPN graph to strata, where in each stratum all agents are independent and can run safely in parallel. Today we have reported about 2008 research project that took the stratification idea and developed a system to assign agents to machines in a distributed environment. Geetika Lakshmanan delivered the talk about this project today. I have posted it on Slidshare. Enjoy !
This project is an interesting example for a life-cycle of a project:
- It started as an academic thesis;
- It has flown to a research project within IBM Haifa Research Lab ;
- After showing promising results in the lab, it evolved to a more "down to earth" project that deals with assignment of agents to machines and threads within the IBM product - WBE-XS (Websphere Business Events - Extreme Scale) which enable event processing on grid environment. This project has to take into account products and their implementation, deal with product instrumentation, and other stuff that pure research projects do not deal with.
While starting an idea in the academia and then going to a start-up has its magic, the work in IBM Research enables to get stuff from academic projects until impact through products, and using my hat as an adjunct professor in the Technion, this is possible. However -- as noted before, getting things pushed in a big company are not necessarily easy.
In my role as the industrial track chair of DEBS 2009, I have been asked to invite an industrial keynote speaker, and this year I chose John Bates, the General Manager of the Apama Division of Progress Software, and a person that made a transition from being a faculty member in Cambridge University, leading one of the first start-ups in this area, and now managing a division in a larger company. This was certainly a good choice; John outlined the assumptions he had as a researcher, and the confrontation between these assumptions and the reality that forced them to find a low hanging fruit, until the bigger market will be mature enough for the ideas for event processing, and they found trading in capital marketing as this low hanging fruit, and the early adopters of such technologies. However, the maturity in thinking about this area now takes it to more areas. As for John's prediction for what will happen in the event processing market in the next five years, he mentioned the following points:
- Event Processing will be used for tracking everything -- car, plane, bag, package, ship
- EDA -- will achieve wider adoption -- federating services in the enterprise, providing agile enterprise nervous system; The next wave of SOA.
- Event processing everywhere -- EP in the cloud.
- Event Processing will be part of bigger event-driven business process management market - BPM, rules, event processing are going to merge.
- Event processing will be embedded in many vertical applications.
More about further sessions in DEBS 2009 - later
This is the Vanderbilt University Law School, where the DEBS conference is being held this year. Today I spent the day in a U-shaped seminar room, with a lot of ex-Deans of the school hanging on the wall. The first day has been the tutorials day. In the first half of the day I attended the "use cases" tutorial in which the EPTS use cases workgroup, chaired by Pedro Bizarro and Dieter Gawlick, who, together with some more members of this workgroup presented some use cases, together with lessons learned. Paul Vincent has written a review (with some criticism) about that tutorial. I wonder how many bearded academics he counted in the conference -- indeed, once in academic conferences one could see all the prominent scientists carrying a beard, but today, most of them are shaving their entire face... Dieter who presented some of this tutorial has a beard, but he is from Oracle and not from the academia. Anyway, the idea to have an EPTS award (donated by TIBCO) to the application of a year is a good one, I'll bring it to the EPTS steering committee. We also discussed (after the end of the program) topics related to the use cases workgroup and relations with other EPTS workgroups.
In the second half of the day, Jon Riecke, Adrian Paschke and myself have presented the "event processing languages" tutorial. I am not reviewing my own talks, so instead -- you can look at the slides and review it yourself. More about DEBS 2009 -- tomorrow.
Early morning in Nashville. I arrived here yesterday, and after some rest, went to dinner with some of the people in the community (this was initiated by Matthew Cooper from Eventzero who invited some people). In the dinner I was sitting next to Dieter Gawlick from Oracle, one of the most active people in the community. Dieter is recently advocating the concept of "state based event processing", based on some work he is doing. Today I'll hear more about it in the use cases tutorial, and then in a later meeting with Dieter to get feedback from the use cases workgroup to the language analysis and reference architecture work groups. My initial impression is that in the term "state" he more or less means what we call "context". I have just written the first draft of the chapter in EPIA book defining and describing the term context (will be available on the Web within a few weeks I hope). It will be interesting to see if how the notion of context will further evolve. Most event processing languages have very modest support in context these days (e.g. time window as a simple case of temporal context). I'll write again about contexts and states in one of the next postings. Today -- the DEBS conference start, and the tutorial day (last year it was the most interesting day of the conference, let's see if we'll keep the standard high). Now -- going over my part in the event processing language tutorial, so I'll not mix up the English words...
Tomorrow night I am planned to start the long way to Nashville, to attend the DEBS 2009 conference. DEBS is originally a conference founded by the distributed computing guys, what is known as the "pub/sub" community. Two years ago, Mani Chandy and myself came to the DEBS steering committee and challenged them to extend the conference such as it will be a general event processing conference, and work with them to make it the scientific "flagship" conference of event processing. Looking at the DEBS technical program, the first day (Monday) is the tutorials day. It will have six tutorials, five of them are about event processing, and two out of the five were created by EPTS work groups: the use case work and the language analysis work group. Giving part of this tutorial (the other parts will be given by Jon Riecke and Adrian Paschke) will be my first role in this conference. We shall make these tutorials available to the public after the presentation. The second day (Tuesday) will start with a keynote address by John Bates, Apama GM, and one of the pioneers of the event processing area. He will be the industrial keynote speakers, and as chair of the industry track, I will have the honor of introducing him, my second role in this conference. The first research session on distributed event processing will have a paper which I co-authored on stratified approach for supporting high throughput event processing applications, however, I will leave the presentation to the younger generation (Geetika Lakshmanan from IBM Watson Research Center will give this talk). The second research session will be about pub/sub, but one of the talks has the event processing phrase in the name. Then we'll have the industrial session, and my third role will be to chair it. This session will consist of four industry report, and industrial panel about academic-industry collaboration in event processing. We plan to have four panelists, two from academia, one customer, and one vendor, and discussion with participation of the audience. DEBS originally was almost purely academic conference, but we have injected some the industrial participation, and believe this partnership is important for both sides, and more important, for the future of the event processing area. In the DEBS business meeting, later that evening, I plan to propose hosting DEBS 2011 in Israel. The third day, Wednesday, will have a keynote talk of Alex Buchmann, an old colleague from the active database era, who is one of the DEBS founders. The research session of that day will deal with complex event processing, and in the afternoon - posters, demos and fast abstracts. The fast abstract session is now a trend in conference to enable people to have short reports about work in progress, not mature enough to have full papers. My fourth role in the conference will be to give a short talk about a work in progress we are doing in the area of geopsatial and spatiotemporal extensions for event processing language.
The last day (Thursday) will feature another keynote talk by Karsten Schwan, and three research sessions about variety of topics. I'll have to skip the last day, since I'll have to go for some meetings in the NY area on Thursday and Friday. I'll report more about DEBS 2009 later, looking forward to meet a lot of colleagues from the developing community.
Yesterday, Peter Niblett and myself, the co-authors of the EPIA book, had a meeting with the publisher and editor in order to summarize the review process of the first 1/3 of the book. One of the topics discussed was the ability of readers to have "hands-0n" experience.
As I explained in a previous posting, our approach is to teach our view about what is event processing and not teaching a specific language. We spent much of the review around this issue, the publisher suggested that for the benefits of readers who would like to have hands-on experience while reading the book we should provide a way to do it. What was agreed is that we'll use the use case that accompanies the book - the fast flower delivery --- to be a bridge between the book and the hands on experience. Since we would like to expose the reader to the multiple approaches that exists in the event processing area -- we'll provide the reader several alternatives to experiment, each alternative will be a language from different type -- it will be emphasized that the languages are just representatives, and the book does not endorse any of them in particular.
For each of the languages we shall provide:
- The fast flower delivery example coded in this language.
- Link to a site from which the reader will be able to download an implementation of this language (e.g. trial version of the product implementing this language) on which the reader will be able to run the example and experience in using that language.
Of course, this should be available to the readers without payment. In the next few days we shall talk with some of the language owners to select those who will be able to support these requirements.
I don't know how many readers will indeed spend time to take advantage of this option, but since 2 reviewers (out of 14) mentioned their wish to have "hands-on experience", I guess that there is certain segmentof the potential readers who will do.
Actually, I am a big fan of learning by hands on experience, my own experience from my long years as student is that courses in which I had substantial programming assignments are the course in which the knowledge gained in these courses is retained over time. I'll tell about my experience as a teacher on this issue another time. Stay tuned, this is going to be fun !
Today, again, I did not celebrate my birthday. I don't have a habit to celebrate a reminder to the fact that I am getting older. Interestingly, I got today much more (relative to recent years) birthday greetings in various communication ways -- E-Card through the Internet, phone calls, Email and even Real-time messaging. The reason that I got much more greetings relative to previous year can be attributed to the fact that I made many friends in the last year, but the more realistic reason is that certain social networks send people reminders about birthdays of other people in the network, which makes the knowledge about the birthday more accessible. Well -- none of my family remembered my birthday, here, at least, no change from previous years.
One of the many meetings I had last week was a teleconference with some IBM customers (I shall not expose their identity), my role in IBM is not in sales or sale support, but I am being called from time to time to participate in meetings with customers, by request of the people handling such meetings. The major issue that they wanted me to discuss is the benefit of using a middleware software for event processing (in the large sense) vs. hard coding it into the application code itself, like this customer is used to do.
Indeed, this is not a clear cut issue, there are two cases in which hard coding the event processing functionality make sense: either in the case that it is very simple, and thus it is not cost-effective to purchase, learn and deploy a specific product, or in the case that the functionality is too specific, and not covered by products, furthermore, it does not represent a ubiquitous requirement that makes it cost-effective for vendors to support it.
There are also some cases in which it is reasonable to write some functions in Assembly languages (even I did some of this in the past), but typically (most) people prefer to code in higher level language.
Likewise, in many applications, satisfy none of these conditions, and for these application is cost-effective to use generic software. The reason is that there are various common functions (e.g. filtering, routing, enrichment, transformation, pattern matching) that are repeating. Hard coding them means -- re-inventing wheels over and over again, instead of reusing existing implementations as "services", and enjoy other people's work that is being upgraded and optimized with time. This is similar to the reasoning of using other generic products -- messaging, workflows, databases, adapters, development environments and others.
The reaction of this particular customer was interesting: "what you are saying makes much sense, currently we are not used to think in terms of separating the event processing from the rest of the application logic, and we need to digest the idea". So, there will be a follow-up meeting to continue discuss it. I have seen this kind of reaction before, I think that a challenge of event processing vendors may be the competition with potential customers who do not understand the benefits above hard coding. But, this was also the case for other technologies that succeeded to cross this bridge. The growing number of event processing customers indicate that this thought is getting traction.
I think that this is also a topic for a community effort, that may be pursued by EPTS. More on this topic -- later.
Stratification is one of the terms that computer scientists borrowed from Geology. In 2007, Ayelet Biger, my former M.Sc. student has done her thesis about Complex Event Processing Scalability by Partition which looked at semantic partition of an EPN graph to strata, where in each stratum all agents are independent and can run safely in parallel. Today we have reported about 2008 research project that took the stratification idea and developed a system to assign agents to machines in a distributed environment. Geetika Lakshmanan delivered the talk about this project today. I have posted it on Slidshare. Enjoy ! 
