On event processing related standards

One of the items in the EPTS charter talked about incubating standards related to event processing.   When EPTS was established, three years ago, the general opinion was that it was still premature to deal with standards.
Recently, there have been two developments related to standards:

1. The event processing manifesto work includes a chapter that surveyed related standards and recommended some action items.   It has been presented within the EPTS virtual symposium last month by Paul Vincent.
2. OMG has renewed its call to express interest in work towards a standard on Event meta-modeling and profile  (in essence:  event structure).

Today, a vote request was sent to all EPTS members to answer two questions:  whether EPTS should establish a work-group to view standards in a strategic way, and whether EPTS should support/play a role in the OMG standard.   


Standards have a potential to give a big push to the area (this has happened in other areas);  the grand challenge presented in the event processing manifesto, the "event fabric" in an internet scale requires the existence of standards as a prerequisite,  however there need to be a shorter term business motivation to make it fly.

EPTS virtual symposium - March 24, 2011

Mark your calanders -- EPTS will hold its next (sixth) event processing symposium as a virtual symposium.
The idea of the virtual symposium is that a large audience will be able to participate through webcast, the symposium will also be recorded, so you'll be able to view the symposium or any part of it,   
The symposium will be co-located with the OMG Technical Meeting in Arlington, VA, USA, some of the speakers will be there in persons and the others will participate from remote.


The agenda was published on the EPTS site.    


The symposium will consist of three general themes:

1. Follow-up to the Dagstuhl seminar.    The concluding document ("event processing manifesto") in its final version will be published around the end of February, and the different teams will present their conclusions.
2. Discussion of event processing related standards:  this has been one of the manifesto chapters and there will also be presentation of Jim Odell about ideas within OMG around standards.
3. Report on several EPTS working groups activities

The EPTS business meeting and awards will be held in conjunction with DEBS 2011 in July.   

More on the OMG event processing symposium in the capital market area

Back home,  looking at some notes I have taken in the OMG EP meeting in NYC last week.  Here are some of these notes:

• The host was Credit Suisse, so we had an opening talk of Eric Newcomer, who told us about Credit Suisse, and also told us that his title was changed from CTO to Chief Architect, since the problem is not evaluation of technology, but coping with the complexity of the systems.  

•  Roy Schulte (Gartner) gave the first keynote talk.   He set the tone of other speakers as classifying event processing as analytics (which is the current hot buzzword),  he said (this is a slight variation about his previous classification) that event processing have three levels: the simple event processing, which we always done, the BPM + state machine which have some event-driven functionality, and the analytics event processing - AKA CEP or ESP, i.e. providing aggregations and pattern matching.  On the financial market front he said that while the original EP applications have been in the front office, it is now getting to the middle and back office in various application areas.   He also sees that the future of inter-systems integrations will be in using events.   On EP as part of analytics he also talked on the BI classification -- descriptive (BAM systems),  predictive (pattern matching is a way to define predictive) and prescriptive (which is still mostly futuristic).    There are also three ways to deliver event processing:  Basic engine,  EP platform (which includes tools for management, debug etc..) and packaged applications  (this classification is not new).

• Thomas Sulzbacher, CEO of Starview, talked about his product. He started his talk by saying that he took a flight from SFO to NYC in order to arrive into the conference, and in order to execute the flight, there were 12M  event-driven decisions required, most of them automatic. 

• David Parker, from Sybase, An SAP company, as he repeatedly reminded us, also classified event processing as part of the analytics solutions of Sybase.  The interesting insight he provided (from Sybase internal sources) is how they view the distribution of the EP market:   42% Finance Sector, 14% government, 11% services, 8% telecommunication 8% healthcare and life sciences, 6% production, 4% retail, 3% education, 2% energy and utilities.     It is interesting to see if other vendors have different perspectives, especially what analysts have to say, since no analyst has produced such a detailed distribution so far.

• Colin Clark, whom I met first time, talked and demoed his cloud event processing solution, that does indexing, clustering. classification, summarization and anomaly detection.  He came along with his business partners that made a draw among the participants that provided them business cards,  and gave some gifts -- I think it was iPad.   

These talks were all before lunch - after lunch I did not take notes anymore;  there were two talks - one by John Bates, interesting as always, given us concentration of incidents in the financial markets area, among other things, and the last talk (before mine) was Matt Meinel from Informatica (came from the acquisition of  29West).

  

Proactive computing -- keynote talk in the OMG event processing CoP conference

Hello from Philadelphia again.  Today I spent the day in NYC (got by train and returned to Philly in the evening). 
I have attended the OMG event processing CoP conference for financial markets that was held in Madison Avenue in NYC.    I'll write about the conference soon -- but for now, just a short posting about my own presentation;  I have given a keynote address (actually in the program it has been written as luminary keynote, not sure I am a luminary, but it seems that Brenda Michelson who organized the conference likes to use this term).  Anyway -- I have given in the first time a public talk about our activities in the proactive event-driven computing area,  talking also about the "first of a kind" program of IBM Research which enables to do research work in collaboration with end users.    I got several insights from the audience about various applications that I have not thought about for proactive computing, and will follow-up with some of them -- this is one of the directions to which the event processing area will evolve, and it seems to be an exciting direction to pursue, more about it - later.

New review for the EPIA book

It is holiday again, and tomorrow I am leaving for a short (2 days) vacation here in Israel with my family, and then in Sunday night, travelling to the USA for a short (4 days) business trip, where the highlight will be participation in the event processing - capital markets conference of OMG EP CoP;  here is the conference's program. I have been travelling too much recently, hope that after this trip I'll have a break in travelling.


Today I've noticed a new review of the EPIA book posted by Tushar Jain, a person I have not been familiar with so far.   Good to see that people like the book.   The reviewer is right that the conceptual model we described in the book still need to obtain acceptance like UML and BPMN,  well -- the next step is to try and work on standard proposal for event processing modeling language, and I'll take advantage of my coming trip to USA to try and kickoff this activity.
One comment -- I don't see the other books mentioned as competitive.  The book of Mani Chandy and Roy Schulte is a business oriented book, and our book is a technical oriented book, so the intersection is fairly limited, furthermore, we mention in the book that we don't deal thoroughly with the business perspective, and recommend Chandy and Schulte's book as a complimentary for those who would like to get deeper understanding of the business perspective.      Later in October I'll start teaching again a course in the Technion based on the EPIA book (first one since the book is out, though I have used the book's draft for previous course).

Announcing the EPTS awards

EPTS has announced on a competition to grant awards of two types:

The innovative application award, in cooperation with OMG Event Processing Community of Practice, will be awarded to an application that have used event processing in an innovative way.

The innovative principles award, will be awarded to a research contribution that had a significant impact on the state-of-the-art in event processing.

The competition is open to everybody. Deadline for nomination submissions is September 30, and the award will be given in November and will publicized in the media.

Full details about the award program can be found on the EPTS external website.

Next week in DEBS we'll hand out a flyer with the announcement and start communicating the award program

OMG event processing symposium - replay is available now

For those who missed the original broadcast of the event processing symposium - a replay is now available. My presentation entitled "event processing - seven years from now" (Brenda Michelson asked me to talk about "event processing in 2015", but I thought that 2017 is safer...) is now on Slideshare. I am just listening to myself now, it is quite strange to hear your own voice and realize your English accent is somewhat strange. Enjoy!

Events of the day - Darmstadt and OMG

I am now in Darmstadt, Germany, starting a long day of visiting the Technical University of Darmstadt, and giving a seminar talk at the university. In the evening I'll have to reach my next destination, Luxembourg, and this seems to be done in combination of 3 trains from here, will still need to figure it out.

Also, today you can hopefully hear me in the OMG event processing symposium along with some respectable members of the event processing community. This symposium has been reshaped as a virtual symposium, so people can tune in using cyberspace.

The event processing grand challenge - take one

In the Dagstuhl seminar on event processing that ended 10 days ago we launched the "event processing grand challenge activity". You may be familiar with the DARPA grand challenge of driving driver-less car through the desert, you can see the two winner cars in the pictures above. We are not DARPA, and don't have money to distribute, but we would like to take the opposite direction, first define the grand challenge, and then convince funding authorities that they want to support it.

Why do we need this grand challenge? The research community has incubated the "event processing" area as we know today --- some research projects in the 1990-ies, such as: David Luckham's Rapide in Stanford, Mani Chandy's Infospheres in Caltech, John Bates' Apama in Cambridge, and our own Amit project in IBM Haifa Research Lab, followed by the various stream processing projects, like Jennifer Widom's stream project in Stanford, and later the Aurora project, to name a few (there are many more, of course).

The state of the practice is now in the hands of the software vendors, and event processing is becoming part of the main-stream of enterprise computing. However, software vendors are by nature advancing technology in an incremental fashion. On the other hand, there is a strong feeling that "event processing" has barely scratched the surface of its potential to impact society, this goes beyond current applications, and even beyond enterprise computing as we know it.

It is now the role of the research community to jump-start and incubate the step-function required to be achieved in order to get this kind of impact. We would like to have a call to the research community in event processing to focus on such a grand challenge, and as an incentive (and enabler) to get funding agencies worldwide to adopt it.

A substantial brain power have been invested in Dagstuhl, and in a follow-up to work on it, there have also been some people outside the community that deal with socio-technical systems and bio-informatics.

As a metaphor we can view many type of IT systems, social systems and biological system as a "live ecology", similar to a single organism with a lot of brains, eyes, ears, hands and feet.
Event processing serves as the nervous system, and events are what flow between the different players. This subsumes the "Internet of things" vision, where many sensors are connected, and also robots of various kinds as actuators, serving as hands. Such an infrastructure will enable changing the life as we know them.

We are working now on various scenarios that will be enabled by such an infrastructure; you can hear some thoughts about it in my talk next week in the OMG Event Processing community of practice that holds its first event processing symposium. Stay tuned for much more on that topic.

On virtual vs. real in event processing

As a follow on to my last posting on virtualization of event processing, one of the frequently asked questions, and maybe somewhat open question can we get really to platform independent models that can be directly compiled to implementation of our choice in fully automatic way. This is one of the major topics I am investigating recently. Two issues are involved: Is it possible to automatically compile to a specific platform that may be semantically incompatible with the platform independent model, the second issue is whether we can create an efficient implementation without taking more platform specific considerations. The answer lies within the compilation process from the platform independent model into the specific platform. This was not really done in event processing yet, but problems of the same magnitude have been handled in other areas, and we intend to leverage the experience of people who have done it in order to see how far this can go. As a related issue, Jim Odell continued his series on event processing and agent technology, hosted in the TIBCO CEP BLOG .

Since the platform independent model we are using has "event processing agents" as its major building block, agent oriented implementation is the most natural one, and the compilation to such a model will be straightforward, however, the platform independent model should also be compilable to centralized implementations, where the agents are all hidden inside a single run-time artifact. My guess is that the shift to agent oriented implementations will happen gradually, I'll talk about it when I'll present my view on "event processing - seven years from now" in the OMG event processing consortium meeting next month. I'll write more in the future on the compilation from model to implementation as we'll get progress in that project,

OMG event processing symposium and other event processing conferences

I was recently invited to give a talk in the OMG event processing symposium that will take place in Washington DC (actually, Arlington VA) in May 24-25, 2010. I will give a talk about -- event processing seven years from now --- talking about where the event processing area goes in the future, this will be a week following the Dagstuhl seminar on event processing, and some of the conclusions may be reported there. The OMG new event processing consortium that will be kicking off in that event is another indication to the importance that the IT community places on the event processing, and the visibility it gains. This is a customers' oriented conference, whose aim is to educate the general community about event processing.

We also announced that EPTS will be doing its annual conference in November, co-located with the Gartner Enterprise Architecture Foundations Seminar, in Los Angeles. In this conference EPTS (possibly in collaboration with OMG) will grant two kind of awards: event processing innovative foundation awards (for research work) and event processing innovative application award. Stay tuned for call for nominations soon.

On optimization criteria for EP applications

This picture shows optimization of sitting on chairs, I actually know a person who sits on a big ball when he works, claiming it is good to his back. I have read with interest Paul Vincent's report on the OMG Real-Time workshop (since I cannot be everywhere, it is good that other people are reporting on what's happening, and Paul is especially good on reporting on conferences), in this meeting there has been a discussion about metrics for metrics for how to measure event processing applications. We don't have a standard benchmark yet, and I don't believe in a single benchmark fits all - but on a collection of benchmarks based on classification of applications. I would like to go deeper into the issue of "runtime performance" mentions there -- interestingly "runtime performance" means different things to different people, and indeed different application have different requirements -- if we just look at the metrics of -- latency and throughput, then we have the following variations of goal functions (this is probably not a complete list):

• min (average e2e latency)
• min (max e2e latency)
• min (variance e2e latency)
• min (deviation from time constraints)
• max (input throughput)
• max (output throughput)

The metrics are not identical - in latency there is a difference if the metrics is to minimize average latency or minimize maximal latency. For example, in Java the maximal latency can suffer from garbage collection that will make it untypically high, while "real-time Java" implementations that smooth the garbage collection minimize the maximum latency, but the price is that the average latency may grow. Throughput can be measured by input or output events, which are not really identical. Each of these goal functions indicates different kind of optimization, and this is just by looking at two parameters of throughput and latency...

This poses two interesting questions: will there be partition of the market according to optimization capabilities, or will be able to generate adaptive software that will be able to be tuned to multiple optimization ? more about performance metrics - later.

My OMG talk

The picture is of Washington DC, as seen from the Arlington side. Two weeks ago in the OMG meeting in Arlington I gave a talk on EPTS and EP standards, the talk has been now posted on the OMG site, so you can download it from there. In the next few days we'll advertise the call for participation for EPTS, towards its official launch - stay tuned.