The Physical Web - by Google

Google recently revealed the "Physical Web" project.  This project is aimed at "interaction on demand" which will be a standard way that everybody will be able to consume data from devices connected to the Internet (AKA "Internet of Things") without the intervention of applications.   
This idea reminds of the idea of the grand challenge posed by the Event Processing Manifesto that was the result of the Dagstuhl seminar in 2010 and talked about "event fabric".     The "event fabric" challenge went further than get events on demand and also included processing event patterns on demand which I believe will be the next step to create access for everybody.  The ability to compose patterns on demand by everybody is a key to making this real-time data useful and complete the IoT revolution....  I am planned to give a talk related to this idea in early November in a workshop adjacent to the ACM Multimedia conference in Orlando... Will write more on this later...

World Stream computing to replace the Web

I came across an interesting article by David Gelernter.  Gerlenter, a Yale Professor, who co-introduced the term "lifestreams" in the 199o-ies, proceeds in the same spirit and talks about "worldstreams" which he defines as a collection of heterogeneous real-time messaging stream  (may be thought as a collection of streams).   His prophecy is that this will be a disruptive technology for the web, browsers, search, operating systems.  We will no longer surf in sites and search the web,  but we'll browse the worldstream, and ask the worldstream to bring us what we want, in the time we want it, instead of search for it.    The event processing grand challenge, which was part of the event processing manifesto,  and talked about the "event fabric" was similar in nature, but the manifesto authors did not claim that the web will be replaced,  it will probably not happen in one step anyway.    However -- certainly interesting observation, and promises bright future...

Event processing in practice

My old friend Roy Schulte from Gartner, who has been the first analyst that published analyst reports about event processing, and one of the founding fathers of the event processing community and EPTS has published a "personal blog" on the complexevents site with the provocative title:  " does anybody care about event processing?".   Roy expresses his opinion (that I've heard from him several times before) about the event processing concept and market. Roy's short answer to the question he asks is YES, but many of those who are doing event processing are not aware that they are doing event processing.   He mentions two main issues:

• Terminology:  the term event is manifested in many different ways, while the community centered around the current products have narrow interpretation
• Market:  Most of the event processing is custom code, or built into vertical products, and not the general purpose event processing platforms, thus he had to scale down his prediction for revenue that count the event processing platform market.

Roy suggests that EPTS will decouple itself from the focus on the product oriented functionality and concentrate on event processing as technology in the broad sense.   

I concur with Roy's observations (we have discussed it last year when I visited him in the Gartner office);  I have written before about the build-buy trade-off,  based on chapter 1 in the event processing manifesto, a chapter that Roy was one of its authors.      I also noted in some of the feedback to my book that I got from developers who told me that the book helped them a lot in architectural thinking and clarity of concepts, but they also said that they'll continue to develop event processing applications in custom code.  

Roy's Blog has been written in the context of the discussions on EPTS and online magazine,  and I have expressed my opinion in the same spirit as Roy --  the online magazine (and EPTS) should appeal to anybody that are doing anything with any kind of event in any form.  This should capture all the "event processing in practice" in the universe.  More news about the online magazine -- soon (the editorial board is now being created, which contains both old members of the community and new blood of practitioners).

On event server as the 21st century application server

Paul Vincent has posted in the TIBCO Blog a post entitled:  "event server as the 21st app. server".   

Paul cites TIBCO CEO Vivek Ranadivé in TIBCO's quarterly earning report, and concludes that an event server will is a requirements in many applications that process events in various ways.   

Getting to the notion of application server (see illustration below taken from an article on Websphere Application Server)

Application servers are intended to support services to applications such as:  transaction, storage, database approach, security, high availability, administration and more.     

In the event-driven world there are flowing events, and with the Internet of Things, most data in the universe will be in form of events.   In the event processing manifesto (that Paul has been one of the contributing members to its creation) we talked about "event fabric" which will enable Internet scale sharing of events and will support many applications. Some of the fabric properties mentioned were providing services of  privacy, security, interoperability among fabric instances, provenance, energy efficiency, autonomic computing support (self-tuning etc...), availability, scalability, anonymity, non-repudiation, QoS with multiple criteria.     These are some of the services, and there are of course functional services like context service, adapter and transformation service, filtering service, aggregation service, pattern matching service and more that should be built into the server and can be used by various applications from various application areas and types (BPM, CRM, Social computing, track and trace and many more).    

Paul rightly notes that standards have key role in establishing such event server,  Paul indeed wrote the standards chapter in the manifesto. 

I think that the equivalent of app server based on events is inevitable since events will be at the heart of all applications that take sensor data an input.     Work on standards in this area is an old dream, and hope that we'll be able to advance towards it.   

AITE says that vendors need to be more aggressive in marketing event processing

I have not read the recent Aite report entitled: "complex event processing - beyond capital markets",  have to check if IBM subscribes to these reports.  But from the promo page linked here I copied the figure above. It is interesting that the numbers are somewhat different among analysts, but all shows growth.   One insight given in the report is that vendors should be more aggressive about their message, since many customers still don't understand what event processing is -- a challenge both to the vendors and to the community.   One of the challenges we'll work in the EPTS level is to disseminate the event processing manifesto and ideas  more aggressively in 2012 -- stay tuned for announcement in this area. 

Briefing a USA federal committee

In this picture you can see me and a tiger during my family vacation in Thailand, in 2007.
Today I met another TIGER - the standing committee for Technology Insight-Gauge, Evaluate & Review which is a committee of the USA National Academics that works on behalf of the USA intelligence community. 

I am still unsure how they picked me up, but they invited me to participate in a meeting with the committee members and some other participants through video conference and brief them on the state of the practice, my vision for event processing in 2020, and the challenges on the way to get there.  This is an indication that event processing has caught the attention of  defense sector as well.    The vision part talked about the Event Fabric devised as the grand challenge in the Dagstuhl event processing seminar,  published as chapter 5 in the event processing manifesto (BTW - in DEBS 2011 there will be a tutorial about this grand challenge). 
Another topic was of course, the proactive world, my favorite topic.  


I was asked what is the killer application that will drive the 2020 vision, and said there is no single killer application, there are multiple of them, and I see many of them outside the corporate IT domain, such as autonomic robots and computing embedded in biological systems.   Of course, defense applications will also be part of the driving forces.    More on this topics -- later

When event processing should be used?

The event processing manifesto, chapter 1, deals with the question - why event processing should be used.
The idea is a service to the community to educate and help resolve misconceptions and confusions that exists among some people on the role and usefulness of event processing approach and platforms.


The slide above is a table copied from the document (table 1.1) which sets the main decision criteria:

• Event driven nature
• Events rate
• Application complexity
• Timeliness

In the table there are various combinations and recommendations.


The event driven nature is, of course, a major factor.  Applications with medium and high event driven nature will naturally gain more than applications with low event driven content.   


The event rate is another factor - although it determines more of the non functional requirements than the decision whether event processing should be used.   However, the higher the event rate is, the higher the benefit of using optimized framework.


The application complexity has various dimensions - I will discuss application complexity in a separate posting, but as seen in the previous posting, the higher the complexity is, there is more benefit to use event processing platforms.


Last but not least is timeliness - if timeliness is high,  then again there is a benefit of using event processing platforms, if timeliness is low and complexity is high, it depends on what exactly are the complexity characteristics of the specific applications.   


More on application complexity and no the manifesto - later. 

Event processing - manual, build or buy ?

The various chapters of the event processing manifesto, published yesterday, hold some interesting insights about the state of the practice and the future of the event processing area. Chapter 1 written by a team lead by Robert Berry has provided some insights and directions about why use event processing systems.  The figure below taken from the manifesto (figure 1.5, page 20) is talking about a frequently asked question - whether an event processing computerized system should be used at all, and if yes - should it be implemented as part of the conventional programming or using a dedicated  COTS event processing system.  

As seen in this figure - this is a function of complexity.  There are various complexity factors, defined earlier in the document:

1. Degree to which the application is expected to change over time, e.g., with new
    event sources, new interactions and new responses expected
2. Numbers and types of event sources
3. Numbers of consumers of information communicated in the events
4. State and context management
5. Opportunity to create new value, e.g., by introducing reflection and introspection





Back to the decision - in some cases the complexity is low, and there are no really processing required except for getting the events and send them to some person directly or displaying them on a dashboard.


 In this case the most cost effective solution is "manual", namely, a human is the event processor and no computerized system is required to do any processing, except for routing, which can be done by any messaging or DBMS system. 


The first vertical bar (from left) is the complexity break-even point between the "manual" approach and the "build" approach, beyond this bar it is cost-effective to construct an event processing computerized solution, but the complexity of such system is quite low, typically doing filtering, some transformations, maybe simple aggregations, typically without strict timing constraints.  In such cases learning and using a COTS system might be an overkill, and it is relatively simple to develop as part of regular programming.


The second vertical bar is the complexity break-even point between the "build" and "buy", beyond this bar, it becomes cost-effective to invest in COTS (the right one which satisfies the requirements, of course).


Note that sometimes the decision should be forward looking, if there is a plan or prediction that the complexity of the application will increase within the short and medium range, then it should also serve as consideration to avoid re-writing of the system within a relatively short period of time.


I think that we need some empirical research to determine how to measure these two break-even points in a more exact way.


Our next mission is probably to devise best practices for the community on this and other issues, and it is already proposed to have it as one of the next working items for EPTS.  

EPTS virtual symposium - March 24, 2011

Mark your calanders -- EPTS will hold its next (sixth) event processing symposium as a virtual symposium.
The idea of the virtual symposium is that a large audience will be able to participate through webcast, the symposium will also be recorded, so you'll be able to view the symposium or any part of it,   
The symposium will be co-located with the OMG Technical Meeting in Arlington, VA, USA, some of the speakers will be there in persons and the others will participate from remote.


The agenda was published on the EPTS site.    


The symposium will consist of three general themes:

1. Follow-up to the Dagstuhl seminar.    The concluding document ("event processing manifesto") in its final version will be published around the end of February, and the different teams will present their conclusions.
2. Discussion of event processing related standards:  this has been one of the manifesto chapters and there will also be presentation of Jim Odell about ideas within OMG around standards.
3. Report on several EPTS working groups activities

The EPTS business meeting and awards will be held in conjunction with DEBS 2011 in July.   

Towards an event processing manifesto - take one

The word manifesto brings to many people an association with the Communist Manifesto by Marx and Engels, which is an example how manifesto can be used, reused and abused. The term manifesto has been adopted by the computing community, and we have seen variety of works entitled: manifesto, some of them are: object oriented database manifesto, the manifesto for agile software development, and the business rules manifesto - just to name a few examples.

In May 2010 I am co-chairing (together with Mani Chandy and Rainer von Ammon) a Dagstuhl seminar, whose task is to compose "event processing manifesto", which we strive to make it as influential effort. A Dagstuhl seminar is a five days retreat in an isolated place in which a selected group of people are convened to make deep discussions about a certain area in computer science, many of these seminars are geared towards the academic community, but it this particular one we strive to keep the balance and have half of the participants from industry (vendors, independent consultants, analysts, and some innovative users), the number of attendees in each seminar in limited, and there is a big competition on this resource (and long waiting list after approved, we have put the application to do it in 2008). Here is a picture of Schloss Dagstuhl (located in Saarland, Germany):

I guess that I'll write a lot about the Dagstuhl seminar, the event processing manifesto, and related stuff in the next few months, so this is the modest starting point.

The intention of the manifesto is simple (but not easy) -- define what event processing is, what is the scope of event processing, are there mandatory features in event processing, what are the various options and variations, what are the synergies with other disciplines, and have as an appendix a call for action for the community: what are the research challenges and what are the pragmatic challenges.

I'll start today with some of my own thoughts on the scope issue.

I view "event processing" as a name of a computing discipline like "data management", "image processing" and "information retrieval" - relatively wide disciplines that cover wide collection of applications. This is also true for event processing. Alas, we still have some misconceptions even on the scope, I have used before the metaphor of blind people touching various parts of an elephant and concluding that each of them has identified what an elephant is, so this time I'll use another animal - ant whose sight is limited to its immediate environment.

I am using ant, since one of the great Hebrew poets has written a poem saying (in free translation): "my world is narrow as the world of an ant". Some examples of it:

1. people who view event processing as aggregation of time-series events, this is definitely an event processing application, but not the only one;
2. other people are looking at detection of anomalies within event collections, where the anomalies types are not predefined as what event processing is all about, this is indeed another event processing application, but not the only one;
3. yet other people are looking at real-time matching of predefined patterns with streaming events, which is somewhat different from the two previous examples.

One might claim that there is no event processing discipline at all, but there are very distinct areas that process events in different ways for different purposes, probably using different technique; while this may be a possible conclusion from the Dagstuhl meeting, my experience shows that there are many applications that require blend of event processing functionalities, and that the space is not discrete, but has some continuity, thus event processing in the wide sense should deal with all of them (and some more) and the synergies between them. So this is a point for discussion.

While the amount of people that will be involved in the actual composition of the manifesto is restricted, we'll strive to disseminate the draft to a wide discussion, both within the EPTS community and other avenues. As a comment, the Dagstuhl seminar is not directly related to EPTS, many of the attendees are not EPTS members, and some are members of adjacent communities and not the event processing community, however I think that EPTS should have a role in follow-ups.

On criteria for evaluation of event processing products

Yesterday, I spent much of the evening in Nofim elementary school, my youngest daughter Daphna, is graduating from elementary school, and they have done their grand celebration, with a lot of speeches, and performance of the children in signing, dancing and playing. It was nice, but somewhat long (the speeches part), and they have done it outside in the hottest day of the year (so far).





Another experience is that I needed to replace the door lock cylinder in my former apartment, since she is going to rent it now. I know that the Americans tend to do everything (including oil change in cars) by themselves, when I joined IBM, I was surprised to discover that I am now a "technical person", actually my current IBM title is - IBM Senior Technical Staff Member; I have never thought about software development as a technical activity, in my mind, a technical person is a person that knows how to hold a screwdriver properly, a skill I never possessed, so I looked at the Internet and called a person whose profession is to do that, we set a meeting at 6PM, and I went out in the middle of a meeting to go there, at 6:10PM I called him, he said: the person is on his way, will be there within 10-15 minutes, I waited until 6:30, and my patient started to run out, I called again -- the answer: he is almost there, give him couple of minutes. I called again in 6:40, got the same answer. My last call was in 6:48, saying -- I am cancelling this order, and will try to find another one that when he says 6:00 he really means it. Today I called another guy from the Internet, and again set a meeting with him at 6:00PM, this time I called him before leaving home, and he said -- I'll be there in 30 minutes, and indeed, he was there 25 minutes later, and changed the lock. I told him that I am happy to see that there are some professional people who arrive on time, and he said that he as also a customer hates service people who are late... I always thought on punctuality as a value, unfortunately, I am (almost) the only one in my family who has a sense of time.
Anyway, I got today some spreadsheet that an analyst made to evaluate various event processing products, the good thing is that people are trying to devise such criteria, however, looking at the criteria there, my observation is that the people devised this list came with a long laundry list of criteria, many of them seem to me irrelevant for many of the applications I know, while others that I would imagine that will be there, are left out, or treated in a shallow way. This gets me back to the difficulty of devising performance benchmarks in this area.

My observation is that the event processing area is not monolithic; people are using event processing for different reasons, and have different functional and non-functional requirements and priorities. Trying to think in a monolithic way, may yield a result that is not valid for anybody. I think that there should be deeper research in this area and provide criteria for classes of applications; this classification is not easy, as the diversity of applications that each individual organization or vendor has, is limited. This is one of the things that require a community effort, and I hope that the EPTS use case working group will be able to provide it, so I am looking forward to hear their tutorial in DEBS 2009. This also relates to our goal of establishing event processing manifesto, which will be the main theme of the event processing Dagstuhl seminar in May 2010. Stay tuned for more.

On the first step in the way to "event processing manitfesto"

It was a very busy week and alas I had to neglect the blogging hobby, now it is Friday night, I am watching a TV program with old Hebrew songs (my favorite), and decided it is a good time to blog a bit, however, our relatively new cat, who looks somewhat like this (this is not his picture, but of a similar cat I've found on the web) decided that I am a good place to rest on, and did not want to move, another creature who is trying to manage me... He is really a kitten that my daughters found and adopted, and as I have written before, giving names in our family is not an easy task, so he has several names and is known by "the cat". I call him Gilgamesh the terrible.

In 2007 we had the first Dagstuhl seminar on event processing, and we the same set of organizers (Mani Chandy, Rainer von Ammon and myself) decided to apply again for a second Dagsthul seminar in 2010, and the seminar has passed the committee, with some clarifications that we need to provide about scope. I'll let you know if and when it will be finally approved.

The intention of this Dagstuhl seminar (that lasts for 4.5 days) is to have an opportunity for a selected group of people to have a meeting in an isolated place to have in-depth discussions. The proposed goal of this Dagstuhl Seminar is to work on "event processing manifesto". There has been several manifestos of different area in the past, for example: OODB manifesto, Hopefully, by the time of the Dagstuhl seminar we'll have advanced work done by the various EPTS working groups that are being launched this month, and we'll be able to utilize their results in order to better define what "event processing" is -- note that I don't use "complex event processing", and I explained the reasons before.





One of the questions asked is what is the scope of "event processing", since working with events is quite wide area - starting from interrupt handling in operating systems, moving through graphical programming and more -- much of this is related to programming with events in conventional programming, and there are even books dealing with this area. However, our scope is more modest: generic tools for processing events in IT systems. This scope talks on what is needed to build a generic tool, and not ad-hoc programming hard-coded for every single application, and IT systems and not operating system, embedded systems etc..

The illustration above is a first step in thinking about -- what event processing system should include -- parts of it should be mandatory and some optional, however from functionality point of view there are:

• Routing and filtering -- the most basic form of event processing.
• Mediation -- transformation, enrichment, aggregation, split -- the next level of sophistication.
• Pattern Matching --- (I called it in the past "pattern detection") which may involve multiple events from multiple types.

On the bottom of the illustration there are two other entities:

Event processing platforms which are enablers for scalability, distribution and other good qualities. Event processing platforms may have their own functions or host others (or both)...

Pattern discovery that falls under the category "Intelligent Event Processing". It can be done off-line (typically this is the case) or on-line - and then the pattern matching may be unified with the discovery.

In different types of applications we may need different subsets, for example: fraud detection requires pattern discovery, security type detections (e.g. denial-of-service attack or intrusion) may use on-line pattern detection. On the other hand, other applications don't require pattern discovery at all, for example: compliance with regulations, where the regulations are given and cannot be discovered, or BAM systems in which the Key Performance Indicatros are determined according to the corportate strategy and cannot be discovered. Furthermore, there are applications in which pattern matching is not required at all, and all processing is of type filtering, routing, enrichment and aggregation.

And I'll finish with a footnote to David Luckham's recent article. David is trying to answer "critisizm on the Blogsphere" about CEP as a marketing hype, and lack of value from the current set of products. First, I never thought that there is over-hype, on the contrary, relative to the potential of event processing there is under-hype. I am re-posting this illustration taken from Brenda Michelson panel presentation in the last EPTS annual symposium.


The hype is relatively low, and in contrast, the analysts report are all indicating that the EP market has grown by 50% or so in 2008, and IDC even claims that for a second year in a raw that is the fastest growing middleware type. About the Blogsphere crtisizm, as I already written before, much of it stems from diferent interpretations of the term "complex event processing", for example, some of the postings of Tim Bass lead me the conlusion that he believes in the equation : complex event processing = on-line pattern discovery. Again, eliminating the quantification "complex", there is a large set of applications (probably most of the applications I know) of event procssing, do not require stochastic reasoning at all.

I'll post a continuation Blogs about application types, and functions they require.. It is very late - going to sleep.