Well - I did not come to Las Vegas just to enjoy the time, I have given two (identical) talks together with my colleague Kyle Brown, an IBM Distinguished Engineer, who belongs to the Websphere Services organizations. I talked about technology, Kyle introduced related application, fun staff. I have posted the presentation on slideshare. Enjoy.
This is a blog describing some thoughts about issues related to event processing and thoughts related to my current role. It is written by Opher Etzion and reflects the author's own opinions
Saturday, May 9, 2009
Beyond the horizon
Well - I did not come to Las Vegas just to enjoy the time, I have given two (identical) talks together with my colleague Kyle Brown, an IBM Distinguished Engineer, who belongs to the Websphere Services organizations. I talked about technology, Kyle introduced related application, fun staff. I have posted the presentation on slideshare. Enjoy.
Friday, May 8, 2009
On Smarter Planet

Still in Las Vegas. The icon above is the logo of IBM smarter planet, which is the main theme of the IMPACT conference. According to IBM, the smarter planet is instrumented, interconnected and intelligent. Event Processing is considered in all talks a key enabler. In the conference there has been a distribution of a mini-book, which includes the first few chapters from the upcoming book of Mani Chandy and Roy Schulte. Sandy Carter, IBM VP for Websphere marketing, has written a preface to this mini book, here is a quote: We at IBM are excited about event processing. As the planet becomes increasingly instrumented, interconnected , and intelligent, event processing is emerging as a key technology for mitigating risk, seizing opportunities, and achieving greater corporate agility. Smart cities, smart healthcare, some retail... these are just a few areas where event processing is adding more intelligent technology to our life. Under the smarter planet imitative there are vertical solutions, and today I went to listen to a talk by Honda Italy about smart manufacturing. Interesting stuff, more - later
Tuesday, May 5, 2009
On IMPACT 2009 - the first day

Las Vegas. I have arrived here yesterday to participate and give a talk (twice) in the IMPACT conference, IMPACT is the annual conference that IBM is holding for its customers, especially the Websphere brand, but also covers anything that is defined under the "SOA portfolio". James Taylor is reporting about it, and is doing much better work than I can do, so trace his Blog for more details, however I wanted to bring my personal angle into it. The first day started in a plenary session with a huge, besides a very funny performance of Billy Crystal, who put three IBM executives on the podium to do sound effects to a story he told, there have been presentations of the senior executives of IBM Software Group, Steve Mills and Tom Rosamillia on the main theme of this year's conference: Smarter planet. When I heard the presentation I could not help smiling to myself, practically all customer examples that have been described under "smarter planet" have included event processing component in a key role. One of the customer examples has been the one described by my colleague Guy Sharon in the 4th EPTS event processing symposium, about tracing people in industrial facility to verify authorization of people to be in a certain place accompanied or unaccompanied, since the customer was exposed in public, I can mention it has been British Petroleum, this is something that the IBM Haifa event processing team has been heavily involved in creating this application. Some more applications in different industries have also been presented. Event Processing is being used far beyond its early adopter, the financial markets industry. I view it both inside and outside IBM that people who have been somewhat skeptic about event processing 2-3 years ago, talk differently about it now. Actually, the interest will yield more investments and advance this area to cope with the various challenges, but this is another story.
The main business value that event processing is achieving, according to IBM studies, is business agility. A study done reveals that in today's environment, almost all CEO are trying to re-think and change their business processes, and often software is an obstacle. Event processing is considered as an enabler for such agility. I always believed (and wrote about it several times) that the main business value is agility and not high performance that is required by a certain segment of applications, while agility is required by many more... Reality has its own strange ways...
A comment about branding -- IBM decided more than a year ago to use the brand BEP (Business Event Processing) for the entire portfolio of capabilities in the various IBM products related to events, and not adopt theCEP acronym used widely in the industry . The idea has been to emphasize the "business user orientation". Anyway, I'll leave the branding discussion to the marketing guys -- the content is more important.
There are more than 20 sessions in the conference that talk about event processing from different angles -- from the business perspective, the technical perspective, various related products in IBM (e.g. enabling CICS applications to serve as producer and consumer of event processing), my own session later this week deals with the future, but I'll Blog about it at later time, it is late, and I need some rest, getting up early every day for some conference calls...
Saturday, May 2, 2009

Packing for a one week (net) travel to the USA. We had recently been informed that the paper entitled: A stratified approach for supporting high throughput event processing application
Here is the abstract of this paper:
The quantity of events that a single application needs to process is constantly increasing, RFID related events have been doubled within the past year and reached 4 trillion events per day,
financial applications in large banks are processing 400 million events per day, and Massively Multiplayer Online (MMO) games are monitoring in peaks 1 million events per second. It is evident that scalability in event throughput is a major requirement for these types of pplications. While the first generation of event processing systems has been centralized, we see various solutions that attempt to use both scale-up and scale-out techniques. Alas, partitioning of the processing manually is difficult due to the semantic dependencies among various event rocessing agents. It is also difficult to tune up the partition dynamically in a manual way. Manual partitioning is typically vertical, i.e. there is a single partition set with centralized routing. This paper proposes a horizontal partition that is automatically created by analyzing the semantic dependencies among agents using a stratification principle. Each stratum contains a collection of independent agents, and events are always routed to subsequent strata. We also implement a profiling-based technique for assigning agents to nodes in each stratum with the goal of aximizing throughput. A complementary step is to distribute the load among the different execution nodes dynamically based on performance characteristics of nodes and agents and the event traffic model. Experimental results show significant improvement in the ability to process high hroughput of events relative to both centralized solutions as well as vertical partitions. We find this to be a promising approach to achieve high scalability without requiring difficult manual tuning, especially when the traffic model and the topology of the event processing network is often changed.
financial applications in large banks are processing 400 million events per day, and Massively Multiplayer Online (MMO) games are monitoring in peaks 1 million events per second. It is evident that scalability in event throughput is a major requirement for these types of pplications. While the first generation of event processing systems has been centralized, we see various solutions that attempt to use both scale-up and scale-out techniques. Alas, partitioning of the processing manually is difficult due to the semantic dependencies among various event rocessing agents. It is also difficult to tune up the partition dynamically in a manual way. Manual partitioning is typically vertical, i.e. there is a single partition set with centralized routing. This paper proposes a horizontal partition that is automatically created by analyzing the semantic dependencies among agents using a stratification principle. Each stratum contains a collection of independent agents, and events are always routed to subsequent strata. We also implement a profiling-based technique for assigning agents to nodes in each stratum with the goal of aximizing throughput. A complementary step is to distribute the load among the different execution nodes dynamically based on performance characteristics of nodes and agents and the event traffic model. Experimental results show significant improvement in the ability to process high hroughput of events relative to both centralized solutions as well as vertical partitions. We find this to be a promising approach to achieve high scalability without requiring difficult manual tuning, especially when the traffic model and the topology of the event processing network is often changed.
More about event processing distribution and parallelization will be discussed in subsequent postings.
DEBS has also issued recently a call for fast abstracts, posters and demos, an opportunity to share with the community work that is in less mature phase. show interesting demos, and discuss ideas.
Wednesday, April 29, 2009
On events and relativism

Today has been a holiday, the Independence day of Israel, and we spent some of the day in going to an exhibition called "Body World", in which there is an exhibition about the human body and its various functions using parts taken from dead people who contributed their body using some preservation method developed by somebody in the university of Heidelberg, actually I also spent some of the holiday working, since there was something that has become artificially urgent. These high-tech corporates makes you a slave, I am getting too old for this...
Anyway, in Israel the day before the "Independence Day" is the "Memorial Day" to remind us that the independence has its cost. It reminds me that in my first year in the USA (I lived in the USA for several years around 20 years ago), there were signs in the street saying "memorial day sale", we thought that somebody is making a joke in a bad taste, which does not really fit the famous "politically correctness" of the Americans, but found out after talking with some local people that the typical American does not attribute any semantics to the "memorial day", and it is just a long weekend with sales and travels like any other long weekend, well -- a cultural difference, since in Israel, memorial day is taken seriously.
Today, I wanted to say something about events and relativism. One of the questions about chapter 1 in the "Event Processing in Action" book on the forum came from Richard Veryard.
The question has been: How do you count how many events? If you have a three-car pile-up, does that count as one collision or two, given that the third car hits a few seconds after the first two? Or three collisions, if the third car hits both of the first two cars?
My answer has been that the decision is relative for the application. From the insurance company or companies of the cars involved it may look as three different events, since the event refers to a single car; from the point of view of the traffic police it may be considered as a single event, where the number of cars involved is an attribute.
Another facet of relativism is whether an event is raw or derived. The event can be raw event from the point of view of a certain application, since it is provided from the outside, however, the consumer is sending an event that has been produced by another event processing application, and from the point of view of the producing application, this is a derived event. There are probably more example of relativism.
Monday, April 27, 2009
More on Revision

Long day today, I got to the office around 8AM and left around 9PM. Since we have holiday this week I am trying to condense the remaining days of the week and the result is long day with plenty of conference calls. The picture above is a glance (from below) on the IBM Haifa Lab (the pair of connected building on the right hand side of the picture), my office is in the back building (known as the "banana" due to its shape), and is not really in the nice part of the building -- the one with the view to the Haifa Bay -- well, one can have everything in life -).
I still need to complete the previous posting on revision. I gave some explanation about the concept of revision, and now I still need to discuss implementation of revision in event processing. To recall -- a revision in event processing is getting later knowledge that asserts that either a reported event did not really happen, or some information associated with the event was wrong.
Let's look at two separate cases, one in which the processing has not gone out of the event processing network, and second that the results of the processing have gone out to the "outside world".
In the first case, there may be an opportunity to revise the impact of the revised event by doing kind of undo-redo for all the event processing agents that it passes directly or indirectly. Direct ones are easy -- those that the revised event participate as an input in them, indirect is more tricky, since we need to trace the causality among events, in this case, an event that is an output of an event processing agent in which the revised event participate (relate to the same context) has a causality relation to the original event, thus, an event processing agent, in which this event participates as an input, also needs to do an undo/redo, and causality is a transitive relation, so it continues as far as the EPN arrived so far. It should be noted that the fact that there is a causality may not require a real undo/redo, take as an example that an event of type E1 designates a bid, and the event of type E2 designates the bid with maximal value arrived in a certain time interval. Let's assume that a certain bid has been revised, however, neither the revised bid, or the revising bid change the selection of E2.
The second case is that the revised event has consequences that have been sent to an external consumer, thus, it may have triggered an action, a collection of actions, or a workflow that has been carried out, and this may propagate further ("the butterfly effect"), in this case, either we can treat it as "too late" and do nothing, however, there may be a cases that it can be critical to undo/redo also the consequences, e.g. the revised event has some financial meaning. In this case we'll need to issue compensation for the triggered action, which may be impossible (the consumer does not support compensation) or difficult. I'll blog again about revising the history and its aspects at a later phase.
Saturday, April 25, 2009
On Revision

Saturday morning, and I am spending some spare time (well -- ignoring my huge to-do list..) in reading the autobiography of Shmuel Tamir, who has probably been the most influential lawyer in Israel, as well as a political leader whom I always had great respect to (I don't admire people).
Today I would like to write about the notion of "revision" and relate it to event processing.
This is inspired, but not a direct response to a thread of discussion started by Peter Lin in the complexevents forum, under the name: mutability and aggregation.
Revision is somewhat different from modification; in modifications fact are modified, in revision they are revised. For example: if John Smith moves from the USA to Canada, then the facts about John Smith are modified, while if, by mistake it was recorded that John Smith lives in USA, where in reality he always lived in Canada, this is correction of recording mistake. Some people may wander what is the importance to make distinctions between the two ?
The first use of "revision" that I came about was in AI, talking about "non monotonic logic", the rationale is that using "classic logic" one can reason about the universe just if there is perfect knowledge, so the example used is that although birds typically can fly, however there are some exceptions -- Penguin does not fly, Ostrich does not fly, bird with broken wings cannot fly etc..
Let's say that Tweety is a bird and we don't know anything else about it, according to classic logic we cannot say whether it flies, however, according to the various non monotonic logics, we can say that since birds typically fly, we can assume for any practical purposes that Tweety flies, as long as we are ready to withdraw from this assumption when new information (such as: Tweety is a Penguin) becomes available, in that case we may need to retract all the assertions that were inferred directly or indirectly from the revised assertion.
Later in life, I have worked on temporal databases; one of the motivations of temporal databases have been to issue "as-of" queries, meaning -- looking what was known from a viewpoint of a certain time point in the past. For example -- if we investigate possible malpractice of a physician (I heard that the national sport of Americans is to sue their physicians) then in order to determine whether a physician made a reasonable decision we need to know what was the information available to the physician at the time that he made the decision. In order to achieve that facts cannot be deleted or changed, but we need an "append only" database, the distinction between "modification" to "revision" is important for the decision analysis, there may be a difference between -- the fever was high in the next measurement, or if the fever was high also in the measurement before the decision, but it was reported wrong and this information has been revised later. Eleven years ago I have co-edited a book about temporal databases which (among other things) discuss these issues.
Now, something about revisions and event processing. Recall that an event is something that happens, and it is reported to an event processing system using its projection which is also known as event (sometimes: event object, event message). An event that happens in reality cannot be modified or deleted, if it reflects something that happened. However, since when go to the projection on the processing system, again, if we assume that the knowledge is not perfect then we can have several cases of revision:

1. The event really did not happen, but it was reported by mistake that it happened, and the mistake was realized later.
2. The event happened, but it was not reported, and this was realized later.
3. The event both happened and reported, but some information associated with the event (reported through the event's payload) had wrong value due to error that was corrected later.
I'll post soon a continuation that discusses the implications of revisions on the processing of events. More - Later
Subscribe to:
Posts (Atom)