Event Processing Thinking

On "CEP and Big Data 2" - comments on Philip Howard's observations.

2012-02-01T19:23:00.001+02:00

Philip Howard from Bloor Research has posted some observations on his Blog entitled "CEP and Big Data 2". Here are some comments (actually nothing new - just summarizing things I have written about before).
Philip deals with three issues:

whether the name CEP is appropriate or should be changed?
who should be credited as the pioneer of this area?
whether CEP implies real-time processing?
who are the CEP big data platforms?

Here are summary of my views on each of this topics.

The name "Complex Event Processing"

Exactly four years ago I posted on this Blog an explanation about - "why I prefer to use the name event processing without any prefix, infix or suffix". My particular dislike of the term "complex event processing" stems from the ambiguity in the name - some people (including David Luckham who coined this term) view it as processing of complex events, some interpret it as complex processing of events, and then debate of when something is complex enough, and what type of complexity is needed to qualify as CEP. Moreover some of the vendors use this term for products that are neither of the two options. I think that two words is enough for the name of a discipline, examples: information retrieval, machine learning, image processing and much more.... Thus, from my point of view the term "event processing" subsumes all other terms like complex event processing, business event processing, event stream processing and more.

Who gets the pioneering credit

Philip as a good UK patriot wonders why the Wikipedia value about Wikipedia and other sources gives credit to David Luckham and forget the Apama work that came from Cambridge UK. Looking at Wikipedia, it has one mention of David, as well as other references (like our EPIA book). It indeed does not mention Apama or any paper by John Bates, but being a Wikipedia, anybody can suggest additions.
David Luckham had major influence on this area, since he was the first one who published a full book and exposed the young area to the general public. An article in IEEE Computer, published in 2009, made some investigation of the history of that area and determined that in the 1990-ies there were four parallel projects that can be classified as starting points in this area: David Luckham's project in Stanford, John Bates' project in Cambridge (UK, not Boston), Mani Chandy in Cal Tech, and our Amit project in IBM Haifa Research Lab. I share Philip's view that John Bates should have full credit as one of the pioneers, and still view David Luckham as the "elder statesman" of the community.

Is CEP necessarily associated with real-time?

I have written several times about this topic, last time in response to Chris Carlson, to whom Philip also responds. There is some abuse of the term real-time in the industry, while its meaning is "within time constraints", many people interpret it as "with very low latency". This is not the same, anyway, event processing is a functionality with applications that require very low latency, applications which require to react within real-time constraints (which can be: 2 hours), some require both, and some require none.

Who are the CEP big data platforms?

I have taken upon myself the limitation not to state opinions on commercial products within this Blog - leaving it to analysts. Thus will make one comment. There is distinction between two types of software entities -
which is sometimes confused in the language used by people.

Event Processing Platform is a software that enables the creation of event processing network, handle the routing of events among agents, management, and other common infrastructure issues.
Event Processing Engine is a software that enables the creation of the actual function - in the EPN term implementing agents.

This is similar to the difference between an application server and a single component (programming in the small vs. programming in the large). Some of the available platforms for "event processing for big data" provide the first one -- it gives infrastructure, but not implementing any type of functionality, but enabling developers to create their own functionality, thus they don't do full-fledged event processing. Seems that many people classify both under the same classification (of course there are products that do both).

On spime

2012-01-31T08:53:00.000+02:00

According to Wikipedia: Spime is a neologism for a currently theoretical object that can be tracked through space and time throughout the lifetime of the object. The name “spime” for this concept was coined by author Bruce Sterling.

Spime comes from the combination of the words space and time, and is said to be enabled by the Internet of Things. In the event processing terminology - spime is the collection of events that happened to a single entity during its life-span, where each event has both time and space properties recorded as part of this event. Any person may have a spime associated with this person, which can span from birth and actually last long time after the person's death, e.g. if I am writing now about Isaac Asimov, this can be considered an event in Asimov's spime, although he is not a living entity. Spimes can relate to something with more limited length like a certain flight, or the event processing course I taught this semester.

In some cases it make more sense to have Spime processing rather than individual event processing and have some patterns associated with Spimes, this, of course, has strong relationship to event processing -- I've recently started to look and spime processing and will write more about it in the future

On Pecha Kucha

2012-01-30T23:01:00.000+02:00

Back to presentation skills, today, while working with one of my colleagues, Avi Yaeli, on a presentation, I've learned a new concept - Pecha Kucha. This is a presentation pattern, in which the presenter presents a topic in 20 slides, and spends on each slide 20 seconds, total of 6 minutes and 40 seconds per presentation.
There is a youtube presentation containing Pecha Kucha style presentation about how to prepare Pecha Kucha style presentations. I should try it once. There are also Pech Kucha nights which seems to be marathon of Pecha Kucha presentations.

Is computer science a science or engineering?

2012-01-28T21:11:00.000+02:00

I remember years ago a heated discussion in a conference whether computer science is a science or engineering, my daughter had a "science day" in the high school that she'll attend next year, and while they teach computer science they don't view it as a science, for them science consists of biology, chemistry, physics and some of their derivatives.

Recently I came across an article in "Scientific American", about U.S. science degrees. In this article, as you can see in the picture below, computer science is neither classified as science nor as engineering, it is actually classified as technology. Interesting -- I think that computer science is not monolithic, and various sub-disciplines may be classified differently.

On presentation skills

2012-01-22T21:25:00.000+02:00

Somebody attracted my attention today that ACM Membernet Europe in its last issue, has written about me in the section "feature ACM European distinguished speaker". In fact, several months ago somebody from ACM approached me to ask what is the meaning for me of being recognized as ACM Distinguished Speaker,
The truth is that I intended to use this program to tour some exotic places in the universe, but did not have time yet to pursue it, thus I answered that my action after this recognition is to coach and mentor young people about presentation skills. Indeed I have added to courses and seminars I am teaching a pitch about presentations (I am a fan of Steve Jobs' style of presentation), while this is a "soft skill", it is very important in today's world, as the picture above shows - sometimes more than what you say. In Israel we have a tendency to underestimate it, and believe that good content will sell itself, this is also true on product packaging. While some people are naturally good presenters, presentation skills is something that can be learned, and it is very rewarding to see young people catching quickly and producing great presentations (last week a students in a seminar I supervise did very creative presentations).

Intelligent Business Operations - a medical use case

2012-01-17T19:34:00.001+02:00

Within the recent year Gartner promotes the term "Intelligent Business Operations" (IBO) - not to confuse with Business Intelligence (BI). Roy Schulte from Gartner wrote about "operational IQ". I am looking now at the concepts and facilities of IBO, in Gartner's view. One way to study it is by looking on a recent post by Jim Sinur (also from Gartner). Jim provides a success story in the medical domain, resource allocation in surgeries. The ingredients of this scenario are:

Simulation-based optimization of scheduling and resource allocation in off-line for all surgeries planned for the next day.
Real-time tracking of everything: physicians, nurses, equipment; monitor of procedure duration and status - using sensors, cameras and in Jim's terminology - exploiting the "Internet of Things".
Determination of things already going wrong (not according to plan) or expected deviations from plan
Re-applying the simulation based optimization (this time online!) and get updated resource allocation plan.

This may be instance of the "detect-forecast-decide-act" pattern we have identified as the basis of proactive computing, although in Jim's scenario it can also be reactive (the deviation from plan already occurred - there is no need to forecast anything).

I'll write more about the IBO concept and some additional ingredients of it soon.
Since the term "intelligence" is now back in fashion, it would be nice to have metrics for the IQ of some operational process like the surgery management.
2.

FFD - the distributed version

2012-01-16T20:33:00.001+02:00

FFD (Fast Flower Delivery) is the example that accompanies the EPIA book. Recently Phil Windley, the CTO of Kyntex, started to teach a course in Brigham Young University, entitled "large scale Internet applications", Phil is using the EPIA book as one of the textbooks for his course. Today Phil posted in his Blog his variation of FFD as a totally distributed system, the illustration above demonstrates it.
This is interesting, indeed we see that event processing systems that started as centralized applications are getting more and more distributed. In Phil's version, there is no single EPN, but a federation of EPNs, per individual player (driver, store etc..), with pub/sub relations among them.

BTW - an interesting phenomenon, last week I have not written any posting in this Blog, have been busy in the last sprint of the EU proposal we submit (deadline tomorrow, so there is a light at the end of the tunnel), however, looking at the statistics, I think it was the week with the busiest traffic on my Blog ever both in page views and number of visitors, including a day which I think was the record high for this Blog for a single day ever, very surprising, I actually cannot explain it -- maybe some people got back from vacation and are catching up?. I'll publish some statistics in the near future.

breadth first as an obstacle for technology investment?

2012-01-09T19:42:00.000+02:00

An interesting post from Adam Deane, about "intelligent BPM", claiming that the obstacle for getting more intelligent software (in that case in the BPM domain) is the fact that vendors are focused on breadth and not depth, meaning providing products with simple (and shallow) capabilities, and focus on selling these products to as many customers, rather than working on providing "deeper" products and work with existing customers to improve the utilization of their products in more mature way. I guess that the accounting-based management of enterprises today contribute a lot to this approach, This observation is not restricted to BPM only, I think it applies to many areas --- maybe even in event processing?

On future work skills

2012-01-07T22:50:00.000+02:00

The IFTF - Institute for the Future has published a report about future work skills for 2020. This report, summarized in the picture above, identifies six drivers of change in the universe:

1. Extreme longevity
2. Rise of smart machines and systems
3. Computational world
4. New media ecology
5. Superstructured organizations
6. Globally connected world

To cope with them, the study determines that ten skills will be needed:

1. Sense making
2. Social intelligence
3. Novel and adaptive thinking
4. Cross-cultural competency
5. Computational thinking
6. New media literacy
7.Transdiciplinary
8. Design mindset
9. Cognitive load management
10. Virtual collaboration

Some of these terms are not self-explaining, so if you are interested in the details - read the article.
It is interesting to see if the educational systems -- from kindergarten until university level will change fast enough to cope with future competencies.

On agile enterprises

2012-01-07T20:52:00.000+02:00

I have not written for a while, I am spending much of my time in finalizing EU project proposal, yes - I know that I determined never to take upon myself coordination of EU project proposal again, but I had to re-learn and get to the same conclusion. The proposal is due in 10 days, and then I am back to normal life, but from time to time I am also catching up on other things. I found a posting by my colleagues from (former) ILOG, Daniel Selman about the "5 principles of the agile enterprise for 2012". The term agile became fashionable, and as seen in the picture above is used today as a medication of various illnesses. I first heard this term around 20 years ago in the context of agile manufacturing, later in the context of agile software development, and now, as you can see from the picture, agile enterprises.

Daniel's five principles are:
1. Exploit historical data
2. React in time
3. Make consistent., high quality decisions
4. Performance,performance, performance....
5. Move from segments to customers

Principles 1 and 4 relate to data -- use historical data, and exploit big data fast
Principles 2 and 2 relate to actions -- react in time (I would say even ahead of time, be proactive!) and make high quality decisions, and principle 5 talks about personalized actions rather than segmentation-oriented actions.

All of these have technological implications and supporting technologies are certainly helpful in making an enterprise agile, but the biggest barrier for agility in enterprises lie in the human and cultural aspects. Using technology to change the culture is one of the next frontiers. My recent posting about proactive thinking as a cultural change is part of this observation. More - later.

Top ten reasons why large companies fail to keep their best talent

2011-12-29T20:11:00.001+02:00

I have written earlier this year about the Plato vs. Aristotle conflict - in the organizational aspect, putting the organization in the middle, and the employee serves the organization goals vs. putting the employee in the middle, and having the organization as a platform to achieve the employee's goals. When talking about top talents, they tend to be in the Aristotle side of the spectrum. A recent article by Forbes discusses ten reasons why large companies are not good at retaining their top talents. I still work at the big blue (IBM Haifa Research Lab), so I probably not qualified as top talent, but I heard all of these arguments before from people who left big companies. The ten reasons relate to large companies' red tape; failing to find a project that matches the top talent's passion; annual reviews and career developments issues; lack of patience from the company's part to the top talents' initiative which can be longer term; lack of other top talents around, and mediocre management that don't know how to manage top talent -- very interesting!

New tutorial on event processing

2011-12-25T21:19:00.000+02:00

Roland Stuhmher, whom you can see in the right-hand side of the picture, on the Jeopardy! set, in DEBS'11, has recently recorded a video piece giving a tutorial on event processing. Some of the slides looks familiar to me (well, he mentioned my name is his slide about "attribution of the slides"). In the slides he mentions a term iCEP, which either means that Apple has a new CEP gadget in its i series. BTW - according to Steve Jobs, the apple "i" (started with iMAC) stands for: Internet, Individual, Instruct, Inform, Inspire -- all start with I... So does any of them apply to iCEP? Another possibility is, of course, intelligent (I have used the term IEP in the past). Anyway - good tutorial, with some glance of the ETALIS project developed in FZI. Enjoy!

On event processing for real-time and non real-time applications

2011-12-24T20:08:00.000+02:00

Chris Carlson has written in the Informatica Blog about the fact that there is a growing segment of applications that are using event processing and are not really real-time, and stating the fact that their share in the event processing market is growing. I have written four years ago in this Blog about the term real-time and the abuse in its daily usage. When some people (and marketing messages of vendors) are talking about real-time, they mean "very fast", while real-time really means "within time constraints", the time constraint can be micro-second, second, 5 minutes, or 2 hours. Indeed, the early adopters in event processing, trading applications in capital markets, are based on low latency and fast reaction. Many other types of applications use event processing for the functionality of filtering-transformation-pattern matching (or continuous queries in the stream oriented programming style), and the non-functional aspects are secondary. The area which I am working on these days , proactive computing, has some applications in which there are real-time constraints, but typically not in magnitude of micro-second, but in seconds to minutes. This is the case where there is a forecast for a future problem (e.g. a traffic jam will occur in 5 minutes), there is a time constraint on activating an action (e.g. within 30 second there is a need to change the traffic lights policies to mitigate the traffic jam). This is a real-time application, but it has to react within 30 second, to impact in 5 minutes. The interesting thing is that low latency applications may be "best effort" and not have real-time constraints. Thus - there are low latency applications, real-time applications, those who have both, and those who have none. Interestingly, event processing applications can be found in all four groups.

five forecasts from IBM - innovations that alter the tech landscape within five years

2011-12-19T18:59:00.000+02:00

The annual IBM predictions on "five in five" has been published. You can read more about it in the IBM Smarter Planet Blog.

This year’s predictions are:

·    People power will come to life
·    You will never need a password again
·    Mind reading is no longer science fiction
·   The digital divide will cease to exist
·   Junk mail will become priority mail

Some EP related news

2011-12-17T18:15:00.000+02:00

Today I had some time to browse through some new staff on the web, related to event processing. Here are some of the things I've found:

I noticed that Sidhhi has posted a presentation on Slideshare. I have written before about this new event processing open source from Sri Lanka. While comparison to Esper, as an open source is fine, the presentation makes the impression that the state-of-the-art today is Esper and some academic prototypes. There are of course many other things in the market -- my previous posting makes a reference to Paul Vincent's list.

Fujistsu announced that it has developed complex event processing technology designed for use with cloud technology that employs distributed and parallel processing. The picture about is taken from their announcement. A new player?

A new course to teach Microsoft StreamInsight is now available on the web. The course is described in Richard Seroter's Blog, with a link to the actual course. The course lasts for 4.5 hours, I have looked at the beginning, seems to be a good one.

More updates - later.

The genealogy of event processing players - December 2011 edition

2011-12-10T10:26:00.001+02:00

(Click to enlarge)

Paul Vincent, who took upon himself to be the record keeper, posted a revised version of his famous market players with some additions. Over the last year I have encountered several more in this area, so if you are a player and want to be on the map, notify Paul and get into the picture.

Cloud Computing Journal on Big data meets Complex Event Processing

2011-12-04T20:47:00.000+02:00

Cloud Computing Journal published some citations from a podcast in which the analyst Dana Gardner, interviews Mahesh Kumar from AccelOps, about correlating streaming events with transient data to get real-time analysis of data in the context of big data, and cloud implementation. The applications that AccelOps targets are mostly availability and performance management and security. The idea is not new,it seems that there are various approaches and realization that data needs to be analyzed in real-time in order to get decisions, before being written to a disk.

Reactive and proactive as relative terms

2011-11-30T19:22:00.000+02:00

This is one of the most famous visual paradoxes, in this picture one can see a young girl or and old woman, some people see only one of them, with some concentration one see both. This is a kind of relative view, in event processing there are some relative terms, I have written a while ago about the fact that the terms raw and derived events are relative, a derived event in one sub-system can be raw event of another sub-system..
There are other cases of relative views (an entity may be both consumer and producer, for instance).

I have reminded on relativism while reading an article about the keynote talk of Jeff Shulman (Gartner's manager of the application integration and web services analysts team). Shulman is talking about cloud, mobile and CEP as the leading trends for application development and integration.
As a remark to Shulman's keynote, the article bring an interesting response of Chris Dressler, VP Technology in Cablevision, he sees a CEP can be used to find and correct issues before the end user has the need to make a complaining call.
This is an interesting example, from the service provider's point of view (cable TV company in this case), this is a reactive application, tracking events that already happened and react to them, from the home consumer point of view this might be proactive, since the consumer may not yet felt the impact of the problem, so from the consumer point of view, this is elimination of problem that has not really happened. More on this distinction - later.

Book review: Event processing for business by David Luckham

2011-11-27T20:28:00.001+02:00

I am holding in my hand a copy of the brand new book by David Luckham, entitled: EVENT PROCESSING FOR BUSINESS. His first book "Power of Events" is the first book that opened the current era of event processing, which made which made David Luckham the prophet and elder statesman of the event processing community. My first meeting with David in early 2004 inspired me to think about the future of this area, and gave me some label and framework for what I was doing at that time. Some of David's ideas like event processing networks and event patterns found themselves as part of the area foundations, not exactly in the form that was defined in the "Power of Events" though.

The new book is aimed at being non-technical book, aimed at people in business and IT departments that want to understand what is event processing, and what are its uses. It is serves a similar target to the book by Chandy and Schulte. In comparison our EPIA book is aimed at more technical audience that would like to understand the building blocks of constructing event processing applications.

The book starts with chapter 1, which has the ambitious title "event processing and the survival of the modern enterprise" - explaining what most event processing is - and gives six principles of how it should be used by enterprises. Then it moves in chapter 2 to a history lesson - surveying all ancestors of event processing simulation, networks, active data bases and more, getting to event driven architectures.
Chapter 3 surveys the concepts that Luckham used in his first book, with definitions and some modifications to the original concepts. Chapter 4 is back to a history lesson - this time from the point of view of the commercial world. Here Luckham repeats his evolution classification that he has talked about in the past: simple event processing, creeping CEP, CEP as recognized technology and unseen CEP. According to Luckham we have just moved recently to the third phase (CEP became a recognized IT). The fourth and last phase is unseen (CEP goes behind the scene since it is ubiquitous and exists everywhere), it also becomes holistic, and in fact part of the infrastructure of every system from household automation to national cyber security. Chapter 5 views the markets - existing and emerging - and talking about industries and applications, with 13 examples (seems that the author is not superstitious!). Chapter 6 explain the notion of event patterns, here is goes more to technical, but stays mainly at the example level, neither trying to define pattern language nor talks about the implementation of patterns in current languages, Chapters 7 and 8 are entitled: "making sense of chaos in real time: part 1 and 2", with some examples and methodological insights. Chapter 9 is the last chapter entitled "the future of event processing" talking about the phases of evolution to the next phase, and some futuristic applications like:solving gridlock in the metropolis. The EPTS glossary is reproduced as appendix.

Overall -- good source of material and insights about event processing especially for the non-technical reader and good summary of the various talks that David has presented in the last decade. A must read for anybody interested in event processing.

Orson Scott Card's empire

2011-11-26T21:28:00.001+02:00

I have been sick and stayed home for a few days, which gave me an opportunity to disconnect from the professional work and read "empire" and its sequel "hidden empire", both by Orson Scott Card, one of the my favorite writers, whose books are always thought provoking. The story is around the 2nd USA civil war, between right and left, where an history professor from Stanford, who dreams about American empire, following the step of the Roman empire, and concludes that like the Roman republic, the American democracy should be eliminated to achieve the empire, thus he orchestrates chaos, civil war, and some other actions, and takes over as an agreed upon president that both democrats and republicans nominate to restore order, later in the second book he takes advantage of a crisis to reshape some of the world and take another step in the empire vision. There are some questions behind the plot - whether the goal justified the means, since many of the book's hero characters, lose their life as a result of some of the actions, whether democracy is indeed a value (and if it is - whether today's political system is really democratic). I think that the clash between the ancient Plato's outlook that the individual exists to serve the society (Orson Scott Card's president is modeled after Plato's philosopher king as he writes in his epilogue to the books), or Aristotle's outlook that puts the individual in the middle and make society as means to serve the individual.
My own observation is that we gradually move from Plato's universe to Aristotle's universe, and that the current young generation puts the individual in the center. Societies of all types (including high-tech companies) will require to adjust to this new world. I have already written about the distinction between these two Greek philosophers, and will revisit this topic again.

More on big data and event processing

2011-11-24T19:54:00.000+02:00

Philip Howard, one of the analysts who follows the event processing areas for many years, recently wrote about "CEP and big data". emphasizing the synergy of data mining techniques on big data as a basis to create real-time scoring based on predictive model created by data mining techniques, his inspiration for writing this piece was reviewing the Red Lambda product. It is certainly true that creation of event processing patterns off-line using mining techniques and then tracking this event patterns on-line using event processing is a valid combination, although the transfer from the data mining part to the event processing part typically requires some more work (in most cases also involves some manual work). In general getting a model built in one technology to be used by another technology is not smooth, and require more work.
The synergy between big data and event processing has more patterns of use -- as big data in many cases is manifested in streaming data that has to be analyzed in real-time, Philip mentions Infosphere Streams, which is the IBM platform to manage high throughput streaming data. The data mining on transient data as a source for on-line event processing, and the real-time processing of high throughput streaming data, are orthogonal topics that relate to two different dimensions of big data, my posting about the four Vs summarizes those dimensions.

On the two-second advantage

2011-11-19T18:36:00.001+02:00

The last shipment from Amazon has brought me (together with the usual collection of science fiction and fantasy books) the book co-authors by Vivek Ranadive, the founder and CEO of TIBCO. I purchased this book to try and have some glimpse into what's going on in TIBCO, being one of the major players in the event processing market. However, this is not a book about TIBCO products, the book is of a type that I would call popular science, making the thesis that people who have been successful in several areas (the first example was Wayne Gretzky, claimed to be the greatest hockey players of all times) recipe of success was their ability to predict something that is about to happen before it actually happens and behave accordingly. Part I of the book analyzes several such cases in the human case. Part II of the book talks about the use of the same ideas in computerized systems and its utilization in several areas: making better wine, ending traffic jams, and explaining why nothing should ever break. In part III the authors talk about the concept of "the two-second advantage" and connecting it to event-driven technologies and claim that it will both make the world better and the human brain better. The book states a vision, I think that two-seconds is more a metaphor and not real time-interval, because for different scenarios more time ahead is needed, it also does not talk much about how to utilize the two-second advantage, since just knowing about things are just part of the picture. The idea somehow reminds of the movie NEXT, which actually talks about seeing two minutes into the future.

On the whole I found the vision in the book quite consistent with our own vision of proactive world, which is a major task I am trying to cope with recently, including the understanding of the cultural change aspect
In any event - interesting reading; it is refreshing to see that CEO of IT vendor can mentally release himself from daily life to write visionary books, maybe this is an implementation of Steve Jobs' seventh rule - sell dreams not products,

On Steve Jobs' seven rules of success

2011-11-19T13:03:00.000+02:00

I have written before about rules and commandments for intrapreneuring.
This time I would like to bring my an1notated version of Steve Jobs' seven rules of success.

1.Do what you love This is probably the most important advice, I see too many people continue with the inertia, it is always easier to continue then to change, one needs sometimes a courage to change, or to quote somebody else: "if you get up in the morning and don't go to work with enthusiasm - go somewhere else". I followed this rule several times in my life, will I have courage to follow it again if needed?

2. Put a dent in the universe When I was a beginning programmer in the Israeli Air-Force, and came to my commander with some radical ideas, his amused response was -- "our mission is not to change the world". Most people think this way, however those who have ambition to change the world are those who have any chance to do it. There are many pressures, especially today when Wall Street metrics governs the behavior of corporations - I am not sure how much I have succeeded so far to change the world, but it will always be the driving force behind what I am doing, and I try never to lose sight of it.

3. Make connections The over-specialization of modern life make people narrowly focused. This is good for achieving expertise of a certain area, but making impact on the world requires making connections between seemingly not related things from various disciplines. To make connections one has to do two things: be broad minded, and explore other areas, and connect with people whose expertise are in other areas and inspire them share the same goal. None of these are easy.

4, Say no to 1,000 things This is the art of focus on the right things, as one has finite amount of time and energy. In Apple it was reduction in amount of products. In the case of an individual -- try to focus the energy on a single goal, and fight temptations to do something else. This is a rule that I am well aware, but must admit that I have not completely followed.

5, Create insanely different experiences Any interaction with other persons is an experience. Apple had its Apple stores providing different experience, in the individual level -- the way things are presented and communicated is not less important than the content. Always tend to (positively) surprise.

6. Master the message I often see people who try to sell or deliver some message that they don't master, it is easy to confuse them with trivial questions, and their entire message seems confused. Mastering the message is important, starting with the message to yourself, if you cannot convince yourself you'll not convince others.

7. Sell dreams, not products In the IT area, both the buyers and sellers are more interested in how something fits the architecture, interfacing with other products, and other technical properties, then in what it really does and what is the benefit. Selling dreams in form of -- let's imagine how you can improve the quality of your life or your business -- is better approach - which makes it necessary to understand the needs and psychology of the customer.

AITE says that vendors need to be more aggressive in marketing event processing

2011-11-18T19:14:00.000+02:00

I have not read the recent Aite report entitled: "complex event processing - beyond capital markets", have to check if IBM subscribes to these reports. But from the promo page linked here I copied the figure above. It is interesting that the numbers are somewhat different among analysts, but all shows growth. One insight given in the report is that vendors should be more aggressive about their message, since many customers still don't understand what event processing is -- a challenge both to the vendors and to the community. One of the challenges we'll work in the EPTS level is to disseminate the event processing manifesto and ideas more aggressively in 2012 -- stay tuned for announcement in this area.

The MSc exam of Ella Rabinovich

2011-11-18T19:02:00.001+02:00

Yesterday I chaired the MSc exam of Ella Rabinovich, who finished her thesis in joint supervision of Avi Gal (who was my PhD student) and myself. Looking at my own record, Ella is my 21st MSc graduate, in addition to 6 PhD graduates (one person, David Botzer, has done both MSc and PhD theses under my supervision - so the list comes up to 26 people). I am doing graduate students supervision as a hobby (the Technion does not pay for it), it enables to investigate idea that are typically one or two steps ahead from what we are doing in the IBM Haifa Research Lab (which by itself has to be some steps ahead of the markets), Ella works also in our Proton team in IBM, the thesis deals with pattern rewriting as one of the means of performance improvement in event processing run-time, I have already written about our DEBS'11 paper on this topic. the thesis was well accepted by the examination team and got a relative high grade. My guess is that we'll hear more about notable research contributions of Ella in the future.

Continuous event processing in Quartet FS

2011-11-13T08:07:00.000+02:00

Continuing to survey additional product related to the event processing area, I came across Quartet, this illustration is taken from Quartet FS' webpage. Quartet FS advertises its product as "aggregation engine", from the description it seems to be some incarnation of active database, where the OLAP cube is constantly updated, this variation is useful for some financial services applications. I guess that we'll discover more event related products coming from different areas within different frameworks (in this case - OLAP/BI).

Proactive thinking as a cultural change

2011-11-12T18:49:00.000+02:00

Yesterday I've returned home from my last trip in Europe (last for the next few months I hope, at least no trip is now planned in the horizon). I spent Thursday in Zurich in a all day meeting with partners in a consortium we have created applying for EU project, we have reviewed several use cases, and talked with domain experts in several areas who told us that in their domains, proactive solutions simply don't exist. Reflecting on that I think proactive computing might be enabler for cultural change. We are very much focused on technology -- what is the architecture, what is missing beyond the state-of-the-practice, what are all the moving parts -- this is of course important -- but let's look at the bigger picture; the society is governed by policies, regulations, rules... Likewise, enterprises like to have strict policies to handle each situations, current analytic software serve this culture -- it typically serves those who make the rules, trying to constantly improve the rules by analyzing the past; the result is recommendation to the policy makers to refine the policies.
Proactive thinking is different, here the aim is to determine when following the current rules will yield wrong result and change it, if possible, in autonomous way. If we take a simple example, in the Zurich meeting we have discussed traffic management systems -- there has been a lot of work in this area, there are system who sense traffic congestion and alert about it; there are systems which learn traffic patterns according to hours a day, day in the week, and other relevant segments, and make optimization for the traffic-light policies. In many cases, traffic light policies are determined by persons who make additional considerations (a powerful politician that lives in some street - entering and exiting this street in certain hours get priority), anyway, people generally want the power to make these decisions, and prefer that the system will behave according to consistent rules. Proactive thinking in this case is identifying anticipatory traffic congestion and change the traffic light policies accordingly, overriding the regular policies, trying to resolve a specific problem.

This is a cultural change, since the culture typically prefers that the measures will be statistical, and the behavior of systems will be consistent, this preference has deep cultural roots in certain societies, and reflect the inability of people to get real-time decisions, or improvise. Those who will adopt proactive thinking may go beyond the cultural handicap and achieve relative benefits.... more about it - later

Darko's PhD defense

2011-11-11T15:01:00.000+02:00

On Wednesday I have been in Karlsruhe, Germany, been on the exam committee of Darko Anicic in Karlsruhe Institute of Technology, one of the elite universities in Germany. After the defense Darko's colleagues gave him this hat with a lot of pictures and some gadgets glued to it, and he had to guess for each of them what is the meaning and how it is related to him (because of me, I guess, they have conducted this ceremony in English). Darko's dissertation was on Etalis, a logic programming based event processing engine, Darko has investigated the implementation of event processing on top of a logic-based knowledge base that contains background knowledge and inference capabilities. He also took the challenge and implemented the Fast Flower Delivery example from the EPIA book; Etalis is available as open source.
I think that Darko has done very good work, both in the conceptual level and in proving the concept (he has done measurements that indicated that the performance is roughly equivalent and some times better than existing engines). I hope that Darko will be able to continue and grow as a researcher.

The Fear Index

2011-11-04T14:23:00.000+02:00

This morning I have finished reading "THE FEAR INDEX" by Robert Harris, who is known for his ancient Rome novels. This time he has a "science fiction" that happens in the present coupled with a thriller. It concentrates around hedge fund that uses AI for algorithmic trading, and succeeds to crash the market and make a huge earning by trading in short. It is inspired by the 2010 flash crash.

As I am a big fan of autonomic computing and it utilization in daily life and in the business world, I often encounter people who have concerns about computers making decisions, and computers taking over. Stanley Kubrick and Arthur C. Clark wrote the famous plot for the film of "2001 space odyssey" in which the spaceship computer took over, the computer name was HAL, which is backwardly-removed one letter from IBM. Some science fiction writers like Asimov also have a vision that computers (in this case android robots) will take over the world, but Asimov considers it as a good property, since he was not a big fan of the human race.

I am not sure we are close to computers taking over the world, but autonomic systems should be constructed with self-monitoring that can be visible. This is something that we'll be exploring (among other things) within new proposed project we are doing with many partners.

On video streams and events

2011-11-01T08:54:00.000+02:00

I have not written in this Blog for a while, last week I have been in a business trip in Brussels and Luxembourg, had a few hours to play tourist in Brussels and chose to go to mini-Europe, in the picture we see the France section - in the background the atomium, Brussels' iconic site. Travelling in the train between Brussels and Luxembourg I had two observations: the train accumulates delay, so it is better to take a big slack of time if you rely on trains in Belgium, also it was interesting to see in stations the amount of people getting off to take advantage of the 2-3 minutes of stopping in order to smoke, the amount of smokers in Europe in general, and in Belgium in particular is probably much higher than those we are used to in Israel (and other countries like USA and Canada).

Getting back to the events business - I have noticed Rainer's posting in the complexevents forum, about video camera on the bus in Ireland that he experienced, actually there are various cameras on buses for various reasons - example video cameras installed on school buses whose aim according to the site is:

•    Deter bullying, vandalism, and other offenses
•    Reinforce driver accountability
•    Document chain of events
•    Substantiate or refute claims
•    External stop arm activated video of passing motorists.

Realizing these goals there is a need to identify the events that need to be processed/notified/acted upon from the video stream. Some are required in real-time - for example stop fights or eliminate accidents. There are various other uses of use of video streams as source of events, I guess that within the "Internet of Things" infrastructure, cameras are important part - while some work on detecting events out of streams exist (e.g. identify car license plate numbers for automatic tolls in roads), more work is needed to fully exploit the opportunities here

Evented API

2011-10-24T23:53:00.000+02:00

I have written in the beginning of this year about the fact that standard APIs is now an emerging trend, and about the benefit of having standard API for event processing. No standard yet but I came across "evented API specification" developed by Kynetx. While the terminology is not entirely consistent with the terminology I am using, it is interesting to review such works, and advance into standard APIs in this area.

On use of event processing in routing

2011-10-22T17:04:00.000+02:00

Eric Roch, chief technologist of Perficient posted a question about the use of event processing for message routing rather than ESB, the router pattern is taken from the enterprise integration patterns by Gregor Hohpe and Bobby Woolf that we cite several times in the EPIA book. Roch mentions several reasons to use event processing, some of it relates to non-functional characteristics such as low latency, and in-memory management.

I think that there is some continuum that relates to the routing requirements - some of the routing decisions are quite simple, they are based on a predicate, typically on a value of a single attribute; some are more complex, and the routing is determined according to a pattern that involves multiple messages, in other cases the routing decision might involved a structure like decision-tree or decision-table, yet in other cases the routing is determined according to probabilistic reasons.

The better solution is not to have ESB and EP systems in isolation, and chose in each case what should be used, instead it is better to use a single programming model that includes all types of routing, and autonomic translation to implementation in various building blocks, some of them are taken from event processing and some from ESB or traditional messaging systems, this is the approach I prefer for event processing in general, not stand-alone, but synergistic with other technologies. In 2008 I had a posting about event processing and ESB, arguing that ESB is a natural carrier of event processing within a SOA architecture, and indeed I think that we'll see more and more ESB products that are enriched by event processing capabilities.

Pura vida

2011-10-20T13:13:00.000+02:00

Pura vida - means "pure life", is an expression that is being used in Costa Rica as: good morning, hello, response to the question "what's up?", general statement that life is good, thank you - and some others. It is the motto behind life in Costa Rica. We spent in Costa Rica 10 days (1 less than we planned), and here are some impressions:

Costa Rica is an excellent place for a trip in the nature - it has nice rain forests, rivers and volcanoes, as well beaches of the two oceans.
Animals -- you see them everywhere in nature. There are various tours in foot, car and boat to see animals.
The Parador hotel we stayed in the last 3 days of the trip is the best hotel I've ever been to (and I traveled all over the world). The picture above is the view from the hotel room, it was elected several times as best hotel in central america (including Mexico), and best spa (I got there the best massage since my visit in Thailand in 2007), t it is located within a forest on a mountain overlooking the pacific ocean in Manual Antonio, it has various animals on site - a hawler monkey provided wake-up call in one of the days in 5am, with extremely loud voice, on the tree in front of the hotel room; a raccoon entered the daughters' room, when they did not close the balcony's door, and scared them a bit, other raccoons visited us at the dinner table, and were scared away by the hotel staff. An iguana caught some sun in the middle of the mini-golf course, and frogs have settled at night in the swimming pool.
The local people in Costa Rica eat the same food - breakfast, lunch, dinner -- rice and beans and tropical fruits (in each they might have some supplements). At the end we got tired from this food and escaped to the sushi menu at the Parador.
Rain -- Costa Rica has a lot of rain, especially in this season. It has a four day heavy rain storm at the pacific side, we had there in the last two of these in Monteverde, but we did most activities in the rain; luckily when we got to the beach the rain storm was over.
The road infrastructure cannot cope with the quantity of rain, thus many of the road have lot of holes, some of the main roads are not paved, and they call travelling in them as the "Costa Rican Massage".
Tourism has become the biggest industry in Costa Rica, over the last 25 years, which directed the education system towards this goal. Most people can speak English.
The amount of Israelies we met was incredible given the distance and cost to get there, it seems that it is now quite fashionable among Israelies who already explored Europe and USA to visit Costa Rica.
Iberia is the worst airline company I have ever used --- in all aspects (including food), the worst thing was that they did not let us get on the connection flight from Madrid to San Jose, Costa Rica, though we got to the gate 15 minutes before the scheduled flights, due to the fact that our flight from Israel was late 1 hour, and they decided they'll not have enough time to load the luggage, though we were 20 passengers coming on that flight, and the fact that they have 1 flight per day to Costa Rica, they did not agree to make any effort to get us on the flight. They told us that since ELAL was responsible for the late flight - it is not their problem (we purchased the ticket from Iberia, and used Iberia ticket with code share). We had to spend 24 hours in Madrid before getting on the next flight -- and lost the first segment of the trip (which is non-refundable as anything else in Costa-Rica). I'll check if it is possible to sue Iberia for damages. The attitude of their agents in Madrid was also very unfriendly - a big difference between the nice people of Costa Rica everywhere, and those of Spain).
And on positive aspect --- Pura vida --- is a motto of positive thinking!

For more picture --- I am waiting that my daughters will put all the pictures taken on facebook.

Now it is a lazy holiday -- back to work early next week, and already a business trip to Europe!

Touring the universe again; current destination: Costa Rica

2011-10-05T20:26:00.000+02:00

In a few hours we are planned to start the long journey to Costa Rica, the chosen destination of our next Family vacation, during the holiday period (here in Israel). Will be out of touch for the next couple of weeks.

More on Internet of Things and event processing for personal systems

2011-10-03T19:24:00.001+02:00

This illustration is taken from a recent Blog posting by Phil Windley, the CTO of Kynetx, whom I met in DEBS'11. Phil writes about IoT as a source of personal event processing, and about "personal event bus", in which any consumer can subscribe to events and process them individually.

I have provided some background about IoT recently and about its potential to be game changer; the personal systems is an interesting aspect. Event processing models and systems of the first generation have been developed in mind of business oriented systems. However, I have stated before that I see the big potential of event processing outside the traditional corporate IT, and provided robotics and biological computing as some areas of interest. Consumer applications is definitely an area with a lot of potential, however the challenges to do it is beyond collecting and subscribing to events, it is the ability to compose event processing application by consumers, which requires progress in the abstraction level of composing such applications. This is definitely and avenue to pursue.

On context and punctuation

2011-10-01T19:54:00.000+03:00

This illustration is taken from the EPIA book and shows the notion of context as a first class construct that groups together events that need to be processed together, where each of its instances is processed as a single unit, context may have temporal aspects, segmentation aspects, spatial aspects, and state-oriented aspects, and the actual context instance may be combined of one or more aspects.

Recently I had some discussion with somebody from the data stream community who was in the opinion that the stream oriented way to do it is more simple since they are based on standard thinking (i.e. SQL).

In the stream research literature, the temporal context is divided between windows and punctuation, all the other aspects are somehow expressed by the group-by construct. I guess that windows and group-by constructs are familiar to many people, punctuation is less familiar, thus I'll explain the idea briefly.

A good source to learn about punctuation is Peter Tucker's site, from where the illustration below is copied:

Peter Tucker has done his PhD (under the supervision of Dave Maier) on data stream punctuation; the idea of punctuation is to enable creating sub-streams, since a stream by definition is infinite. The notion of windows in classical data stream models consists of variations of sliding windows, either by event count, or time, however, in reality there is a need to explicitly end stream, i.e. when the bid ends. The definition of
punctuation in the database encyclopedia (the value was written by Maier and Tucker) says:

In other words, this is a "dummy event" put into the stream to denote the end of a sub-stream.

Now the question is whether using windows + punctuation + group-by is indeed simple. The relational model's claim to fame is its simplicity, however, from early days of the relational model, it has been realized that there are semantic anomalies, and all the normalization rules came to resolve those anomalies, yet, the overall model has been simple. However, this is not really true for the extension of the relational model for streams, while the relational model has a single entity: relation, the extension has multiple entities: relation, stream, window. Punctuation is a kind of trying to add semantics as a kind of logical patch.

IMHO, the context model is semantically cleaner. It has a single semantic notion for all the constructs that determine how to group event together (in the stream terminology, create sub-streams). In the context model there are no dummy event; the notion of temporal context (window) can also support event intervals - time windows that start by event and end by event; the events that start and end the window are real events, and can be used for other purposes. Moreover, looking at the definition of punctuation above, it defines the end of a sub-stream, but assumes that the next bid starts when a new bid event arrives. However, if we look at a situation where bids have not only "end bid" events, but also "start bid" events, then we would like to ignore bids that arrive when no bid is open, and this has no natural representation, and has to be implemented with tricks, so we also need the equivalent of "start sub-stream punctuation". So, we view this as kind of context based on , and not as kind of event within the stream with a special semantics.

The model we presented in EPIA (which we keep evolve) to be a semantic model that provides abstractions above the current implementations of both event and stream processing systems; several product owners have already told us that they use concepts from the book as inspiration to next versions of their products, and I guess that full implementation that is based on this model as native is yet to come.

On the four Vs of big data

2011-09-30T19:57:00.000+03:00

In my briefing to the EU guys about the "data challenge", I have talked about IBM's view on "big data", recently Arvind Krishna, the IBM General Manager of the Information Management division, talked in the Almaden centennial colloquium about the 4Vs of big data. The first 3 Vs have been discussed before:

Volume
Velocity
Variety

While the 4th V has just been added recently - Veracity -- defined as "data in doubt".

The regular slides are talking about volume (for data in rest) and velocity (for data in motion), but I think that we need velocity to process sometimes also data in rest (e.g. Watson), and we need sometimes also to process high volume of moving data; the variety stands for poly-structured data (structured, semi-structured, unstructured).

The veracity --- deals with uncertain/imprecise data. In the past there was an assumption that this is not an issue, since it would be possible to cleanse the data before using it, however, this is not always the case. In some cases, due to the need of velocity in moving data, it is not possible to get rid of the uncertainty, and there is a need to process data with uncertainty. This is of course true when talking about events, uncertainty in event processing is a major issue still need to be conquered. Indeed among the four Vs, the veracity is the one which is least investigated so far. This is one of the areas we investigate, and I'll write more about it in later posts.

CICS event processing improved version

2011-09-28T10:36:00.000+03:00

IBM CICS is an example for smart producer of event processing system, it does not do event processing inline, but instruments CICS transactions to emit events, and works in a loosely coupled mode with any event processing engine that can read its emitted events. CICS TS 4.2 released recently has several improvements in the CICS event producing capabilities. Among these improvement are:

Including the event emission to be part of the transaction, by doing the event emission as part of the commit process. Note that since it is loosely coupled with the event processing itself, this does not becomes atomic unit with the event processing itself, I have recently written about the relationships between transactions and events, and identified this area as one that need to be investigated more.
Change management inside the event instrumentation in CICS with appropriate tools
Inclusion of system events inside the CICS instrumentation (e.g. connection/disconnection to databases, transactions aborts etc..).

Since the strength of a chain s typically equivalent to the strength of the weakest link, in many cases the producer is the weakest link, and the amount of work required to emit the right events and the right time is often much larger than the rest of the system. Smart event producers like CICS making this weakest link much stronger.

On Actian and action applications

2011-09-25T19:52:00.001+03:00

Ingres, one of the oldest DBMS companies which produces open source DBMS, and the first of the sequence of companies that Mike Stonebraker founded and sold, has recently changed its name to Actian, and positioned itself as focused on "action applications in big data". The stated rationale about "action applications" is that current BI create reports and then it is left to the human to read the reports (or screens) and decide what to do, in "action applications", the application trigger actions automatically in response to data events and thresholds. It seems that people from the BI community re-discovers/re-invents the Event-Condition-Action model? so they'll probably get to more advanced event processing at some point.

It is interesting to note that the motivation they state on the Actian website (you'll have to press on "action apps" to see it) is - "BI is not working, more than $10B are spent every year on a pile of reports with no actions". I guess that this is somewhat consistent with my previous posting citing a study that indicate that human decision makers don't succeed to get fast decisions based on BI. Maybe BI is getting in the hype cycle to the phase of disillusionment, and maybe people in this community like SAS CEO who said last year that event processing has limited appeal to BI (along with BI in the cloud), would have second thoughts.

On proton-based chips

2011-09-23T14:46:00.000+03:00

Asimov talked about positronic brains for robots, maybe one step was the construction of proton-based chips, instead of the regular electronic-based computing (reference made by Rainer von Ammon on the complexevents forum). The proton-based chips can have better potential to communicate with biological processes and serve as an actuator within the human body, even the human brain. When we called our proactive computing project in the name "proton", we did not know about such a chip, but it can have natural connection: The proton system makes decisions that are carried out using the proton chip that serves as an actuator for the proton system. Cool idea -- somewhat futuristic, but we are getting there. While the scenarios we are working on now are still within the familiar domains, I believe that the big potential of proactive systems are in robotics and computerized-control of biological processes.

On human inability to make real-time decision

2011-09-23T11:08:00.000+03:00

WW2, besides all the bad things it did to humanity, also accelerated thinking and technology in many areas, Winston Churchill's war room has advanced the methodology and practice of real-time decision making. There are today many decision support systems that help decision makers to make quick decisions, however, Chris Webber from the Economist Intelligent Unit (EIU), who was one of the speakers in the Progress Software conference last week said that their studies show that the obstacle for getting quick decisions is the human decision maker that has difficulty to get quick decisions even if all information and decision support tools are available for that decision. Maybe, in some circumstances, the human decision maker can be skipped and be replaced by an autonomic system that will be fed with goals and intentions and will make the actual decisions?

What's in a title?

2011-09-22T21:21:00.000+03:00

Some people noticed that I've changed my title in LinkedIn and sent me queries about it. I did not change my job today; as I noticed that my business cards are about to end and I need to issue a new one, I thought that this is also a good time to re-think on the content, and determined that the title I have there today does not really reflect what I am doing, so I switched to a better title, principle investigator of the Proton project, which is part of the IBM Research's "far reaching research" program (FRR), which we have extended, to explore the emerging paradigm of proactive event-driven computing. In the past I moved from working on concrete project to a more general technical strategy role, and then coming back to research, in working on requirements and challenges on the next generations of event processing, and towards the end of last year I moved back to concentrate on a single project which I believe has a huge potential. I have written about proactive computing before, and we exposed some of the ideas in the DEBS'11 paper.
I'll write more about this project in the future, and of course I am still writing about general event processing issues.

On the data challenges

2011-09-17T18:48:00.002+03:00

I guess that many people can easily identify this place, Paris Montmartre. I have been in Paris earlier this week, for a meeting with the management of the information management business unit of the EU ICT program. They arrived to a meeting with IBM to talk with them about IBM's view of the research challenges in data management, towards the future program of the EU ICT called "horizon 2020". I have been part of the IBM team that talked about several aspects, here is a slide that summarized my part of the presentation:

I have discussed six different challenges - I already written something about the old and new worlds,
There are also more challenges, and I'll write more about the rest.
Why am I there? it seems that today events are becoming an important thread of database research, while in the beginning database people thought (and some still think) that events is just a kind of data that should be stored in databases, and the from there we know what to do, there is more and more recognition that events are first class citizen in the data world, and that they play important role in new data models, data semantics, execution models, transaction models and more. In fact, events play a role in each and every of the data challenges we discussed.
I also spent a day to meet some potential collaborators for a proactive computing project that we are trying to devise.
Also had a few hours to play tourist, something I did not have time to do in my last two visit in Paris, actually last time I was a tourist with my family was 10 years ago, and then I was mugged in the Paris Metro. Overcoming the trauma, I used the Metro again, and even the RER train to the airport. Still crowded and hot (no air-condition). Montmartre was one of the places I visited (I still have in my office a portrait of myself painted by a street painter at that place, but did not go to be painted again).
Also talked with a person who reads my Blog and remarked me that the frequency of my postings becomes slower. This may be true, some periods I have more time and energy to blog and some less. Looking at the statistics on Blog posts, I think that in 2011 I can at least match 2010, but not 2009. Besides, I have written before that I refuse to enslave myself to quantitative metrics. More about the data challenges - later.

SpatialRules

2011-09-12T21:11:00.000+03:00

Sometimes I am getting notifications that reveal players in the event processing space that I was not aware of, the one I've discovered today is SpatialRules by ObjectFX. From the description on the website it is difficult to understand the exact capabilities of this product which is described as CEP for geospatial data. It seems that it support tracking events that relate to spatial objects - entering, exiting, are inside/outside/close to areas, with also some spatiotemporal capabilities. It seems that this product is not new, but I must admit it is new to me, maybe recognized in the GIS community. I've written before about spatiotemporal event processing capabilities in event processing products such as: Microsoft StreamInsight or Oracle CEP. ObjectFX seems to come from the other direction, instead of extending event processing to have spatiotemporal capabilities, it extends spatial platform (ObjectFX has other spatial oriented products) to event processing. It will be interesting to see whether the two approaches meet.

On the Internet of Things (IoT)

2011-09-10T13:44:00.000+03:00

Over the years I faced a lot of skeptic people, for the entire idea of event processing. In the early days when I explained people what event processing is, one of the common reactions was: this is not really practical, you would never be able to get hold of the events you need for this application. This has some truth in it, the producer and consumer parts of event processing are often the most time and cost consuming parts of an event processing systems, However, the availability of events increase with time, since many of the current trends, like the IBM smarter planet, are based on the assumption that events of many types and many sources are available. The picture above the the quasi-logo of the "Internet of Things", a phrase I always thought as somewhat funny. The idea is that any "thing" - car, building, cell phone emits events to an Internet scale infrastructure - which gained the name IoT. There is now a lot of work around IoT,
Research and Markets recently issued a report about IoT, called it a potential game changer. The report also mentions event processing as one of the ingredients of IoT. Here - the relationships comes from both directions, IoT infrastructure is a source of many events that can be used of many applications with event processing functionality as part of them. The IoT infrastructure itself requires event processing for filtering, transforming, and reducing the number of events by reporting derived events based on trends, or situations of interest. We are now looking of some potential use cases for our proactive computing project, and the relations to IoT is clear. While IoT typically works on the producer (sensor) side, there is also a consumer (actuator) side to it, I'll write more about sensor and actuators examples later.

On "All my sons" and the western culture

2011-09-05T09:42:00.000+03:00

Yesterday I spent the evening in the theater, watching "All my sons" again (in Hebrew). The first time that I watched this play was in the age of 19, I have tagged this play as the best theater play I have ever seen and the actors who played there as the best actors I have seen on stage. These two assertions have not changed over the years, it is still the best play I've ever seen, and the actors who played first time I saw it are indeed the best actors I've seen on stage. The actors played yesterday are good, but not in the same league of the two giant actors from the past.

"All my sons" is now in Israel plays to the popular sentiment. It talks about taking the greed, and making the "business' and profits in the center of the universe. In this play it takes it to extreme. shipping damaged airplane parts to "save the business" and causing many people to die is considered as extreme behavior, but the principle of putting profit in the center, is one of the cornerstones of the western culture. Israel is now experiencing mass demonstrations with the slogan "the people demand social justice". In Saturday night there was the biggest demonstration that the country has experienced. This wave challenges the western culture thinking, talking about "swinish capitalism", where cost of living for much of the population is too high since a relatively small group of people who are very rich, control most of the branches of the Israeli economy, with monopolies or oligopolies, and keep the cost high.

Maybe in a social utopia, big corporations would say: "we earn a lot of money, we can give up some profit in order to raise the standard of life for the common people, either by offering some cheap products if they are in the retail business, or sell directly to common people, or if they sell to such businesses, do it indirectly, by reducing price and in return ask them to reduce prices of some products to the end consumer, or if the customer is a government, use the reduction to offer more social services",

Of course, social utopia has nothing to do with reality, the companies work in the real world of Wall Street analysts, executive bonuses linked to accounting metrics, and stock prices. I wonder how many companies that would face the Joe Keller's dilemma from "All my sons" -- do something morally wrong, or risk losing the business, would resist the temptation. No wonder that Arthur Miller's play was accepted in unease in the USA in the 1940-ies, and earned him the title of communist. But I think it is good as a food for thought.

EPTS event processing glossary 2.0 is now available

2011-09-02T21:07:00.000+03:00

The event processing area keeps evolving and there are more terms being used. The EPTS glossary workgroup, co-chaired by Roy Schulte and David Luckham has published a new version of the EPTS glossary. It contains both old and new terms.

On interactions between transaction and events

2011-09-02T15:09:00.000+03:00

As a follow up to the transactions vs. events posting, I would like to discuss some interactions between event processing and events. ACID (Atomicity, Consistency, Isolation, Durability) are the properties of classic transaction systems that guarantee serializibility.

Some people think that event processing and transactions live on two separate universes, as event driven architecture advocates asynchronous mode and decoupling. Note that there are transactional asynchronous systems (e.g. messaging with guaranteed delivery).

One of my past Master students, Boris Shulman, has investigated the issues of interactions between transactions and events. There was no paper published from his thesis, he has done the thesis while working for IBM, but then went to work in a start-up and did not have time to complete a paper, and the topic was not on top of my priorities either, but recently I am thinking about it more.

In the thesis he pointed out several different motivations to look at the combinations of events and transactions:

A monitoring system that monitors a transaction system, events are emitted from the running transaction systems, in a way, emitting such events before transaction commit, violates the "isolation" requirement of the ACID. However, at some cases there are timing constraints, since the monitored event need to be processed without delay. This gets complicated when the transaction aborts, and all transactions operations rollback. In this case the monitored event conceptually never happened.
Event processing is done as part of an actual transaction, the producer and consumer are part of a transaction, and the event processing is doing some processing that happen to fall into the functionality it support, but it is part of transaction. Note that this does not view event processing as a stand-alone functionality, but as a functionality that can be embedded in other systems that can be transactional. In the past when we did the Amit nodes inside IBM Websphere Message Broker, it worked exactly in that mode, event processing as a specific node within a transactional message flow.
The event processing network (or a certain path in it) may have a requirement to behave as a single unit of work. This is true when an EPA reads/write data, but also the raw events and derived events may issue transactional behavior.

There might be other cases, but these cases indicate some cases for interactions between transactions and events. The transactions that relate to events, may not be pure ACID transactions, I'll discuss other model of transactions that might be appropriate - later.

Transactions and events: old world vs. new world

2011-08-29T21:07:00.001+03:00

Paul Vincent posted on the TIBCO Blog some thoughts on transactions and events. His story was mainly about bookings in a sleeping train that should have been in the same cabin, but the system did not have any way to do it in a single transaction, In DEBS 2011 one of the keynote speakers Johannes Gehrke talked about similar things, of not being able to coordinate travel reservation of two co-workers who want to fly together (but still want to have independent reservations for the rest of the trip). This is a symptom of the "old world" vs. "new world" phenomenon. The "old world" in IT is the world of OLTP, which is centered around ACID transactions (maintaining: atomicity, consistency, isolation and durability). The "new world" is a world of mobile apps, social media, NOSQL, Web 2.0 and more - this world has started from scratch, the "cool" geeks who created it don't want anything to do with the nerds that work in the old world (slang I got from watching American TV programs, starting from "happy days"), Anyway -- it seems that some relations do exist between the old and new worlds:

New world applications should use some old world techniques like transactions, but this is not necessarily ACID transactions.
New world applications should sometimes talk with old world applications
Old world applications should support some of the new world characteristics -- and have to evolve.

All these three relations do exist, I'll write in the future about each of these topics.

Last but not least -- back to the title of transactions vs. events. Event processing seem to have some of the characteristics of the new world, however, there are some synergies again between transactions and events.
One of my earliest posts on this blog, in 2007 had the title: "Event processing and transactions - real, real time, and real time enterprise", where I discussed briefly the idea of event processing within transactional system. Since then there was some progress in the thinking - I need to revisit this issue on the blog.
So made some promises on future posts on this Blog - and now back to the new world of looking at what is new on Twitter, Facebook and LinkedIn... More -later

Siddhi - an open source event processing engine from Sri Lanka

2011-08-27T18:21:00.000+03:00

According to Wikipedia, Siddhi is translated into: perfection, accomplishment or unusual skill. In the picture we can see the eight primary Siddhis. Siddhi is also a name of an open source event processing engine, recently advertised. From the basic description it is aimed to support event processing as stand alone applications, and experiment with optimization algorithm for the run-time engine. Siddhi came from University of Moratuwa in Sri Lanka, a university who claims an ambition mission: "to be the most globally recognized knowledge enterprise in Asia". I like ambitious goals, and probably making open sources like this can contribute to the recognition, of course, if this will get traction.

I know of event processing open source and products coming from the USA, UK, Germany, Austria, Sweden, Israel, Australia, and I probably miss some (will be happy to update this list) So now Sri Lanka to the list, We are starting to see more activities in Asia in recent years, and hope to see more.

On streams, events, programming-in-the-large and programming-in-the-small

2011-08-27T12:24:00.000+03:00

In the tutorial I've given in VLDB 2010, one of the first slides was a rhetorical questions - see above.
There are four opinions: some people think these are aliases, some people view stream processing as a subset of event processing that deal with ordered events, some people view event processing as a subset of stream processing, saying that event stream is one type of stream, and there are also other type of data streams such as voice stream, video streams, and there are also people who think that these two are actually totally different concepts, relating to different types of applications. There is something of true in each of them, looking at some interpretations, but IMHO none of the above is really true,

Curt Monash decided to renew the old terminology discussion on his Blog, Taking the "stream" approach which is favored by the database people which look at "data streams" as data in motion, and view events as type of data that does not need any real special handling.

The difference of opinions and terminology stems from the fact that some people are thinking about apples and some about oranges.

What is the apple? - let's take as an example the S4 from Yahoo Labs, in the Blog post I referenced here I mentioned that S4 is a platform for doing "programming in the large" for stream processing, what does it mean? -- it supports a data flow graph, where streams are flowing on the graph's edges, and the processing logic is embedded in the graph's nodes. How is this logic implemented? this is not part of the model, each developer can use the platform and implement the nodes, the platform takes care of the flow, and some non-functional properties (distribution, fault tolerance, cluster management, scalability in some aspects etc..).
It is a pure programming-in-the-large framework. There are others like that, in this case the model is blind to the type of stream, and the stream can indeed be video stream, voice stream etc.. I would call such a framework as "stream processing".

What is the orange? -- if we look at the abstract model of event processing, the way we defined it in the EPIA book, it is a model that is centered around the programming-in-the-small, with language primitives that related to the semantics of events: mainly the notion of context (when? where? to whom?) of events and patterns over multiple event occurrence. The orange does not sound at all like the apple.

Can something be both apple and orange? -- the answer is positive, while event processing can be implemented using various "programming in the large" models, we advocate the "event flow" one, and the "event processing network" can be mapped to the data-flow graph model of streams. So it is possible, but not necessarily to implement event processing as a kind of stream processing. It turns out that there are some benefits to do it, and we see that indeed this seems to become a dominant way for "programming-in-the-large", while the programming in the small is still based on the semantics of events.

The view point is always the hammer and nail issue. Those who have the stream processing "programming-in-the-large" see event processing as just an applications of their platform, and think that the platforms is the main thing. Those who are having event processing language view the semantics and functionality of the language as the main thing, and the platform as facilitator.

The intersection is not an overload, in stream processing one can add a node dealing with audio processing, but the event processing language might be of little value, likewise, there are implementations of event processing that are based on other programming-in-the-large models (such as: logic programming framework) and not on the stream model.

When looking at current state-of-the-art, we see that many of them indeed lie in the intersection of both, thus each of the sides can classify them its own way. The fact that most classify them as event processing may show where the market thinks that the value is.

Doing what one is inspected to do instead of what one is expected to do

2011-08-26T20:47:00.000+03:00

Lou Gerstner, the now mythological CEO of IBM that many still miss, wrote an excellent book about his tenure as IBM's CEO, much of it is dedicated to his organizational culture battles. The most famous quote from Gerstner's book is: people are doing what they are inspected to do, not what they are expected to do.

This is indeed true in IBM until today, but IBM is just a reflection of the western culture, in which anything is weighed and measured, the metrics are intended to achieve some goals, but get a life of their own, often not keeping close contact with the original goals. Furthermore, people find creative ways to satisfy the metrics in ways that have nothing to do with the original goals, since satisfying the metrics become the goal.

Some examples: In Israel the traffic police guys are measured by the quantity of traffic violation ticket they get credit for. I remember many years ago that I've committed some traffic violation (turned left in a place in which turning left was forbidden between 7-9am, it was indeed a few minutes before 9am), a police car that was quite far saw me, and then started a movie-style chase, the police car not only turned in the forbidden way, but traveled in the opposite side of the road, turning the traffic to stand in the side, and drive in a way that was very dangerous to the traffic, and when they caught me they wrote on the traffic violation ticket" -credit XXX(name)". Hope that the reward for the credit was worth the effort!
Other examples: One of the previous "legal advisers to the government", a position that in Israel is among other things that head of the state prosecution, wrote in his autobiography that he had again and again to remind the prosecutors that their goal is to ensure justice, however, since they are measured by the percentage of conviction, while they have decided to prosecute a person, they cannot back off, it will spoil the statistics. More examples from the education system: I have talked once with somebody who dealt with admission of graduate students in one of the best business schools in the USA, he told me that sometimes they miss students that some indications (and faculty members) show they can be exceptional, but their score in the GMAT is not that great, it is virtually impossible to pass these candidates through the admission committee, since it would spoil the statistics of minimum and average of the admitted students, and this is a metric that the business school measures itself against other business schools. More in the educational system - My long term observation is that for many of the students the goal is maximize the GPA, and gaining useful knowledge becomes secondary, thus elective courses popularity is often determined according to the past statistics about grades distribution. One of my students had a special talent to know how to learn to exams, get very high grades, and not remembering much a week later. According to the metrics he won -- he was on the president list. This talent still helps him in his life. Getting to the business world, the fact that corporations that are traded in the stock market, are measured by Wall Street analysts quarterly on accounting measure (Earning per Share) has a major impact on how the business world behave, first considerations are often short term (due to the quarterly metrics), and second, accounting thinking is not always consistent with economic thinking, furthermore, it enforces some low common denominator in the behavior of the business world and reduce variance in goals and natures of companies.
While metrics are not necessarily bad, there are two observations about them:

There should be constant check whether the metrics still reflect the goals, and whether those who satisfy the metric do it in a way consistent with the goals - and adjust the metrics (or the goals) to avoid misuse (same kind of thinking as "fraud detection").
There are cases in which goals are not translated well to metrics, or there are exceptions that are consistent with the goals, but not with the metrics -- people should be brave enough to stick to the goals and ignore the metrics (or handle them later).

I started with quote from Lou Gerstner --- which I see as one of the cornerstones of my own behavior, over the history I had several times "motivation talks" with people instrumented by metrics to advise me how to behave, and I'll end with another phrase I like , which I use for this people: My behavior is driven by a a compass and not by a weather wane"

A video clip from NEXT - demonstrating the proactive idea

2011-08-24T21:36:00.000+03:00

The credit for this idea goes to my colleague Zohar Feldman.

A good way to illustrate the proactive idea is to watch the video clip from the movie NEXT,

In this movie Nicolas Cage acts like a person who can see two minutes into the future, in the clip he is trying different alternatives how to approach a woman he is trying to pursue, understanding why they fail he improves it until success is achieved. This illustrates the proactive idea: event happens, their consequences are predicted and a decision what to do in order to achieve some goal should be taken. There might be different decisions, but we need to predict the consequences of these decisions and make the best one that will get us to the desired solution. Enjoy!

On flood prediction from IBM Research

2011-08-23T18:05:00.000+03:00

There have been some visible floods in recent years, like the one in Australia. Prediction of the course of floods can be tricky, since rivers can have multiple splits. Today IBM announced that IBM Research together with UT Austin a prediction model to predict the course of flooding. This combines IBM analytics research with UT research in the physics of rivers.

Note that disaster management is an area where proactive computing hold significant potential, however in order to realizing this potential, strong prediction abilities are required.

Additional presentation - talking about AI and EP

2011-08-23T17:38:00.000+03:00

To complete the presentations list -- another interesting presentation from RuleML'11 (the first out of two) by
Alex Artikis and Nenad Stojanovic talking about AI approaches to situation detection and remaining challenges.

Some EP related slides and videos on the Web

2011-08-22T21:08:00.000+03:00

With the help of Google Alerts, I have found some recent presentations that relate to relatively old staff, but has some interest nevertheless.

Surprisingly, somebody posted a presentation about RFID/ event processing panel in the first EPTS symposium in March 2006, relatively old staff but interesting even today.

A video describing the CICS event system has been posted on the IBM Education Assistance server. This is a system that instruments CICS to act as event producer that was released in 2009.

A presentation by Informatica on "Data Governance". where event processing is one of the enabling technologies.

Last, but not least. an academic presentation from INRIA with the interesting title "undoing event driven adaptation of business processes". The terminology here is somewhat weird, but it relates to the retraction issue, and proposes the use of EP for doing that.

There are also some presentations from DEBS 2011, on the DEBS website, hope that more presentations will be posted.

On temporal databases and DB2

2011-08-19T21:12:00.000+03:00

I have written before about temporal databases in this Blog, and in general I worked on temporal databases around 15 years ago, and co-edited the book whose cover is shown here in 1998. Temporal database is noted by the fact that academic people tried to drive standard in this area: TSQL2 lead by Rick Snodgrass and his colleagues. At that time it did not succeed since none of the DBMS vendors had an interest to see it as a high priority, these were the days were the Internet emerged along with XML, and the DBMS vendors had many other things to worry about. However over the time some DBMS vendors have adopted temporal capabilities within their products. Oracle already implemented temporal extensions supporting TSQL in its DBMS product. Recently IBM produced its own version of temporal database within DB2. It seems that there is now traction for temporal databases in various industries. Today my colleague Guy Sharon attracted my attention to a new article on IBM DeveloperWorks entitled "going back in time" describing the DB2 temporal capabilities, the traditional dimensions in temporal databases: transaction time and valid time got converted to new names: system time and business time (I make a note to write a post about the overuse of the term "business" ). These two dimensions enable to ask queries like: what was the value of a certain attribute in 7/7/2011, as observed from 8/8/2011. This can have different answer from the observation time of different days, since the knowledge about the past is changed in time. While the title of that article talks about "going back in time", and indeed using temporal databases is typically viewed about recording the past, temporal databases can also be used about recording the future, this was noted in a work published in 1994 by Arie Segev, Avi Gal and myself entitled "retroactive and proactive database processing" (I don't think that online version is available). Since we are dealing over the last year in proactive event-driven processing, the issue of looking at predicted future events that can be revised with time is very useful, and we are indeed looking on temporal database techniques for that. More on that - later.

Book review: Managing event information by Gupta and Jain

2011-08-16T15:08:00.001+03:00

I am reading the book "Managing event information" by Gupta and Gain, which is part of the series of short books by Morgan&Claypool synthesis lectures series. The authors, professors from U.C. San Diego, and U.C. Irvine, have concentrated in the modeling of events, the illustration below was copied from the book to demonstrate the flavor. The book starts with an introduction and running example about news agency, it also defines the setting of various types of events: spatio-temporal database events, sensor events, multi-media events (video, audio). Chapter 2 deals with "Event Data Models" starting with temporal database as a tool for event modeling, such that the temporal database represents the collection of states, while event is any transition in any state (value change), then it moves to discuss conceptual temporal models and discusses E* - a graph based event model using RDF and ontologies. formalizing E* and showing some example of modeling using E*. It discusses several modeling constructs such as disjointness (causal or temporal) of events; coverage; ordering' lifespan and spacespan of events, and constraints on graph relationships, such as follows, sub-class etc.. In chapter 3 the book discusses the implementation of the event data model; the illustration below is a UML diagram showing one of the supported patterns - events that participate within a situation, there are other patterns supported such as causality. Later it discusses storage model of various types of events. Chapter 4 is entitled "querying events" and discusses the possibility to query a collection of events. Example of queries are: spatiotemporal queries such as: "which meetings are scheduled in this hotel today after a certain talk", which looks like a regular query in spatiotemporal database, the queries can also support aggregates (looking for frequent meetings with a certain characteristics), and hierarchy, expanding to sub-events. It can also query continuous events, graph relationships etc -- these queries seem to be the same as queries on spatiotemporal and graph databases, except he fact that the database represents events. Chapter 5 brings a major application of the event model - the creation of a story based on the event model. The storytelling is of multimedia type, a thread of research that follows Brooks' paper.
Kevin M. Brooks: Do Story Agents Use Rocking Chairs? The Theory and Implementation of One Model for Computational Narrative. ACM Multimedia 1996: 317-328
The book is summarized with some applications, conclusion and references.

The E* model is interesting in the sense that it shows various relationships among events, and enable to get observation on the events and their relations to states. Since there are no standards in this area, the terminology used in this book is sometimes inconsistent with other publications in this area, but it is generally clear. The book proposes a holistic approach of event modeling, claiming that current event modeling systems are looking isolated aspects of events and cited the famous metaphor on the blind men that touch an elephant from different points, I have used this metaphor talking about misconceptions about event processing.

It still remains to be seen whether such models will penetrate the real-world systems.
I'll give as a project topic in my event processing course more investigation of E*.

More on Almaden and San Francisco

2011-08-11T18:34:00.001+03:00

Packing to start the long journey home; meanwhile some more trivia and pictures of notable places.

I have taken one of the days to visit the IBM Almaden Research Center, in San Jose. This is an impressive place, as seen in the picture it resides in a middle of a nature reserve, on top of a mountain. I have visited IBM sites in various countries -- Israel, USA, Canada, UK, France, Germany, China (hope did not forget anybody), and Almaden is by far the nicest one, has a very relaxed atmosphere, and also responsible for some of the breakthroughs of IBM research. A possible collaboration may lead to more visits- the minus side, it is far away, and 10 hours time difference for a conference calls.

In San Francisco itself, there are many interesting places, I have driven (not for the first time) through the famous part of Lombard Street which looks like this:

There was a long queue to enter this part of the road, which required stop and go in a steep hill climbing up, but it was worth the experience. I also drove on the Golden Bridge and stopped there to find out that it is much colder and foggier than down on earth. In Pier 39 I found the store for left handed

known as "Lefty's". One can purchase clocks that move anti-clockwise and a lot of stuff that can help to improve the life of left-handed people (like can opener, I gave up on the ability to use can openers many years ago!). I actually never understood why anti-clockwise clocks are relevant to left handed people, but discovered this time looking at some books that left-handed people always draw circles anti-clockwise (true for me). I purchased a book for lefties as a present to my daughter Hadas, the only of my daughter which inherited this noble property from me. I'll write more about lefties after digesting this book, but it is now packed.

Last trivia item -- in the past I thought that Chinese restaurants are dominants in SF, they surely are present, but it seems that the dominant ones are Japanese. I ate a lot of Sushi this week (I am a Sushi fan anyway -- the only form in which I eat fish).

Driving up and down the hills is fun, but fun is over --- driving to the airport soon.

On computational sustainability

2011-08-11T06:14:00.000+03:00

Every once in a while I learn a new phrase - the one I have learned today is computational sustainability, in AAAI there is a track dedicated to it, it turns out that there is a community and even an organization (originated in Cornell), their site has a rotating logo, this is one of their logos. They define computational sustainability as the way that computer scientists can help in improve the ecology, economics and society - pretty large. They have a "green" orientation and various environmental projects. The session I attended today in the AAAI track was dedicated to transportation management -- every speaker emphasized the possible impact on the environment -- if people stay less in cars, the amount of CO2 will be reduced, but the presentations themselves were quite interesting about intersection control in autonomic cars, avoiding starvation and gridlocks, and another talk was about a system deployed in several cities in Oregon, taking reports from the city (in Portland -- periodic every second, in Eugene - event-driven every time that a traffic light is changed) about the traffic light status, displaying the result on mobile phones for subscribers (sounds dangerous to watch while driving) and advises about desired speed to get the green light and advice to go in alternative road, if they'll have enough subscribers they'll be able to load balance the traffic. Cool stuff -- they actually process events.

BTW - I sat in the conference reception in a table with two robotics guys (they are a lot of them here) that deal with autonomic cars, they said that autonomic cars cannot be trusted to react quickly enough to dangers, so there is actually a human in every car that takes the control when needed. So the Google driverless car is not entirely driverless nowadays; this stems from the current limitations in perception and action speed in robotic technology that I've written about before.

We can label some of the things we are doing as computational sustainability -- a trend to follow!

On event processing and robotics

2011-08-11T04:24:00.000+03:00

In my standard presentation about proactive event-driven computing (I gave one of those in Almaden earlier this week), I always show these two pictures: I am about to spill coffee on my laptop, and my robot detects it and grasp the cup. This is the simplest example to explain the idea and people understand it and typically smiling (or laughing), in one instance one commented that it is not cost-effective to use robot for these purposes, which is probably true, but he missed the point of what I've tried to do.
BTW -- spilling coffee is something I do always on a daily basis (today I spilled ice-coffee), I am very talented in spilling coffee on everything (though until now not on a laptop, I have a rule of not drinking the coffee too close to the laptop), so having such a robot would be extremely useful to me!

In the EPTS meeting in Trento we heard a talk by Sebastian Wrede about event-based robotics, but I admit that I have never followed up with the robotics guys.

Today in AAAI there has been a keynote talk by Kurt Konolige about robotics, after the talk I came forward and introduced myself by saying that the closes I came to robotics was reading Asimov books (which probably was a bad starting-point for a conversation with him), and explained him a little bit about what event processing is and then moved to the coffee spilling scenario. His reaction was that in the robotics world they deal which more more basic stuff: How to improve the vision and perception, and how to calibrate the robot's arm to be more accurate, so he thinks that the speed in which the robot will identify that I am going to spill coffee and the accuracy in which he can grasp the cup are not in the level that the current robotic technology can do this scenario -- which does not mean that I'll stop using it! His message is that before robots will be able to participate in "high level" tasks, as he called what I described to him, there are many "low level" problems to be solved. Well - I'll have to wait patiently, or find some other pointer in the robotic community who will be interested in the synergy between robotics and event processing -- I should definitely start active search.

On Artificial Intelligence

2011-08-10T10:19:00.000+03:00

This morning, in the opening session of AAAI'11 there has been a panel of people who participated in the first instance of AAAI in 1980 who compared AI research then and today. Hearing them I could not have seeing in my mind the picture of the late Professor Yeshahayu Leibowitz, maybe the smartest Israeli of our time, who was known in his blunt controversial statements and opinions (e.g. he has been religious, but claimed that religion is just a set of axioms he taken upon himself and had nothing to do with reality). The reason I remember him is because in the height of the Artificial Intelligence boom he wrote an article in one of the daily newspapers entitled "Artificial Intelligence - an oxymoron", which claimed that intelligence is inherently a property of a living entity and cannot be artificial. Since then I always addressed the term AI with a grain of salt. AI research is very diversified, and there are sessions in this conference about: multi-agent systems, description logics, social networks, machine learning, natural language processing, search, knowledge representation and reasoning, planning, search engines, reasoning under uncertainty and more. It also seems that like academic research, each paper concentrates on a narrow aspect of one of these areas, it will probably take integration of all of these and more to create a real artificial intelligence that will break the Turing test. It does not seem that this is a focus of current AI research, it actually broke away from the attempt to create artificial intelligence to solve specific goals. Maybe the AI community needs a grand challenge. The first keynote address was of Dave Ferrucci, the principal investigator of the Watson project who won the Jeopardy! game earlier this year. It is probably the strength of industrial research that unlike academic research can gather multi-disciplinary people and focus them on a single goals. Dave told us that in order not to stay focused on the goal, the researchers did not publish a single paper in four years, and started to publish all the scientific results only after the win - this is against the academic DNA, where people are measured on quantity of publications. Tomorrow another day of AAAI, starting the l-o-n-g trip home on Thursday. I also visited the IBM Almaden Research Center yesterday, but will write about Alnaden in a separate post.

AAAI 2011 tutorial on event processing and its challenges

2011-08-08T07:16:00.000+03:00

San Francisco. Just had dinner in an English pub here, it also serves the same food of pubs in England, and the receptionist at the hotel is saying "lift" instead of "elevator", but I am really not in England, but in San Francisco.

Today I have given a tutorial together with Yagil Engel in AAAI 2011 about event processing and research challenges in establishing the next generations of event processing that might be of interest to the AI community.
The tutorial is now available on slideshare.
It consists of two parts - the first one is introduction to event processing which is similar to previous talks I have given, and is in essence a short version of the VLDB 2010 tutorial, the second part talked about the challenges, the outline slide of the second part can be seen below:

More details in the tutorial itself -- enjoy!

On San Francisco and Storm

2011-08-07T06:13:00.001+03:00

Today I had a single day to play a tourist in San Francisco. Arrived yesterday afternoon, and is planned to stay until Thursday I am attending AAAI 2011, giving a tutorial there tomorrow, and plan some side trips and meetings, meanwhile spent all day out. Since I have been in SFO many times before (although last time was in 2007!), I decided to pick up a place that I have never visited before, looking at the brochure that I found at the hotel I chose the Walt Disney Museum, that resides in the Presidio which was a good opportunity to take a look at the Presidio (never been here before). The Disney museum was interesting and surprisingly I spent there more than four hours (including the Disney old film of the day - the "Absent Minded Professor" in black and white). Any visit in SFO is not perfect without walking near the bay, and eating sushi in Pier 39 looking at the waves. San Francisco is one of my favorite cities, although driving here require patience, as the center city is totally congested. And what about the storm in the title of this posting? well - the weather here is nice, no storms!, catching up on news I found a news item entitled "Twitter to open source complex event processing engine", which of course attracted my interest, but drilling down to the details -- the title is somewhat over-statement. Twitter has acquired BackType which created Storm, which is a kind of real-time Hadoop, on which one can implement event processing engine, but Storm itself is a platform and does not seem to have event processing language implemented, similar to Yahoo's S4. It should be noted that there are some similar works that have implemented at least part of the event processing functionality over a kind of real-time Hadoop. One of them is Colin Clark's DarkStar, the other is HStreaming.
Nevertheless, it is interesting to see the Twitter is getting there.

On spatiotemporal event processing

2011-08-03T00:06:00.000+03:00

In the EPIA book we referred to the spatial dimension of context, and also to spatial patterns of events, there is also a paper on this topic that I've written last year with Nir Zolotorevsky, There are also combination of the spatial and temporal perspectives, both in sense of composite contexts, and also spatiotemporal patterns
(such as: going north).

It seems that spatiotemporal event processing is becoming popular. Alex Alves from Oracle presented a paper about it in the industrial track of DEBS'11, and also recently wrote a short posting about it in his Blog;
Seems that Oracle is putting a support of spatiotemporal features in its product.

I also came across some description of spatiotemporal processing within Microsoft StreamInsight,

The current support in spatiotemporal capabilities is quite elementary, and various extensions are possible (a student project for the next semester?). It will be interesting to see more about applications that utilize the spatiotemporal capabilities, and their functional requirements.

On tents and demonstrations in Israel

2011-07-31T11:02:00.000+03:00

This is typically a professional Blog, with some off-topic deviations; this posting is one of them.
Nowadays nobody that lives in Israel can ignore the phenomenon of the last couple of weeks: mass protest in term of people demonstrating by living in tents in various places, mainly central Tel-Aviv, and the demonstrations all over the country last night (see picture above from the Tel-Aviv demonstration). What is the demonstration about? -- it started as protest about the cost of housing both purchase price and rental price, and accumulated other reasons for protest - talking generally on "social justice". In Israel the political focus as well as the major voting pattern is determined according to the war-and-peace decisions. Thus there is some abuse of the terms "right" and "left" in the Israeli politics, where (in gross generalization) typically "left" is referred as those who believe that Israel should take active steps towards peace, and support the establishment of Palestinian state, and "right" is referred as those who don't believe that peace is possible in our generation, and are against the establishment of Palestinian state. Since many of the people are frustrated from the "peace process", there are more in Israel who voted for the "right" block in that sense, and currently we have the most extreme-right coalition ruling the country ever. However the politically right administration happens also to possess social and economic right ideology, and it seems that this is the break point between the government and the people.
There are two types of socio-economic systems: the American system in which there are relatively low taxes, and relatively low social services; and the system of some of the European countries - with high taxes and high social services. It seems that gradually Israel adopts the bad aspects of both -- taxes are high (used to maintain the enormous defense expenses as well as supporting sectors that are part of the current coalition), social services are gradually decreasing. On top of it, the "free market competition" ideology that the current Prime-Minister imported from the USA in its extreme form, does not seem to work here. The reason is that the country is too small to have real competition, thus in each branch of the economy (banking, communication, food...) there are a small number of players who realize that it will be good for all to divide the market and maximize the revenues from its market share, and not really compete. The Cartel oriented market and high sales taxes in various forms results in the fact that the prices of everything in Israel are now more expensive than almost anywhere else in the universe. Those who suffer most are the young people that try now to establish themselves, and discover that even with relatively high salaries, they get to financial difficulties, this is the population that started the protest, and this is the root cause of the current protest. The required solution -- total change in the priorities and returning to the socially-based country. Is it going to happen under the most right-wing government ever? Will this become a social revolution or just phase out? we'll have to wait and see.

A footnote: the entire protest started within Facebook, which still serves as a major vehicle for coordination.

Richard Tibbetts on "visual programming for complex event processing"

2011-07-31T08:08:00.000+03:00

This picture is taken from the first slide of the presentation given by Richard Tibbetts last week in Oreill'y OSCON Open Source Convention. The presentation is now available on the web.

In the presentation Richard brings five lessons for the implementation and use of visual programming from Streambase experience. Note that the visual programming in this case is intended for developers, building visual programming for domain experts is a different saga.
In my event processing course, students have a choice to do projects in various available platforms (including implementing everything hard-coded), I found that many of the students prefer this style of visual programming, but some prefer to work with text-oriented programming that looks more like the programming they are used to, claiming they can have better control that way, I guess from the same reason that there are people that still prefer to drive manual cars. Personally I think that visual programming is the way to move forward, but we still have a challenge to do visual programming for semi-technical people.

More on event processing as analytics/business intelligence

2011-07-30T12:42:00.002+03:00

This is nice picture I borrowed from a BI blog, talking about seven ways to report BI results.

Last week I have written on the fact that nowadays we see various sources who classify event processing as a kind of analytics, Gartner, in a recent report, used the term "active analytics" in a report classifying event processing as analytics. Jeff Wooton from Sybase said in the Sybase Blog that I am pondering whether event processing is analytics, and stated his positive answer to that, After looking up the meaning of the word "pondering" (I still have some holes in my English vocabulary), I am not sure it is the right phrase to express my attitude. I have drawn an observation that people did not use to classify event processing as analytics in the past, and started to do it in the last year or so. Since I did not see a big shift in what event processing is doing at that time, then either there is a shift in how analytics is defined, or that the event processing marketing guys are riding on the analytics hype. I guess that there is a truth in both. The analytics guys are talking about event processing as part of analytics from their side.
Paul Vincent noticed that there is a recent Communication of ACM article entitled "An overview of Business Intelligence technology". Paul complains about the fact that the authors looks at event processing as equals to continuous queries, while in reality many of the models (and products) are base on other programming models such as rules (Paul has illustrated the distribution into programming models on a genealogy illustration of event processing languages). Paul is right in the sense that people from the database research community view all the universe as centered around their terminology, and are sometimes ignorant about other approaches. What makes it more interest is that one of the CACM article authors is Umesh Dayal (currently in HP Labs), Umesh (which is one of my early professional inspirations) is considered to be the father of "active databases", and the HiPaC project in CCA that he managed has coined the Event-Condition-Action architecture, and in a conversation I had with him a few years ago he thought that the ECA architecture had a lot of traction; Seems that Umesh did not keep track on the descendants of HiPac!
Anyway -- besides the complains, this is an indication that the BI guys adopted event processing to be part of BI from their side, which is somewhat extending the definition of what BI is. Interesting!

The 10 rules of scalability

2011-07-29T09:58:00.001+03:00

Yesterday I got an Email from LinkedIn with 77 pictures that show those of my contacts who moved from the beginning of 2011, while some of them just sent title within their organizations, most of them indeed moved to another organizations, some from senior positions in big companies to start-ups. Is it a trend now that 2011 is a year in which many people are moving?

ACM keeps sending me hard copy of the "Communication of ACM", in addition to send Email whenever the electronic copy is available. Yesterday I browsed through the June 2011 issue (it takes more than a month to get it delivered), and found out the paper entitled: "10 rules for scalable performance in simple operation's Datastores" by Mike Stonebraker and Rick Kattell. The 10 rules are summarized in the illustration below.

This is a mix of various types of advises: from - use high availability and automatic recovery, to don't try to build ACID consistency yourself, through don't be afraid to use high level languages, and even use open source. The domain of this paper are - as declared in the title - "simple operation's data stores", the question is what can we learn about scalability in event processing -- which is somewhat different -- neither focused around data stores, nor around simple operations. Also, scalability in event processing have several dimensions, not just scalability in the number of events, in fact in the DEBS 2011 tutorial we mapped all the scalability dimensions in event processing

I guess that shared-nothing architecture is always a good practice, the use of high level languages and the utilization of main memory are also good practices. Recovery is a matter of application's requirement, for some application recoverability is vital, for others - it is not really necessary. As for the use of open source, it is again depending on the context. In summary -- some of the rules are rather well known best practices, some are subjective, and some are context dependent.

Event processing = big data + real time ?

2011-07-26T20:24:00.000+03:00

BIG DATA is a hot phrase, one can also be labelled as "BIG DATA NERD" by purchasing the shirt shown above. The term BIG DATA refers to the explosion of data in the universe, especially methods to store and process huge amount of data.

Paul Vincent poses the question: CEP = real-time bid-data? Actually he doesn't answer this question on his blog posting, but the spirit of the postings makes the impression that the "=" relations is evaluated to "True".

The answer is that it is not really an equality, but more like an intersecting concepts.

Event processing of course does neither necessarily works on big data nor it necessarily has to satisfy real-time constraints.

On the other hand, real-time big data, is not necessarily event-driven and process events at all, as big data can be fairly static. As an example for real-time big data, I'll return to an invited talk in DEBS 2011 on Watson, in which there has been a live demonstration of the Jeopardy! game with the Watson computerized system, you can see in the picture that Paul Vincent is playing (very skilfully) the host.

Watson is a "big data" crunching application, it has very strict real-time constraints, so it certainly can qualify as "real-time big-data", alas, it has nothing to do with event processing, in fact the data exists prior to the game and no additional data is added during the game.

While the equality does not work, there is an obvious relation; in some cases there are substantial number of data inputs, which much of it can be reduced, or filtered out. Event processing can be used to process streaming data on the fly, and not follow the paradigm of store now process later. This is important especially one the processing is required in real-time.

More on event processing and big data -- later

On event processing critical success factors

2011-07-25T21:52:00.000+03:00

Some other input related to student's work, is the posting of Sascha Retter, based on his diploma thesis (the term discloses the origin in Germany). The observation is about critical success factors of event processing systems - integration with workflow management system, and modeling tools. There is some truth in both observations, but I would like to take broader view.

There is a market for stand-alone event processing systems, but we have observed that the larger market is event processing embedded in other systems, workflow management systems, or in the more modern name "Business Process Management" (BPM) is one of these systems, but this is not the only one, there are both other systems in which event processing can be embedded inside, e.g. analytic frameworks, sensor networks and more. It may also be embedded within packaged applications - like trading platforms, supply chain management and much more.

Modeling tool are a vehicle for usability, and indeed usability and ease of use has been identified as a critical success factor. This is now a well recognized fact.

There are other critical success factors for various applications, like non-functional requirements (performance). I also believe that standards are critical success factor for the entire area.

The EPTS use case survey provides more insights into properties and preferences of event processing customers.

On terminology again

2011-07-25T21:07:00.001+03:00

While the EPTS glossary team is about to produce additional version of the glossary, I received by Email a question about terminology from a student investigating this area, whether "event driven architecture" and "event processing" (both titles of books) are indeed the same thing

The answer is -- not really, but the difference is simple:

Event Driven Architecture is an architecture in which there are event producers, event processors, and event consumers, with the following principles: 1). All players are decoupled; 2). All the communication between players is that they send events to one another: 3). All communication between players is asynchronous.

Event Processing is a type of processing that takes as input one or more events and produce as output one or more event applying functions such as: filtering, aggregation, transformation and pattern matching.

The concepts are related in the sense that it is common to implement event processing on top of event driven architecture, but it is not tightly coupled:
Event Driven Architecture can implement pub/sub systems that are doing routing only and do not do any processing of events;
Event Processing can be be implemented in synchronous mode and not on top of EDA.

Another implementation of the "Fast Flower Delivery"

2011-07-22T19:57:00.000+03:00

In the EPIA book, we had a running example used for demonstrating all constructs in the book, the example described a scenario called: "Fast Flower Delivery". During the book writing we approached the event processing community and issued call for implementations, there has been six implementations that were ready during the book's writing: Aleri (currently Sybase), Apama(Progress), Esper, Etalis, ruleCore and Streambase. It seems that more implementations are being devised, I was asked for permission to use the "Fast Flower Delivery" scenario as the running example in an upcoming book teaching the use of one of the products, will write about that when this book will be out.

Recently, an implementation of this scenario in IBM Websphere Business Events (WBE) was posted on IBM developerWorks as a tutorial to teach the use of that product.
Seems to becoming the "Hello World" of event processing.

On Watson and Event processing

2011-07-19T18:08:00.003+03:00

This is a picture taken in the DEBS conference last week, we had an invited talk by Eddie Epstein from the Watson team, and he ran a round of Jeopardy!, two of the conference participants against Watson, here you can see that Watson has -800 point (click on the picture to see it clearly), but it recovered and won. We invited the talk on Watson, not because it somehow related to event-based systems, but because the conference took place in the Yorktown auditorium, where the famous Jeopardy! game in which Watson succeeded to beat two of the all time champions was recorded.

Today I've found a Blog posting by Shalin Shah from Vitria, with the promising title:

IBM’s Watson: What Does Complex-Event Processing Mean For Customer Experience Management?

So I tried to understand what the author thinks is the relationship between the two, the answer according to the posting - both of them can be used for operational intelligence.

Indeed, there are now efforts within IBM Research, to determine what are the next steps, since in essence Watson is "deep question answering machine" there are some areas that seem to be killer applications of this technology, among them are: medical diagnosis and helpdesk/contact center in which agents need to answer questions in a lot of areas. There are some others as well.

From technology point of view, Watson works in a different paradigm relative to event processing. It is not event-driven, but is based on a knowledge stored in books, encyclopedias, and other sources. What it does in real-time is - question understanding and question answering using statistical reasoning, and massive computational power.

The interesting question is what can be a synergy between question answering machine and event processing, here we can think of two sides: an event processing system is being assisted in Watson-like system in order to determine contextual information that can be used for evaluating assertions, or classify events into context instances. On the other hand a question answered can trigger event. Or the question answering system can be monitored by an event processing system. One can also think about real-time update of Watson's knowledge-base as a result of event, which is not the way Watson currently works. I think that there are various more synergies between the two types of systems.

More - later.

Event processing as analytics

2011-07-19T08:29:00.000+03:00

Recently, I hear more and more that people are classifying event processing as a kind of analytics.
This is partially due to hype that exists around analytics, and partially due to taking the word analytics in more broader term that denote general use of computerized quantitative tools beyond the traditional use of statistical processing. In sense it also reflects the fact that event processing is in many cases used as OEM inside sophisticated solutions, and not sold as a middleware per se. Is it the right classification? --- there are pros and cons, but linking to a hype seems to be a good marketing strategy especially towards people who don't know what it is. From research point of view, it is certainly a distinct discipline, though there are synergies. I'll write more about the differences and synergies in follow up postings.

DEBS 2011 awards

2011-07-18T09:50:00.001+03:00

Back in my office now from the DEBS trip, after spending Saturday in NYC and watched the matinee' show of Wicked, a wonderful musical.

Last remaining fact about DEBS 2011 is that it is the first instance of DEBS to grant awards, the award granting ceremony occurred at the conference banquet's on Wednesday evening. The awards are noted on the DEBS 2011 webpage. Here is the list of awards and award winners:

Best Paper Award:	Gabriela Jacques Da Silva, Buğra Gedik, Henrique Andrade, Kun-Lung Wu, Ravishankar K. Iyer.
	Fault Injection-based Assessment of Partial Fault Tolerance in Stream Processing Applications.
DEBS Challenge Award:	The ETH team:Lynn Aders, René Buffat, Zaheer Chothia, Matthias Wetter, Cagri Balkesen, Peter M. Fischer, Nesime Tatbul.

Best Poster Award:	Nihal Dindar, Peter Fischer, Nesime Tatbul.
	DejaVu: A Complex Event Processing System for Pattern Matching over Live and Historical Data Streams.
Best Demo Award:	Sinan Sen, Ruofeng Lin, Bijan Fahimi Shemrani.
	Complex Event Pattern Evolution based on Real-Time Pattern Execution Statistics.
Best Idea in the DEBS Gong Show:	Mike Lefler.

On pattern rewriting - DEBS 2011 presentation

2011-07-16T13:35:00.001+03:00

Among other things I have done in DEBS 2011, I have also delivered one talk to present the paper on patterns rewriting, the paper, like all DEBS 2011 papers can be obtained from the ACM Digital Library.

The paper is co-authored by Ella Rabinovich (the primary author), Avi Gal and myself.

The presentation starts by classifying optimization tools in use for optimizing the performance of event processing applications, the classification is into blackbox optimization where the actual implementation is taken as a blackbox, and whitebox optimization, The paper deals with one of the types of whitebox optimization - pattern rewriting, which mean rewrite a pattern into a collection of other patterns that yield equivalent results: same output to the same input.

While rewriting exist in different areas such as rules or SQL queries, there is some inherent complexity in some of the event processing patterns such as SEQUENCE. The difficulty stems from the temporal relationship imposed by sequence, which imposes restriction on assertion splitting, and from the fact that policies may not be transferred in the rewriting process nicely. This was an exploratory research to see whether rewriting is a useful method for optimization, especially where complex sequences are involved. The experimental results in simulation for sequence with 16 operands (a real application we have done in the past in the web commerce area), shows that rewriting alone can improve the latency by more than tenfold and also the throughput, and no matter how the latency/throughput trade-off is weighted, some rewriting will give better result than no rewriting in this case.

The presentation can be viewed on slideshare. Enjoy!

Summary of DEBS 2011

2011-07-16T04:11:00.000+03:00

In this picture you can see me explaining the rules of
the game for the "gong show", while Francois Bry, who was the gong master, is standing behind with the gong. You can also see that the podium has the Jeopardy! setting, and indeed we had in one of the sessions live game of two of the participants against Watson.

Paul Vincent, skilfully played the host, Scott Schneider and Michael Olson were the contestants, although the audience helped them Watson won this time also.

These pictures and many others were taken by Roland Stuhmer, and can be found on Flickr.

Besides the fun part we also had 33 papers, poster and demo session, DEBS challenges, tutorial day, PhD workshop and 4 keynote speakers

Overall, the community that includes researchers and some people from industry is becoming stable. the terminology becomes more coherent (everybody are talking about patterns), and the quality becomes higher.

There were participants from 23 countries from all the 5 continents including the first representative from Africa, a faculty member from Tanzania. I think that still the largest community is in Europe.

Organizing a conference is a lot of work, but is also a rewarding activity. My next conference in early August - AAAI in SFO, but there I am just a participant.

More on DEBS 2011 -- later

DEBS 2011 tutorial on non functional properties of event processing

2011-07-12T13:50:00.000+03:00

This is the picture that opens the tutorial on non functional properties of event processing that has been delivered yesterday within the DEBS conference by Tali Yatzkar-Haham and myself. It shows variety of various cakes, each of them can be entitled: "cheese cake". Likewise, there is a substantial variance among event processing systems based on their non functional properties. We surveyed a collection of topics on performance, scalability, availabilty, usability and security/privacy. One of the conclusions is that the usability area lacks comprehansive research. At the end of the day we went (with some other friends) to eat dinner in the cheese cake factory in White Plains (I did not eat any cheese cake).

The tutorial is now available on slideshare - enjoy!

EPTS awards announced

2011-07-12T05:54:00.000+03:00

DEBS'11 is quite a dense conference with a lot of activities, the only free evening is the tutorials day evening, which we used to have a co-located EPTS meeting, whose primary focus has been to grant the EPTS awards.

I am copying the notification sent earlier to all EPTS members:

Today, at the EPTS session that took place in the evening of the DEBS'11 tutorial day in Yorktown Heights, NY. EPTS announced its first series of EPTS awards.

Awards were given in two categories: EPTS innovative application awards and EPTS innovative principles awards. In each category two awards were granted.

The EPTS innovative application awards were granted to:

Starview Technology for the work on Advanced Logistics and Planning Solutions - ALPS^TM

Roberto Baldoni, Gregory Chockler and Giorgia Lodi for the work on CoMiFin

The EPTS innovative principles awards were granted to:

Hans-Arno Jacobsen and Mohammad Sadoghi for the work on BE-Tree: Boolean Expression-Tree
Elke Rundensteiner, Di Wang and Richard Ellison III for the work on Active Complex Event Processing

Congratulations to the winners, and many thanks to all those who submitted award nominations, and to the EPTS award committee: Alex Alves, Mani Chandy, Brenda Michelson, and Themis Palpanas.

DEBS 2011 tutorial day

2011-07-12T05:46:00.000+03:00

The first day of the DEBS conference is traditionally the tutorials day. There was a relatively large number of participants that attended this tutorials. I have listened to one tutorial given by Nenad Stojanovic and Pedro Bizzaro on the event processing grand challenge set in Dagstuhl, May 2010. I think that while all the ingredients are there, there still need to be some thinking about phrasing it in the grand challenge way, of an ambitious measurable goal. The second tutorial I attended (as a captive audience) was the tutorial that I presented together with Tali Yatzkar-Haham. I'll post this tutorial slides on slideshare, and write more about it within the next few days. Our tutorial covered the various issues of non-functional properties of event processing; there was some interesting discussion on several topics, like correctness, usability in general and transactional properties of event processing. Tomorrow the main part of the conference starts. More - later

Welcome to DEBS'11

2011-07-11T12:44:00.000+03:00

After a year-long preparations, DEBS'11 is sailing out this morning. I have arrived yesterday to the USA, and now writing from the Westchester Marriott hotel in Tarrytown, which ACM selected as the hotel of residence for the conferences's participants. The conference itself will take place in the IBM Research headquarter in Yorktown Heights.

I am general chair of this conference, which meant a lot of work; the local organization chairs who took care of all logistics - Bugra Gedik and Gabriela Jacques Da Silva, have done excellent job, so hopefully everything will be smooth.

Looking forward to meet a lot of all friends, and some new ones, it seems that the event processing research community is growing, and that there will be quite a lot of industrial participants as well.

Besides the "back office" work, I'll have also some "front office" roles in the conference - starting today, I'll deliver a tutorial on non-functional properties of event processing, together with Tali Yatkar-Haham from my team, for her it will be the first participation in DEBS, another person of the team Yagil Engel is planned to present here as well. I'll also deliver one paper presentation in this conference (following Ella Rabinovich's thesis), and have some speaking roles in the opening session and in the banquet.

Will be extremely busy days for me, will also try to post some during the conference, I am sure it will also be covered by other bloggers (e.g. Paul Vincent).

More -later.

Putting event-driven rules in context in real life

2011-07-05T19:12:00.001+03:00

We are often talking about the fact that event processing is done within a context, but what about event-driven rules in real life (not in computerized systems)?

Earlier this week there was an incident that the engine of one of the cars in the parking lot of our lab was on fire (I did not take a picture of that incident, the picture you see above is just a car on fire I found in the Web).

There was an announcement to all employees to move the cars from a certain potion of the parking lot.

The rule that local administration has is: When a fire event occurs, shut down all elevators.
However the one who composed the rule forgot to put it in context and say: when a fire event occurs in the building, shut down all elevators, and the rule has been activated. The result was counter-productive, since people ran in the stairs (we have an 8 story building).

More context-sensitive rule would be helpful.

Preview of the DEBS 2011 tutorial on non functional properties of event processing

2011-07-02T11:54:00.000+03:00

During the next few weeks I am planned to attend two conferences in the USA, DEBS'11 and AAAI'11, in both of them I am delivering tutorials (each with another colleague). The DEBS tutorial is about non functional properties of event processing, and it is now in advanced phases of preparation.
Non functional is a strange name, since typically name refer to positive aspects and not to negative aspects (starting with "non"). The slide below sows the agenda, the two large topic we'll discuss are the "performance and scalability" and "usability". The other topics are discussed more in brief.
I think that systematic treatment in these aspects are both interesting and important -- for example, we found out that in the two major performance criteria - throughput and latency, different people mean different things and measure different things. I'll dedicate some posts in this Blog on the insights we collected during preparation of this tutorial, on these and other aspects. After the DEBS conference I'll post the full tutorial slides on slideshare, as done in the past.

A course based on the EPIA book

2011-06-27T23:10:00.001+03:00

I got a request today to share the slides that I am using for the event processing course at the Technion, based on the EPIA book, in order to use it for an event processing academic course. At this point I will share it in a controlled way for those who wish to use it to give a course, if you are interested to give a course based on this book, please let me know and I'll give you access to the course's site.

On backwards windows

2011-06-27T22:13:00.000+03:00

Continuing the series of posts about temporal windows:
Like the clock going backwards in this picture, one type of window is a window that is opened backwards. The idea is that when event occurs, open a window backwards, to start either in some time offset, or by an occurrence of a specified event. This is another type of window that is supported by some models, and emulated by using queries to databases that store past events in most models.

An example is that an event processing system detects, using some pattern detection, that a person is a suspect in money laundering, but substantiate the suspect is done by looking for another pattern on past event history. This is known as "reinforcement" pattern use.

The main difference between forward and backwards windows, is that in regular forward windows, events are being kept in the internal state only if they are required by some EPA which lives within this window, thus the events that should be kept in the system's state are well defined.
In the backwards window case, since the window is not open when the event occur, then it should still be kept, thus we need to know of all possible backwards windows that might access this event in order to determine if an event should be kept, or just keep all possible events (which in the most flexible case, since it allow unrestricted addition of backwards windows, but may not be practical, as we may not know that an event even exists until some EPA that consumes it within a backwards window is defined.

This posting concludes the series of postings on temporal windows - for now.

Revisiting "Right Time"

2011-06-23T19:37:00.000+03:00

This illustration (as indicated in the bottom) is taken from the "enterprise irregulars" site, posted by Ray Wang.
Ray Wang cites a relatively old posting of mine, talking about real-time, right-time and other time related concepts. I admit that sometimes I abuse the term real-time (like other people do, but this is not a good excuse!), but I have not adopted the term "right-time". In that posting I bring some classical definitions of various types of real-time. Wang is making a somewhat different classification as a matrix with two axes: the reactive/proactive axis, and the business value axis (low/high). The high value proactive is called "anticipation", and the low value of proactive is called "nice things to do". My interpretation is that both deal with notifications that may allow proactive behavior, but not necessarily automated proactive behavior of the type that we talk about (see my keynote talk last year in the OMG conference), on the reactive front, the high value are mission critical reactions. and the low business value are called "timeless responses". Here, I am not sure it is the best title, as there are reactions that have low value, but are time dependent, since they lose their relevance in time. Example here is that getting an alert on available discounts in a nearby store may not be that important for me, but the discount is applicable only within the next hour, so if I would like to respond, there is time bound on this response. Anyway - interesting classification.

Plato vs. Aristotle

2011-06-20T18:21:00.001+03:00

Carole-Ann Matignon recent Blog posting was entitled: "Plato or Aristotle", the two great Greek philosophers, that were once labeled as those in which the entire western culture is a footnote to their writings. In this context, I guess that Carole-Ann meant the major difference in their outlook of life. Plato saw the individual as part of a society, while his student Aristotle, saw the society as a collection of individuals. The difference is -- who is in the middle: the society or the individual. Carole-Ann's posting was in the issue of privacy, or data accumulated on people, which in some cases it is good for the society in general, a government agency, an ability to get credit decisions and more, but can harm the individual's interest. This is an issue that is also dominant in dilemmas about event processing for years, the relatively ease in obtaining information about events, in a world full of sensors and cameras, and the privacy considerations.

Security and privacy in event processing is one of the topics we'll discuss in the DEBS tutorial about non-functional properties of event processing in DEBS'11. Stay tuned for me.

Who do you work for?

2011-06-19T19:44:00.000+03:00

I came across an interesting posting by Matthias Kaiserwerth, the director of the IBM Zurich Research Lab, entitled "who do you work for?". In this posting Matthias raises the dilemma, that it is much easier to identify with a smaller organization, sometimes until the level of a single project or department, rather than to identify with a big corporate, or any division of it.

I certainly agree, in the past, the department I managed had a logo, and I found that people identified much easier with this department, relative to the lab, division, or IBM. While this had a great contribution to the people's motivation and "unit pride", it was not popular in the environment (which I never thought it is a strong consideration). Creating excellence requires to have identification with a goal, it is much easier to identified with goals that one feels partner in, and not of some abstract entities.