On the patterns collections list

Back on dealing with the EPIA book, we are now in the process of the 2/3 book review, and started to work on the last 1/3. Right now I am working on a section talking about temporal issues in event processing, but before talking about that, I still wish to get back to the previous chapter that deals with event patterns, continuing the discussion that I have started in this posting, and continued in this posting. In the book we bring a collection of patterns, these patterns are not meant to be complete, and we expect to grow the collection of patterns over time using the book's website.

The patterns collected are of several types:

• Logical operator patterns: all, any, absence that designate conjunction, disjunction and negation event patterns.
• Threshold oriented patterns: count of events, average/maximum/minimum of some attribute of a collection of events has some binary relationship (e.g. > ) with a given threshold.
• Relative patterns: relative max/relative min selects the events with the minimal or maximal value for a certain attribute over a collection of events.
• Modal patterns: sometimes, always, select a collection of patterns if a certain predicate is satisfied over all/some of the events in this collection.
• Not select pattern: This is a second level modal pattern that selects events that were not selected by a certain patterns.
• Sequence pattern: A temporal pattern that denotes a conjunction of event that occur within a predefined order.
• Trend patterns: Temporal patterns that detect trend, e.g. a value of a certain attribute is consistently increasing with a context.
• Spatial distance patterns: These are similar to the threshold patterns, but relate to the distance of events from some point in space.
• Spatial relative patterns: This are similar to relative patterns, but relate to the relative distance of events from other events
• Spatiotemporal patterns: This combine temporal and spatial properties, and designate direction of movement (e.g. moving consistently north, moving towards some entity).

The current full list of patterns consist of 30 patterns, and this list will probably grow. Each of the patterns is defined in the book and demonstrated using an example.

More on patterns - later.

More on responsive, reactive and proactive computing

Earlier this week, I've posted in this Blog short definitions of the terms: responsive computing, reactive computing and proactive computing. Somehow, terminology always gets questions and responses, and there has been a thread in the Complex Event Processing forum that started to discuss it. So as a follow-up to this discussion, some clarification. The computing modes (responsive, reactive, proactive) are indeed mutually exclusive, however a single system may have combination of all of them.

Active databases are example that start with responsive computing, the basic operations are: insert, modify, delete or retrieve from a database. Then the active database engine takes these database operations as events, and apply reactive computing to execute rules reacting to these events.

The opposite direction is a reactive system, doing some kind of event processing, which during the event processing operations need to consult a database in order to enrich an event with more information. The database query issued is a responsive computing.

Proactive computing may also be combined with reactive and responsive systems.

Hans Gilde has posted on the complex event processing forum an example that combines all three.

While responsive computing is the bread and butter of computing, and reactive is now being more understood, proactive is still lagging behind in terms of realizing the potential, maybe the next hill to climb.

On internal recognition

Today it has been announced that the "event processing" activity done over the years in the IBM Haifa Research Lab has been recognized, internally in IBM, as "outstanding accomplishment" in the category of "emerging market".

IBM Research has an annual process, as one of the way it is measured in IBM, for recognizing accomplishment (from various ranks and classes), most of the accomplishments are either helping IBM products or scientific accomplishment; the "emerging market" accomplishment in the "outstanding" level, was given, first time in the history, as far as we know, which made it quite challenging. This is not a personal award, but a recognition for an activity.

We have started working on event processing (not using this name, we called it "active technologies") in 1998, and moved through a long way inside IBM. Getting a big company to get into a new area is not an easy task, Lou Gerstner's book Who Says Elephants Can't Dance? provides some glance on how decisions are being taken in big corporates; I can write a book of my own about the event processing case.

Impact decisions, when you are part of the research part of the corporation, and is sitting in the back part of a big vehicle, is even more challenging. It was not many years ago when I've heard a senior architect in IBM saying about event processing: "this is just a hallucination of research people".

The recognition we achieved today is a recognition that we were critical factor in IBM's decision to get into an emerging market (recognizing that it is an emerging market). Even when this was done, getting official recognition about it is often not trivial.

Is it a reason for celebration --- not really, recall that I have stated that my motto in life is represented by the poem IF by Kipling. Quoting Kipling:

If you can meet with Triumph and Disaster

And treat those two impostors just the same;

Typically I don't tend to become depressed by the many frustrating moments, and also don't tend to be conceited by the few encouraging moments, it is all just a game.

While event processing has gotten over the years from being considered as "hallucination of research people" to an emerging market recognized by multiple analysts and all big software vendors, it still has a lot of challenges, so the work is far from being completed... there are more disruptive technologies ahead to conquer.

On the EASSy consortium

Back to regular work (and life -- I am trying now to contact the electrician to do some repairs in the house) after submitting the EASSy proposal for the EU project, waiting for evaluation and decisions (cometition is very tough).

Some more details about the EASSy consortium. EASSy is an acornym of: Event-based Adaptiveness of Service-based Systems. It consists of 16 partners, some of them are big commerical vendors (IBM, SAP, Software AG, Nokia Siemens Network, Siemens), some are system integrators (Thales, TXT, CITT, Corrvino), some academic/research institutes (FZI Karlsruhe, University of Trento, Aston University, National Technical University of Athens), and some public organization (Maccabi Helatchare and City of Genoa). The proposal itself is a very detailed document explaining the project approach, plan, management aspects, and planed impact. If this project is realized, I believe it will be of interest to the event processing community. Here is the proposal's abstract that explaines the idea:

The future Internet will play a major role as a medium for connecting service providers— with consumers in the supply chain (business-to-business connection), or with customers such as patients and healthcare organizations (business-to-consumer connection); thereby leading to the concept of the "Internet of Services".

Within service provision, the velocity of changing situations while maintaining the ability to adapt properly and proactively is becoming a notable factor in the ability to maintain business competitiveness, and can determine the survival of small and medium enterprises. The need to adapt arises in all areas. Examples include proactively adapting the treatment protocol for patients ("smart healthcare"), initiating changes to car-fleet delivery assignment based on travel conditions ("smart transportation"), adapting the working procedures of an emergency team in extreme conditions such as flooding ("smart emergency management"), and adjusting the plan and distribution of work between different parts of the supply chain when partial power failures occur in a specific region ("smart supply chain management").

During the past couple of years, Event-Processing Platforms have emerged as the fastest growing segment of application integration middleware (according to Gartner, Forrester, and IDC analysts). Many event producers are connected to the Internet (the "Internet of Things" phenomenon), and many business processes are instrumented to provide events from the service execution, so the combination of event processing and Internet services provides a tremendous opportunity to achieve the adaptivity described above.

The proposed EASSy project will take advantage of this opportunity and generate a solid framework that will revolutionize the capability of future Internet services to be adaptive. This framework will seamlessly provide both event processing capabilities and service adaptation capabilities. The event processing capabilities will determine when adaptation is needed, and the service adaptation capabilities will deal with the question of what adaptation should be performed, either autonomously or requiring human interaction; furthermore, the system should be accessible to business developers who do not possess programming skills.

Achieving this ambitious goal requires the project team to push the envelope beyond the current state-of-the-art and best practices, and will strive to contribute to the commercial world as well as to science in various areas, such as modelling, event processing, adaptation schemes, human-computer interaction, security, privacy, quality of service, and accountability. In addition, there are ambitious engineering challenges: to make the system function effectively and efficiently, be able to work on top of legacy systems, and deal with a variety of different types of event sources.

The EASSy consortium has built a well qualified inter-disciplinary team, which includes several world-class experts in the key research areas, as well as a strong engineering team, a test validation plan using strategic use cases, and a well established exploitation plan covering various software sectors and industry domains.

On Proactive Computing

In the picture above you can see some products that are designated to protect your skin from various things under "proactive skin care", actually preventive medicine is also aimed at protecting one's health from future problems.

Mani Chandy has recently posted in his Blog an article about Proactive computing. I thought it is worthwhile to say a few things about this notion.

We can view three types of computing: responsive computing, reactive computing and proactive computing.

Responsive computing is the way most computing is done, I am using now some software for Blog editing, sitting on some server and waiting for me to connect, and use it, in the rest of the time it sits idle (well, the server is, for sure, being used for other important stuff), and does not do anything unless explicitly requested, this is typically the way we are working with Web servers, databases and mostly everything else when using computing.

Reactive computing is the way that event-based systems are working. An event occurs and then it triggers one or more computational processes, without being response to any request. Event processing is a vehicle for doing reactive computing,

Proactive computing is the case when a computing system arrives at a conclusion that some entity will reach an undesirable state, and attempts to eliminate it from getting to this state, or mitigate it by transferring this entity to a more desired state instead.

Some examples of proactive computing:

• The traffic lights policy are set before a football game is going to end and the traffic is expected to flow in certain directions.
• A repair technician is late in one of its sessions, and combining with traffic conditions he will be very late to his next customer, something that will cause several chain effects.
• A progression in the state of a patient will make his treatment protocol inadequate

There are many other examples as well.

As said that we are now at an era of the responsive computing, reactive computing had caught on in some segments, and more segments of computing applications are now discovering it, but it is far from reaching its full potential. Some of the reactive programming applications are aimed at proactive computing, by the fact that some combination of events may be interpreted as getting to some future state that should have some action, and there are some commonality among them, it is still a good idea to look at proactive computing as as its own notion, and understand what a proactive computing application is like, and what are its characteristics.

I'll write more on that later.

On the event processing languages experience

I have realized that I have not written in this Blog for over a week, this is neither because I am away, nor because I am dry of topics to write about, but in the last few weeks I spend many hours a day in wrapping up a proposal for creating a project or the European Union program, by a consortium of 16 partners, beside IBM Haifa Research Lab (the organization that pays my salary) the consortium includes SAP Research, Software AG, Thales, Siemens, Nokia Siemens Networks, TXT, CITT and various universities all over Europe, total of 16 partners from 9 countries. I'll write about the proposal itself after we'll submit it. Alas - It turns out that preparing such a proposal is a very time-consuming task, especially if one is the project coordinator, so I reduced anything else in life for a minimum, and working longer days than my usually long days. Luckily, there is a deadline next Monday, after which I'll be able to return to normal life. Doing collaborative research among many partners from different countries, and different types of organizations is very interesting game.

I also started to teach the event processing course in the Technion yesterday. I am using the EPIA book draft as teaching material. The students assignment for this course will be quite interesting. Since we have made the EPIA book language-neutral, and use our own meta-language to define all parts of the application, we also wish to provide the reader with the opportunity to experience with event processing programming that actually runs, we have asked around, and got agreement of six language owners (four commercial products and two open source) to participate in this game. All of them agreed to implement the FFD (Fast Flower Delivery) which is the example that accompanies the EPIA book (see below a slide


The students will be divided to teams, each team will study a language, using the fast flower delivery, and in the process, help stabilize the various implementations for the readers, and play with them. They'll also get a mission to do some extensions of this example on their own, to experience the language also in an active way. This may be an interesting experience in understanding of the various languages that are out there. Will be quite interesting. This is also the first time I am teaching according to the upcoming book. Stay tuned to reports about this experience. More - Later.

On the ingredients of pattern definitions

In the previoius posting I started the discussion about the notion of pattern, and stated that it is a function that selects an event subset; continuing to drill down on patterns, I'll say a few words about what are the ingredients of pattern definitions, what information do we need in order to perform pattern matching in a well-defined way:

• Pattern type: which determines WHAT the pattern matching is looking for. There are a variety of pattern types, and I'll dedicate a posting to the pattern type I collected so far. Some examples: sequence (temporal pattern), moving north (spatio-temporal pattern), trheshold pattern (e.g. related to average of some value over a set) and more.
• Participant set: the set of event types whose instances issue the pattern matching set.
• Context: the context to which the pattern is associated with (actually the agent executing the pattern is associated with).
  
• Pattern assertion: Some patterns have assertions associated with them. Assertion can determine if events are relevant (e.g. if we are looking at a sequence of two events, say event of type E1, and event of type E2, where in order to do a pattern matching we require that E1.A > E2.B, where A and B are names of attributes. There are also some pattern associated with certain pattern type, e.g. if the pattern type is a threshold pattern than there is an assertion associated with it, e.g. Average (e1.A) [over each context partition] > 40.
• Pattern policy: Pattern policy determines when a matching set is going to be activated, how many times, can event count for more than one pattern, how repeated events are treated and more.
  

This was only in title level, and in the next postings I'll provide more information about pattern types and pattern policies.