On Decision Agents

Decisions are part of the enterprise life as well as the life of every individual. Take a decision that many of the readers have experienced: naming children. My wife and myself have realized in very early phase that we have completely different taste in names, so we have decided to agree on a protocol for how names are selected: taking turns - one of us is making a list of five names, and the other selects a name from this list. The selection is done only after the birth, and the list can be modified until the selection made. To be fair we need even number of children so each will play any of the roles equal amount of times (indeed we have four children which satisfies this requirement). This is actually work, none of us got the first priority, and none of us have to tolerate a name we really hate.

In the paragraph above I have described a manual decision, but increasingly decisions are made by computer software which makes or recommends decisions. James Taylor is constantly talking about "Decision Agents". I thought that it will be interested to look at this notion and discuss how it is related to event processing.

We can say that a "decision agent" has various properties that we can qualify as answer to questions:

• Why ? what is the reason that the decision agent activated. In the example above -- the birth of a child.
• which ? which information is needed in order to make this decision. In the example above --- the list of five names (which have been obtained by another decision or collection of decisions) .
• How? How the decision is made. In the example above --- by some human cognitive process, which may have reflection in a computerized decision agent.

Back to our world of event processing, we are using the term "event processing agent" to denote a software artifact that gets one or more event as an input, does something, and produces one or more events as an output.

The question is what is the relationship between decision agent and event processing agent ?


In a (not very recent) postings, Carole-An Matignon from Fair Isaac has attempted to demystify some terms. She used an analogy to the human body, saying that BRMS is the brain, while event processing is the sensor for the brain to get the decisions. So is event processing agent is a sensing agent ?

The answer is --- the terms decision agent and event processing agent do intersect, but none of them subsumes the other.

Returning to the decision agent questions.

The why question: A decision may be required since some situation has occurred, because some relevant fact has changed, or because somebody made an explicit request to activate the decision agent. In the first case (a situation has occurred), then this situation may be a simple event, but also may be a leaf of an event processing network. In this case, there may be some event processing agents that are part of the decision to activate the agent, so it is part of the brain.

The which question: Here again the information needed can vary -- it may relate to the present state, to the history of states, and transitions. Event processing agents can be used to prepare the required information, by taking events and filter, transform, enrich, aggregate, split and more. In this case the event processing agent is indeed a sensor, creating input for the decision.

The how question: There are various techniques to get a decision, detecting patterns on the event history may be a method to obtain a decision, together with other techniques, such as inferring from facts and rules, applying stochastic decision reasoning and more. This does not say that every event processing agent which performs "pattern detection" is indeed a decision agent, sometimes it just derive event that will be used directly or indirectly as input to a decision.
Interestingly, this is true for business rules as well. A business rule may derive new fact, the fact itself is not a decision, for example, it may be classification of a customer based on demographic information. The output of this rule is used as input to (another) decision agent. So BRMS like event processing can also play the role of both sensors and brain.

To summarize:

• An Event Processing Agent may be a Decision Agent, or provider of input or trigger to other decision agents.
• The same statement is also true for "state processing" business rule.
  
• A decision agent may be Event Processing Agent, but also can consist of several other types of agents.
• There may be blend of decision agents of various types inter-related. I'll write more in the future about this assertion.
  

On state processing and event processing

Yesterday, I got visited by my (now- Ex) Master student Elad Margalit, about his thesis regarding dynamic setting of traffic light policies I have written before. For some strange reason he decided that I deserve a gift for his graduation so he brought me a "flip clock" that looks like this. Strangely enough it switches the labels to show the correct time, all people who somehow got to my office yesterday thought it is a cool gadget, and it is now located in front of my eyes.

Today's topic is some echo to the discussion started by my friend and ex-IBM colleague Claudi AKA patternstorm on the forum in the complexevents site. Claudi has defined state as a sequence of events, while several others answered that this is not really the definition.

Before getting to definition, there was also very concrete motivation that Claudi mentioned -- if we equate state to "sequence of transitions" than state processing becomes a kind of event processing. I think that it is important to discuss this statement.

While state is not exactly a sequence of transitions, it is true that the value represented by the state can be reconstructed if we apply the series of transitions on an initial state, and considering that the initial transition is null and the first transition creates the state, we can obtain all information as part of series of transitions.

Let's take a simple example. The state represents the value of my balance in the bank checking account. The transitions start from a one that opens this account, going through deposits, withdrawals, commissions of the bank etc.. I have opened my current checking account in 1984. Assuming that I would like to process this state, such as getting an alert everytime that my account balance becomes negative (unlike the USA, in Israel overdraft is a common practice). I can make it an event processing activity by taking all transition from 1984 and reconstruct the state with each new transition, however, this is not an efficient way to do it, first I'll need to keep all the historical transitions forever, second, it is much more cost-effective to maintain the balance as an entity, and process it.

State processing and event processing are complementary, in states processing we are processing the snapshot of the present time, while in event processing we process the history of transitions. If I want to get alert on overdraft -- this is state processing, If a compliance officer looking for money laundering suspect is seeking if three deposits with more than $10,000 each were done to my account within a single week, he is doing event processing. In reality we need both, but each of them has other techniques for its cost-effective processing.

More on this topic -- later.

On 20000 visitors in the Blog

The numbere 20,000 typically reminds me of the famous book that you can see some poster taken from the related movie, however, today it means something else -- the 20,000th visitor has visited this Blog. Many of them just arrive there somehow while scanning the cyberspace, more interestingly, around 1,500 visitors are quite frequent repeating visitors, and a similar number are visiting from time to time (but in total visited at least 15 times). 10 percents of the visit were direct, and of the rest, most were referrals from varios type of Google options, and some refering sites: Complexevents (and the forum), Tim Bass's Blog, TIBCO's Blog, RuleCore's Blog and Apama's Blog.

More statistics: The most popular posting, by far, is : "On Unicorn, Professor and Infant"

written in June 2008, and still fresh. The next one is: On Agnon, the dog, playing and downplaying. Soon I'll write a follow up to this one. The third one talks on event stream processing, quite an old one. The next one, like the current posting is gossip about the Blog itself, last time I have written about this Blog, almost a year ago, the Blog had 3,000 visitors.

In terms of geographical distribution -- still most of the readers are from the USA, followed by UK, Israel, Canada, Germany, France, Japan, India and Australia. The number of countries is now 135 - some of the new ones are: Reunion, Guam, Swaziland and Namibia.

As far as cities go -- London keeps the first place, followed by Haifa (my home town) and New York.

That's all for today -- a professional posting will foloow tomorrow.

More on Event Pattern Detection and Discovery

One cannot ignore these days the change of president in the USA, something with affects the entire universe. One minority the Mr. Obama belongs to is the minority of left-handed people, as can be clearly seen in the picture, while four of the last USA presidents were left handed (which make his fifth in the last seven presidents), conference rooms in the USA or university classes - all of them have desks only for the right handed majority. Here is a picture of a left handed desk,
I am sure that the USA president has much more urgent items on his agenda, however, unlike his predecessors he may also do something for the deprived minority of left-handed people. BTW - the situation in Israel (whose current prime-minister is, surprise...a left-handed person) is somewhat better. We, Left handed people may be a small minority (around 10% of the population) but our collective impact on humanity is unproportionately huge -- starting from Alexander the great, Julius Caesar, Napoleon, Queen Victoria, Lewis Carrol, Mark Twain, Escher, Michelangelo, Leonardo da Vinci and many more....well enough of that for now.

Going back to one of my previous posts
that has explained the difference between event pattern detection and event pattern discovery.
In the wake of some questions, here is more about the relationship between these two terms:

• Event pattern detection is performed for patterns that are known in advance, the pattern detection in done "on-line" when the event occur.
• Event pattern discovery is performed typically off-line, it can use machine learning techniques on past events in some cases, it can also use some natural language understanding technique to derive pattern from legal documents (e.g. regulations) in other cases.
  
• A pattern discovery creates patterns that are detected on-line by pattern detection, so they are complementary techniques.
• In some cases there is continuous discovery, and thus the patterns are updated in a dynamic way, however, still the discovery feeds the detection part on-line, and the respective roles are preserved.
• Last but not least, the discovery process may use simulation techniques that use detection of simulated events in order to check assumptions about patterns.
  

Typically, event processing products contain event pattern detection capabilities, in one form or another. The event pattern discovery is considered as add-on, typically using techniques that are not particular to event processing.

On Complexities and event processing

For those who read the title and grinned -- not again, discussion about what is the meaning of the term CEP, relax --- I explained in a previous posting entitled: "Is my cat CEP" , why such a discussion is futile, and I am typically consistent. BTW - when I have written that posting I did not have a cat, since than my daughter has adopted one, and he does not seem to me complex.

However, I would like to answer a more interesting question that somebody asked me recently -- what are the sources of complexity in event processing ?

In high school we have learned about "complex numbers", we liked this topic, since it was one of the most simple topics in the matriculation exam in Mathematics... Complex number is just a combination of two numbers, thus the complexity is in the structure. David Luckham also coined the term "complex events", where the complexity is also in the structure. However, there are more levels of complexity that may serve as a motivation to use COTS instead of hand-coding this functionality. What type of complexities can we observe beside the structural
complexity ?

Complexity derived from uncertainty:

• The applications specification is not known a priori and has to be discovered, example: fraud detection. This is related to the "pattern discovery" I have discussed in the previous posting.
• There are no reliable sources to obtain the desired events, or the events achieved can have uncertainty associated with them. This is a distinct complexity, since there may be the case where the application specification is well defined but the events cannot be obtained, and vice versa-- the patterns are unknown, but once discovered, the required events are easily available.

Complexity derived from connectivity:

• Producer related complexities --- semantic differences among various sources, problems of time synchronization among various sources etc..
• Consumer related complexities --- similar to the producer ones, these two are, of course, orthogonal to each other, and to all other complexities.
• Interoperability complexity where various processing elements are involved.

Complexity derived from functionality:

• Complex functions requirements -- e.g. complex patterns that may involve temporal, spatial, statistical operators and combinations of them.
• Complex topology of the event processing graph, with a lot of dependencies among the various agents, which creates a complexity in validation and control.
• Complex subscription / routing decisions.
  

Complexity derived from quantities

• High throughput of input events.
• High throughput of output events.
• High number of producers
• High number of consumers
• High number of event processing agents (imagine 1M agents in a single application)
• Requirement to maintain high amount of space for processing state.
  

Complexity derived from quality of service requirements:

• Hard real-time latency constraints.
• Compliance with QOS measurements such as threshold on average latency, threshold on percentage of events that don't comply with some latency constraint etc...
• High availability requirements.
  

Complexity derived from agility requirements

• Dynamic, frequent changes in the logic of the event processing
• Need for programming by various types of "semi-technical" people among the business users community...

I am sure that this list is not complete, but it provides some indication...

Of course, a single application may be the ultimate complex application of event processing and need ALL of these complexities, finding this application is, for sure, the dream of every researcher --- getting a lifetime of research challenges, but in reality different applications have different combinations of complexities. An application can be simple in all metrics, but have hard real time constraints, it can have very complex functionality, but no quality of service, or quantities issues. Another applications may need pattern discovery, but again the rest is simple, another combination can be relatively simple application, with complexity in quantity of producers and consumers and in semantic integration with all of them, and with the wonder of combinatorics, one can get to many more combinations....

More on complexities - later.

On Event Pattern Detection vs. Event Pattern Discovery

This drawing, in various forms, has been used by us for many years to illustrate the notion of pattern, actually in the PowerPoint version it is animated, and the geometric shapes are keep moving. The term pattern is a bit overloaded in event processing, as noted my DEBS 2008 tutorial on this topic, but this illustration refers to the pattern which shows some combination of events, to be more accurate it is a predicate on the event history that if evaluated to the value of
"true" something should happen. This illustration was created by Tali Yazkar-Haham from IBM Haifa Research Lab as an exercise in a presentation course, and was used in dozens of presentations ever since (including presentations of some people outside IBM who typically forgot to give credit to the source).

Paul Vincent, in a continuous debate with Tim Bass, on the complex events forum, has written about "detection of new instance" and "detection of new type". While these terms make sense, I prefer not to overload the term detection and use the terms event pattern detection and event pattern discovery.

Pattern detection deals with detection that a predefined pattern has happened. This is what illustrated in the picture above. Some example can be: a patient is hooked up to a heartbeat monitor, and the physician is pre-setting a pattern "the heartbeat is monotonically increasing within 10 minutes, and the amount of increase is more than 30 during that period". This is actually a predicate over a part of the event history of a single source and type (other examples can involve multiple sources and types, but the principle is the same).



So event pattern detection is defined as detection that a predefined patterns has occurred.
This is equivalent to what Paul called: Pattern instance Detection.
In contrast, when we talk about event pattern discovery we mean that the pattern is not known in advance, and the pattern discovery function determines what is the pattern. The legend says that Archimedes discovered his famous laws about floating bodies when sitting in the bathtub and shouted: Eureka (this illustration was taken from the homepage of a company who has the word Eureka in its name, again animated in the source).
A pattern can be discovered by machine learning techniques using decision trees, statistical modeling, Bayesian Networks and numerous other methods. At the end when a pattern is discovered then it also need to be detected in reality; there are also cases in which there is a continuous detection since the patterns are changing after a short time.

Getting back to the previous example about the heartbeat, it may be the case that this pattern has bot been set by a physician, instead it was detected by some method that has looked at past events and found out that this pattern has some significance.

Most people thinking about "complex event processing" are actually talking about pattern detection, regardless of whether the patterns were composed by a human or discovered by machine learning. The illustration at the top of this page illustrates what people typically mean. As stated in the past, I don't want to get into the meaning of TLAs, and leave it to my colleagues who are doing marketing. Thus the term that I used "event pattern detection" is the more accurate one.

Another observation about the difference is that event pattern detection can be applied as COTS -- a user can use such a product, compose some patterns, hook it up to event sources, and get the pattern to be detected.

On the other hand --- while there are many tools that can help in the event pattern discovery, we cannot hook it up to event source and tell it: discover all. There is a need to do some formal modeling of the system, kind of patterns that are sought etc... In other words, this is not something that a typical developer or business analyst can do, since it requires some expertise,

It is getting late - so I'll finish at this point and return to this topic at some later point.

On Another Event Processing

This is past event that was held in Waterloo almost two years ago and dealt with "quilt hanging".

One of the interpretations of the term "event" in The Webster dictionary online is : something that happens. This one has been used by the EPTS glossary editors. But there are also some other interpretations, one of them is: social occasion or activity

Today in a meeting with some students they have attracted my attention to an interesting set of products that are processing events, however, the event they process are of the second type and not the first type. There are several companies that have products in this area, and the one name I remember is Eventful.
I Mention it just as an example, as I have not investigated this area, I understand that there are some others as well.

As a food for thought -- what is the difference between the two types of "event processing" ?

First Try:

Products in the first interpretation are typically geared towards the enterprise market, an event typically is assumed to occur within a single time-point (actually an interval reduced to a time-point), and the reason for doing it is typically one of the list that I have mentioned in a previous posting.

Products in the second interpretation are typically geared towards the consumer market through the Web, thus the business model is totally different; an event typically occurs in an interval, the participants in the events are often at the center, and the processing related to them. The motivations related to person's free time, and sometimes to social or other collaboration.

An interesting question is whether from technology point of view these two interpretations are totally distinct or have some commonality (e.g. both have producers, consumers, processing, events, subscription etc..), is this commonality interesting enough to try and generalize them, and whether there is any good reason to inter operate and mix event of the two types...