James Taylor on top 10 excuses to avoid business rules

James Taylor has recently written series of posts in his Blog on 10 top excuses to avoid business rules.    These excuses are brought mainly from the IT people point of view, among them are: business users don't want it,  you want business user to program?, we're doing fine, we tried before and failed and more...

I wonder if these blogs are motivated by the author's observation that the business rules area doesn't get enough traction?   for each of the 10 excuses, James has a counter-argument.   I guess that the question here is more general about the ability of business users (= non programmers) to develop any type of  applications - in sense of expressing and customizing business logic.   Business rules is an instance of this.

I am looking for any research work about the success factors, approaches and limitations for this general questions -- will share insights when I'll have them.  More - later. 

In search of intention language for event processing: revisiting the decision model

I have written recently about the need to establish intention language that will serve as higher level abstractions for event processing.  Last week, I had almost a day without laptop, when my laptop went to the system guys for re-installation of Windows (and thus all the software), due to some repeating failures.  I used one hour for my daily walk, some of the time to bother other people in the team, and had some time left for reading, so I decided to return to the copy of the "decision model" book that I have in my office, something that I had in mind doing since the visit of Larry Goldberg in my lab.    The decision model provides a tabular way of looking at decisions, with various propositions that are intended to enforce consistency, and normalization forms inspired from the relational model.    This expresses decisions that are assertion based,  and are intedned to serve as a semantic layer abouve business rules.   The event processing area is somewhat richer, as decisions can be based on events but also on situations that are based on detection of event patterns, or on aggregation or composition of events.  The results of event processing are derived events that may either trigger an action or issue a notification (which is also a kind of action).   The interesting question is whether this relatively simple tabular model can enrich its semantics in order to express the intentions behind event processing functions.    Getting such a cannonic way may be quite appealing,  however I am not sure it is going to be easy -- one direction to consider in search of intention language for event processing -- more on this one later. 

On IBM Websphere Decision Server

IBM recently announced "Websphere Decision Server".  This announcement states that this offering combines business event processing with business rules management system to accelerate decision making.


There are several ways in which event processing and business rules interact, some of them are: an event, derived by pattern detection, triggers a rule, which makes it event-driven decision; the other direction is also valid:  an execution of rules brings into decision and this decision can be reported as an event and may influence other decisions.  Triggering decisions is one of the major uses of event processing (of course it is not the only one,  e.g. event processing can be used for diagnosis or information dissemination), and it is a component in the automated decision making process, but these components mainly exist within islands (EP, business rules, optimization software and other means of decision support systems).    One of the areas we are working on (we had a short report on it within the "fast abstract" session of DEBS 2010, as we were not in a position yet to deliver a full paper) is a unified conceptual model of event processing and business rules, where both are generalized as decision agents.  this is still in the research phases, and not part of the product, nevertheless this announced offering provides step forward in achieving such an integration.   More synergies in the decision space are expected.     

The decision model - Larry Goldberg's talk

A few months ago I have written a review about "THE DECISION MODEL" book. Today I have hosted in the Haifa Research Lab one of the authors, Larry Goldberg, who gave us a live talk about the model. The decision model provides table oriented representation of rules, where a table designates a family of rules which share the same consequence (i.e. the "right hand side" assigns value to the same fact), tables are connected in a way that a fact that is an outcome of one table is an input for another table, this brings some order to rules, and also fits both inference rules or computational rules. I see strong benefit of using such structured way, and also possible to build an hybrid flow of "rule families" and "event processing networks". I think that there is a future there. I was involved in the past in some similar effort to provide some structure to data-driven rules, a short ACM SIGMOD RECORD paper is referenced here.

I'll have a short presentation in DEBS 2010 around the relations between event processing and business rules (in the fast abstract session).

Book review: The Decision Model

The last package from Amazon brought me the book entitled: "The Decision Model: A Business Logic Framework Linking Business and Technology" by Barbara von Halle and Larry Goldberg.

I have read a draft of the book before, at Barb's request, and wrote a review, from which one line was quoted on the back cover; I believe that the trend of modeling decisions and look at them in perspective of higher level abstractions will become more pervasive, and I view technologies like business rules, event processing and various analytics as building blocks in decision platforms that are going to be notable part of enterprise computing and managing much of the operational decisions. The book has three sections:

Section I puts the decision model in context, explains what is decision model, providing a background comparing decision models with data models, and positioning decision model in the SOA and BPM universe, it also explains the business value. This section is intended mostly for business users and managers that want to get an overview.

Section II explains the decision model in detail, discussing the structural, declarative and integrity principles, and comparing the decision model to the relational model, a motive repeating in previous books by Von Halle. There is even a chapter that is called "The decision model formally defined", but the formalization is in terms of explanations and tables, and not by formal writing, which I guess fits the target audience.

Section III is called "Commentaries" and is actually a collection of articles by the authors as well as by various people active in this space (John Zachman, James Taylor, Bruce Silver and more) discussing specific related issues such as: relations to enterprise decision management, standards, business decision maturity model.

Event based decisions and event processing are mentioned several times within the book, but are not thoroughly discussed. The focus is on facts and rules kind of terminology; a combined model that combines both rules and events is a natural extension, from the point of view of this decision model as well as from the point of view of event processing modeling. I have written before about decision agents, and since that time advanced on the thinking about such a decision agents framework. I'll revisit this issue in one of the following postings.

Bottom line -- the decision model book is a very good book to explain the book to various types of readers (the introduction maps the chapters of the book to the various types of users) and possible basis for both pragmatic foundations of rules technology, as well as a possible basis for a more formal basis for extended decision agents framework. More on this topic -- later.

On Decision Agents

Decisions are part of the enterprise life as well as the life of every individual. Take a decision that many of the readers have experienced: naming children. My wife and myself have realized in very early phase that we have completely different taste in names, so we have decided to agree on a protocol for how names are selected: taking turns - one of us is making a list of five names, and the other selects a name from this list. The selection is done only after the birth, and the list can be modified until the selection made. To be fair we need even number of children so each will play any of the roles equal amount of times (indeed we have four children which satisfies this requirement). This is actually work, none of us got the first priority, and none of us have to tolerate a name we really hate.

In the paragraph above I have described a manual decision, but increasingly decisions are made by computer software which makes or recommends decisions. James Taylor is constantly talking about "Decision Agents". I thought that it will be interested to look at this notion and discuss how it is related to event processing.

We can say that a "decision agent" has various properties that we can qualify as answer to questions:

• Why ? what is the reason that the decision agent activated. In the example above -- the birth of a child.
• which ? which information is needed in order to make this decision. In the example above --- the list of five names (which have been obtained by another decision or collection of decisions) .
• How? How the decision is made. In the example above --- by some human cognitive process, which may have reflection in a computerized decision agent.

Back to our world of event processing, we are using the term "event processing agent" to denote a software artifact that gets one or more event as an input, does something, and produces one or more events as an output.

The question is what is the relationship between decision agent and event processing agent ?


In a (not very recent) postings, Carole-An Matignon from Fair Isaac has attempted to demystify some terms. She used an analogy to the human body, saying that BRMS is the brain, while event processing is the sensor for the brain to get the decisions. So is event processing agent is a sensing agent ?

The answer is --- the terms decision agent and event processing agent do intersect, but none of them subsumes the other.

Returning to the decision agent questions.

The why question: A decision may be required since some situation has occurred, because some relevant fact has changed, or because somebody made an explicit request to activate the decision agent. In the first case (a situation has occurred), then this situation may be a simple event, but also may be a leaf of an event processing network. In this case, there may be some event processing agents that are part of the decision to activate the agent, so it is part of the brain.

The which question: Here again the information needed can vary -- it may relate to the present state, to the history of states, and transitions. Event processing agents can be used to prepare the required information, by taking events and filter, transform, enrich, aggregate, split and more. In this case the event processing agent is indeed a sensor, creating input for the decision.

The how question: There are various techniques to get a decision, detecting patterns on the event history may be a method to obtain a decision, together with other techniques, such as inferring from facts and rules, applying stochastic decision reasoning and more. This does not say that every event processing agent which performs "pattern detection" is indeed a decision agent, sometimes it just derive event that will be used directly or indirectly as input to a decision.
Interestingly, this is true for business rules as well. A business rule may derive new fact, the fact itself is not a decision, for example, it may be classification of a customer based on demographic information. The output of this rule is used as input to (another) decision agent. So BRMS like event processing can also play the role of both sensors and brain.

To summarize:

• An Event Processing Agent may be a Decision Agent, or provider of input or trigger to other decision agents.
• The same statement is also true for "state processing" business rule.
  
• A decision agent may be Event Processing Agent, but also can consist of several other types of agents.
• There may be blend of decision agents of various types inter-related. I'll write more in the future about this assertion.
  

Welcome the ILOG team to the blue giant

Recently, the acquisition of ILOG by the blue giant (IBM, the company who pays my salary) has been completed.
It was interesting to read the InformationWeek article which had event processing in the title of this article.

ILOG is, of course, well known for its business rules product ( I had some history in this area in my past...).

There are various relationships between business rules and event processing technologies, and there are also various opinions about the various relationships.... I have blogged a year ago about this issue.

While I looked at business rules from EP perspective, Some ILOG guys also investigated the opposite direction, in the last EPTS meeting we had met Pierre-Henry Clouin, who blogged about it , prior to the EPTS meeting. Now we can continue the short discussion we had in Stamford .

Another ILOG guy who had a guest posting about EP issues is Changhai Ke.

Joining a big corporate is certainly a cultural change (shock for some), but also opens some horizons, so I wish Pierre-Henry, Changhai and the rest of the ILOG team a smooth transition and we'll certainly meet in one of the blue corridors.... Welcome on board.

On basic classification of terms

Besides the fact that there is still a sort of uncertainty among the general IT community about what is event processing (or CEP, which is the most common TLA associated with it), some discussion on the Blogsphere lead to further confusion --- e.g. claiming that CEP equals BRMS. Carole-Ann Matignon,VP, Product Management at Fair Isaac, has nice posting in her Blog, talking about this confusion, and providing what she calls simplistic view of the definitions of BPM, BRMS, and CEP. I'll go with the simplistic direction of thinking and claim that this is a mix of terms from three different domains -- application, technique and function.

Function answers the question --- what is being done ?
Technique answers the question -- how something being done ?
Application answers the question --- what is the problem being solved ?

Examples

• Business Activity Monitoring (BAM) is an application type, it solves the problem of controlling the business activities in order to optimize the business, deal with exceptions etc...

• Business Rules are type of technique --- which can be used to infer facts from other facts or rules (inference rules) , or to determine action when event occurs and condition is satisfied (ECA rules) and more (there are at least half a dozen types of rules, which are techniques to do something).

• Event Processing is really a set of functions which does what the name indicates -- process events --- processing can be filtering, transforming, enriching, routing, detect patterns, deriving and some more.

Of course there are inter-relationships among them --- some types of business rules are a non exclusive way to execute some of the functions of event processing -- e.g. ECA rules can be used for routing, inference rules can be used for event derivation and more. However, there are other techniques that can be used for each of this functions. Likewise, a BAM application may use event processing functions as part of its implementation, but may also use other techniques (e.g. being data-driven rather than event-driven and do all its processing in retrospect, looking at committed data periodically and not on events as data-in-motion). Business Rules can be used for various utilization in BAM and in BPM applications, with or without the use of event processing.

The functions of event processing is typically used for several motivations:

• Observation into business processes, business activity monitoring.
• Dynamic reaction and modification to transactions and business processes
• Diagnosis of problems and finding root-cause.
• Prediction of future problematic events that need to be eliminated or mitigated
• Dissemination of data in motion to arrive to the right person at the right time in the right granularity.
  

These motivations are not applications --- there are multiple applications that perform diagnosis -- system management, car repairs, medicine diagnosis, oil drilling management; there are also multiple applications that perform more than one of these -- e.g. observation and reaction or prediction and reaction. I have blogged before about the metaphor of blind people touching an elephant -- there are some people who are looking at a single line of applications, or a single implementation techniques and saying CEP is X, while another person who is thinking on another concept is saying the CEP is Y, thus a confusion is being created - the following elephant's picture illustrates it (taken from the original postings).


Bottom line: the confusion is a result of equating terms from various classes, and resolving this confusion is indeed very simple.

On Decisions and Event Processing

Saturday morning at home, after busy week and one resting day (our weekend is Friday and Saturday, tomorrow - Sunday, is again long and busy working day...) I am back in the Blogland.
Today I am in reactive mood, so will join a discussion thread that started by James Taylor who seems to have discovered that "event rules" and "decision rules" are not the same. Since this has been a reaction to a previous posting by Marco Siero, Marco actually agrees with him, saying that event rules and decision rules are not the same - saying that the "event rules" are in fact activation rules - saying WHEN a decision is needed. If they both agree then the thread of discussion can end.. -- nevertheless, I have some comments to this discussion:

1. The word "rules" is overloaded - there are many types of rules. Some rules deal with obtaining more facts (if the rules are based on logic it is called inference, and if the rules are based on data models it is called derivation), some with activating actions. Moreover, there are rules which are assertions about what is allowed and what is not (constraints) which are none of the above.

2. People have used the term "rules" to denote -- detect a pattern in events --- due to several reasons, one of them seems that rule has been the most closely known term that they could anchor at.

3. Today, when "event processing" is known by its own right, the use of term rules, when it is not needed is just confusing, and it is better to use the correct terms for event processing types.

4. I personally used the term "rules" for several years, but in the last three years have avoided doing it. It is difficult to get people change terminology, but sometimes it is required.

This, however, was just a comment on the term "rule", but let's get to the more interesting part -- as the title of the posting hints -- relationships between decisions and event processing.

The active database people have coined the term "active rule" which is also known as ECA rule, following its parts (event-condition-action). One can look at event processing network as an extension of ECA, in the sense that there is an entire processing done to get to the event that we actually want to react to, a raw event can be filtered, translated, aggregated, enriched, and
combined with other events that satisfy a certain pattern -- at then the event at the edge of the network (AKA situation) is the one that triggers the action.

The question whether an event processing application is doing decision or just notification that decision is needed, is not dependent on the event processing part - but on the type of action, if the action is notification -- notify somebody that the situation has been detected, I guess that it is what Marco meant in "activation rules", however, if the action is an automatic action (or several possible automatic actions based on conditions), then the result of the event processing is certainly a decision. There are cases in which the event processing ends in notification, e.g. most cases of BAM, and there are cases in which event processing ends in automated actions, e.g. send order to renew inventory, block a certain financial transaction and many other examples.

It also should be noted that the process of "event processing" may include decisions about the detection process itself -- examples: diagnosis, fraud detection - require application of decision components in order to determine what is the situation that occurred, which is also manifests itself in the decision.

James Taylor makes more interesting assertion: Event rules are not the same as the rules behind business decisions, even if the same language can be used to express both. One is tied to the events and the event handling system, one is not. I am not sure what is "tied" to the events, as said, the "action" takes part outside the event processing network, and it certainly can be a case of hybrid system, in which an event processing network detects what happened and the action attached to it is a "business rule" that decides what the reaction should be based on the situation. More- Later.

On Forward and Backward CEP

Paul Vincent from TIBCO has recently posted some thoughts on "goal oriented event processing" . As usual, Paul sees event processing during the " through the rule spectacles, so I'll try to predent it in a wider perspective. It is agreed that the role of CEP (which by itself is part of larger EP) is to detect patterns over multiple events. What is forward and backwards in this context ?

• Forward: For each event -- check if this event completes some pattern
• Backward: Given a pattern -- check if this pattern have been satisified (in some time context) in the past.

Both are useful for different cases, and sometimes we need a mix; looking deeper at the differences:

• Forward is always event driven (it is done as a reaction to event)
• Backward is request-response (it is done as part of explicit request), note that the request may by itself be trigerred by an event that makes it hybrid model.
• Forward is used when patterns need to be detected immediately, it is most cost-effective to optimize for all patterns that events may complete then to look for all patterns and see if they are completed. Example: detect arbitrage opporunity between two exchanges.
• Backwards is used when patterns are done periodically, or as ad-hoc queries. Example: At the end of each month, Find all stocks that during the last month have satisfied the following conditions: The stock closing values at the end of the day were strictly increasing over a period of five consecutive working days, anywhere during this month; The stock value in the beginning at the end of the five days value was at least 30 percent more than its value at the beginning of the five days period.
• Mixed is used when in some cases -- a pattern may need "reinforcement" in past patterns -- Example: A person that has deposited (in aggregate) more than $20,000 within a single working day is a SUSPECT in money laundering. To reinforce the suspicion the following retrospective patterns are sought:
  There has been a period of week within the last year in which the same person has deposited (in aggregate) $50,000 or more and has withdrawn (in aggregate) at least $50,000 within the same week.
  The same person has already been a "suspect" according to this definition within the last 30 business days.
  If any of these patterns are satisfied – the event "confirmed suspect" is derived.

Forward and backward pattern detection can, of course, be implemented in various ways - business rules, SQL (stream and regular queries as backward), Scripts and specialized patter-oriented implementation.

More thoughts on Rules in the context of Event Processing

I spent today (and tomorrow) in the IBM Hursley Lab - my second home in the last couple of years, this is a picture of the "Hursley House" (well - from the back side) - a countryside English manor that serves today as place for meetings and conferences, and the office of the Lab Director - besides this building there is a complex of connected building with multiple systems for room numbering that can provide in-door navigation challenges. I'll write today about some ideas that came out from discussions in the CITT meeting earlier this week about the role of rule technology. I still hold my opinion that although it is possible to take a technology that has been developed for one purpose and "hack" it to use it to other purpose, it may not be the most natural/effective/efficient way to do. This is true for SQL as well as rules when we are talking about pattern detection in complex event processing (I am working now on tutorial on the issue of patterns). However, this does not say that rule technology does not have a place in event processing in general. Here are some places it can be used:

(1). Decision-based routing in event processing networks.

(2). Transformation of events.

(3). Validation of events.

(4). Orchestration rules.

(5). Intelligent Event Processing.

Note that different type of rules are being used for the different cases -

for routing and transformation - it is typically - if-then rules/decision trees/decision tables.

for validation - constraint oriented rules.

for orchestration - ECA rules (note that orchestration rules are in the domain of the consumer that receives an event from the event processing network and has to decide what to do with it).

for intelligent event processing - all type of rules - deductive, inductive, abductive, rules with uncertainty - can play in different cases.

More about ECA rules and event processing - soon.

Agnon, the dog, playing and downplaying

My literature teacher in the 7th grade has told the class one day a story about her days in as a university student; one of the literature classes in the university has been dedicated to Agnon, the only Hebrew writer who received Nobel Prize in Literature (Agnon's picture is in the left hand side). One of the famous stories of Agnon has a dog (such as the one in the right-hand side) as its hero; thus, the class of students at the university had discussed this story and had a debate about what does the dog represents, one group of students thought that dog is a symbol for Agnon itself, and one thought that the dog is a symbol for the entire Jewish people. Since Agnon was still alive those days, they have decided to send a couple of students to meet Agnon, set up a meeting in his house (well - Email questions were not on the radar at that time), and explained the dilemma - what does the dog represent? Agnon was somewhat amused to hear about the debate, and told them briefly - "I don't know what you are talking about, the dog represents a dog".

Going back for the childhood Reminiscences to the contemporary issues - I have read with interest the recent blog of Paul Vincent about "CEP vs. Business Rules" in which he mentioned me by reference (and not by name) saying: "Others seem to downplay the idea of rules for CEP". I have a substantial respect for Paul, and generally agree with most of the assertions he made in that blog. Let me add some more insights into this issue - I am using this blog to reflect about event processing, and make the point the "event processing" is a discipline that stands on its own feet, and is not a footnote to anything else - be it databases or business rules.

CEP functionality can indeed be implemented in either SQL or in an extension to business rule language as some of the vendors successfully demonstrated. It can also be implemented in half a dozen different ways (at least). Since I am trying to investigate "event processing" as a discipline, and not a specific implementation, thus is the play, and indeed I downplay specific implementations for now, and try to think on the fundamental issues. I may return in future blogs to talk about what type of implementations fit different types of functions/applications - which is a difficult question, and I don't really see that there is an issue of "CEP vs. Business rules" as these are conceptually orthogonal issues (and I agree that in practice there are various relations), and Paul's blog does a tremendous job of highlighting the differences and relations between the two.

For now - my dog is "event processing" and it represents "event processing" and not other concepts that are not fundamentally necessary to understand what "event processing" is.

More - Later.

More about event processing and business rules

Since my previous post on this topic has triggered some discussion, comments and questions - I am returning to the issue of "event processing and business rules". One observation I got today is that if we'll some of the "event processing typical applications", example: Paul Vincent's post about EDA and CEP in Insurance are considered by the business rule people also as typical applications for business rule engine (and the insurance area has strong presence of business rules). To answer that I'll cite another CEP blogger, whom I don't have the honor to know personally, who used the eternal-wisdom title: When all you have is a hammer everything looks like a nail, while the post talks about SQL, this is true also for business rules. In fact, if some programming tool has enough expressive power, one can, somehow, express everything using this tool, which does not say a lot, since everything can be expressed nicely by a Turing machine (this wikipedia link also points at some freeware implementation of Turing machine, so we have a single programming tool to do everything...). However, there are things that are naturally fit into a certain programming paradigm and things that look rather awkward. Of course, there are some cases in which one can equally implement it within multiple paradigms, this is typically one of two cases - it may be very simple, or there is not (yet) a specific paradigm to express it nicely, so multiple paradigms are awkward in the same way.
It is not surprising that the same phrase about hammer and nails is also going in the opposite directions- I have seen cases where people who had CEP engines, "force" them to do things that are not natural for this paradigm, by creating artificially events and patterns.

And one comment about ECA rules (Event-Condition-Action) --- the contribution of the glorious "active databases" community to the computing industry. Complex Event Processing deals with the E part - taking events, apply some function, and at the end deriving more events. Each derived event that arrive to the "edge of the event processing network", i.e. sent to a consumer, can then conditionally trigger an action. Thus, from architecture point of view, ECA rules are really part of the consumer logic, and not part of the "event processing network". More - later.

event processing and business rules

New week is starting after all holidays (in Israel the working week is Sunday - Thursday), so let's start it with writing something in the Blog. I have stated that SQL is not the ideal way to express event processing functionality, so, people ask me what is the alternative - business rules ? the answer is - not really. The term business rules is somewhat overloaded, however, we can group the rule types into two types:

• knowledge-creation rules - rules that create more 'knowledge" - facts, data-elements etc... These rules can use various inference techniques - deduction, induction, abduction...
• behavioral rules - rules that cause something to be done, or prevent something from happen.

The EPL should include constructs with some similarity to these two types, but this does not say that existing rule languages can be taken as a basis, this is because the domain in event processing is events (not facts or data), and that event has a distinct semantics.

• The first type of construct in EPL is the "event derivation rules" which includes transformations (filtering, translation, aggregation, split, and enrichment) - these typically transform an event (aggregation is a special case, we'll dedicate a blog to it), and "pattern based derivation" which derives events, such that the instances of the derivers are selected as part of a detected pattern. We can see that all these types require specialized semantics, thus while there is similarity in ideas - it is not really rule semantics.
• The second is somewhat similar to the "orchestration" constructs of EPL - this is being performed in the leaf of the network - after all derivations occurred, the result event may (conditionally) perform some action related to a consumer (notify, orchestrate etc...), which is also known as ECA rules (event-condition-action). Note, that from architecture point of view, ECA rules are not done in the event processing network (i.e. they are not MEP or CEP), but they occur in the territory of the consumer - at the border of the EPN.

An interesting observation id that Complex Event Processing (CEP) as defined in the glossary, deals with derivation of new events, it is neither an instance nor an extension of ECA rules, since CEP deals with "knowledge creation" and ECA deals with behavior.

Some more discussion on this topics, and relation with "reaction rules" - later.