On agile enterprises

I have not written for a while,  I am spending much of my time in finalizing EU project proposal, yes - I know that I determined never to take upon myself coordination of EU project proposal again, but I had to re-learn and get to the same conclusion.    The proposal is due in 10 days, and then I am back to normal life, but from time to time I am also catching up on other things.  I found a posting by my colleagues from (former) ILOG, Daniel Selman about the "5 principles of the agile enterprise for 2012".  The term agile became fashionable, and as seen in the picture above is used today as a medication of various illnesses.  I first heard this term around 20 years ago in the context of agile manufacturing, later in the context of agile software development, and now, as you can see from the picture, agile enterprises.


Daniel's five principles are: 
1. Exploit historical data
2. React in time
3. Make consistent., high quality decisions
4. Performance,performance, performance....
5. Move from segments to customers 


Principles 1 and 4 relate to data -- use historical data, and exploit big data fast
Principles  2 and 2 relate to actions -- react in time (I would say even ahead of time, be proactive!) and make high quality decisions, and principle 5 talks about personalized actions rather than segmentation-oriented actions. 


All of these have technological implications and supporting technologies are certainly helpful in making an enterprise agile,  but the biggest barrier for agility in enterprises lie in the human and cultural aspects.   Using technology to change the culture is one of the next frontiers.  My recent posting about proactive thinking as a cultural change is part of this observation.  More - later. 

On Complexities and event processing

For those who read the title and grinned -- not again, discussion about what is the meaning of the term CEP, relax --- I explained in a previous posting entitled: "Is my cat CEP" , why such a discussion is futile, and I am typically consistent. BTW - when I have written that posting I did not have a cat, since than my daughter has adopted one, and he does not seem to me complex.

However, I would like to answer a more interesting question that somebody asked me recently -- what are the sources of complexity in event processing ?

In high school we have learned about "complex numbers", we liked this topic, since it was one of the most simple topics in the matriculation exam in Mathematics... Complex number is just a combination of two numbers, thus the complexity is in the structure. David Luckham also coined the term "complex events", where the complexity is also in the structure. However, there are more levels of complexity that may serve as a motivation to use COTS instead of hand-coding this functionality. What type of complexities can we observe beside the structural
complexity ?

Complexity derived from uncertainty:

• The applications specification is not known a priori and has to be discovered, example: fraud detection. This is related to the "pattern discovery" I have discussed in the previous posting.
• There are no reliable sources to obtain the desired events, or the events achieved can have uncertainty associated with them. This is a distinct complexity, since there may be the case where the application specification is well defined but the events cannot be obtained, and vice versa-- the patterns are unknown, but once discovered, the required events are easily available.

Complexity derived from connectivity:

• Producer related complexities --- semantic differences among various sources, problems of time synchronization among various sources etc..
• Consumer related complexities --- similar to the producer ones, these two are, of course, orthogonal to each other, and to all other complexities.
• Interoperability complexity where various processing elements are involved.

Complexity derived from functionality:

• Complex functions requirements -- e.g. complex patterns that may involve temporal, spatial, statistical operators and combinations of them.
• Complex topology of the event processing graph, with a lot of dependencies among the various agents, which creates a complexity in validation and control.
• Complex subscription / routing decisions.
  

Complexity derived from quantities

• High throughput of input events.
• High throughput of output events.
• High number of producers
• High number of consumers
• High number of event processing agents (imagine 1M agents in a single application)
• Requirement to maintain high amount of space for processing state.
  

Complexity derived from quality of service requirements:

• Hard real-time latency constraints.
• Compliance with QOS measurements such as threshold on average latency, threshold on percentage of events that don't comply with some latency constraint etc...
• High availability requirements.
  

Complexity derived from agility requirements

• Dynamic, frequent changes in the logic of the event processing
• Need for programming by various types of "semi-technical" people among the business users community...

I am sure that this list is not complete, but it provides some indication...

Of course, a single application may be the ultimate complex application of event processing and need ALL of these complexities, finding this application is, for sure, the dream of every researcher --- getting a lifetime of research challenges, but in reality different applications have different combinations of complexities. An application can be simple in all metrics, but have hard real time constraints, it can have very complex functionality, but no quality of service, or quantities issues. Another applications may need pattern discovery, but again the rest is simple, another combination can be relatively simple application, with complexity in quantity of producers and consumers and in semantic integration with all of them, and with the wonder of combinatorics, one can get to many more combinations....

More on complexities - later.

On cost-effective EP when high performance is not required

Brian Connell has asked me to provide an example in which Complex Event Processing can be used when low latency is not required and it is still be better than other solutions.

I have already referred to this issue in the past. There are several concrete examples that come to mind - here is one (real system):

An enterprise has internal regulations to handle suppliers, these requirements relate to the interaction with suppliers, that may be represented as events, and involve actions that must be done, or should not be done, within certain amount of time, or until some event happens. The application is to monitor the compliance with these internal regulation in audit mode, meaning the results are alerts (which are also treated as events, since there are time-outs from alert sending) and not direct interference in the business processes.

The throughput is far below what is considered as "high throughput" and it is several thousands events per day. The latency is also not required to be extremely low -- if the alert will be issued within a minute or two, it is still very fast auditing. It also does not require any analytics of intelligent procedures, since the regulations are given and deterministic

What are the benefits for the customer to use EP software and not any other solution?

First - some of the regulations are fairly complex - writing them in hard-code can be time and resource consuming, putting everything on databases and using SQL was also considered - but some of the regulations are not easy to express in SQL.

Second - Regulations tend to change frequently, the users wish to control these changes, and getting them through the slow IT development cycle will delay the introduction of the change.

So in this case the customer's motivation are - agility and de-complexity; more later

Hype vs. value -- a constructive view

Besides free advertising to an e-book, the illustration above - together with the title indicates that I am going to write something constructive - in the previous posting on "bitter pills" I have provoked the evil spirits and got sick with a flu, so with some almost sleepless nights, I have been too tired to Blog -- today is much better - so back in business. There are several people complaining about -- we need to get from hype to more impact on business, something is lacking today etc --- all true ! I also said that we need to take a constructive view - not only complain, but see what should be done. So here is an initial attempt to do it, actually nothing new here, just listing some observations done by various people:

1. More packaged applications based on EP technology should be constructed -- the model of using an EP tool as an application development tool covers only small part of the potential market, since, in the end of the day, business executives are interested in applications and not in technology.
2. Enable business users (non-IT developers) to control the behavior (e.g. define, modify, compose patterns/rules), since agility is an important expectation of the customers
3. Learn how to articulate the business vale -- yes we all know "threats and opportunities" but this is somewhat too abstract for the decision makers -- the business value is also not unique, there are various types of applications with different business values, and explaining the right one is crucial - more thoughts about the various types of business values -- in one of the next postings.
4. Advance on standards -- customers don't like proprietary.

Of course - this is only the initial list, not even talking about advances in technology, to start the discussion - more later.

On Event Processing and Now.

After some more technical postings, back to macro-level issue. David Luckham has written recently in his website about the history of CEP (I'll refer to the content of this article in another posting). All indications are the event processing is not a new thing, however, as some people indicate, there are a lot of interest, events, maybe hype around it NOW - the question is what is new ?



The first observation is that unlike the past, today there are commercial "on the shelf" products whose main purpose is to provide event processing capabilities, while in the past there were event processing capabilities in other types of products (simulation, databases, network and system management, middleware, real-time systems etc..), there is also a start of event processing as a discipline. What are the reasons for the interest now - some of what happened in the last few years that supported the shift from hard-coded event processing functionality to COTS are (based on discussions with people in multiple industries):

• Some contemporary applications which are event-driven by nature - such as: compliance with regulations, the need to detect frauds as two examples - have become pervasive in multiple industries
• The increasing complexity of inter-process integration that is simplified with event-driven interaction
• The need for flexibility and agility to gain market advantage in different areas - thus, move away from hard-coded solutions.
• The substantial growth in the number of available events - e.g. since RFID technology became pervasive.
• Some industry trends like - BAM, RTE, "on demand" - that are also based on responding to events.
• The drive to save expenses in back offices by automating exception handling - trends like STP.

I am sure that there are more of these.

Thus, while event processing functionality is not new - "event processing" as a first class citizen in the computing world - with its own dedicated products, community and emerging discipline is new. More - Later

"The only motivation to use EP COTS is to cope with high performance requirements" - true or false ?

Somehow, I find myself using my collection of analysts' presentations to help me make some points, this time, I am showing a slide from Roy Schulte's presentation in the Gartner EPS summit - I'll return to the content of this slide shortly, but will start with some discussion I had yesterday about the reasons that enterprises are using COTS for event processing. I have heard (and not in the first time) the assertion - the only reason that one will want to use EP software and not hard-coded solution is the ability to cope with high throughput / low latency - in short, to deal with high performance requirements. If there are no high performance requirements there are other solutions, e.g. the database guys think that in this case one can insert all events to a database and use simple SQL queries for CEP patterns, or just using the good old C/Java programming for this purpose. This is somewhat inconsistent with my own experience, where customers that did not have "high performance" requirements were eager to use CEP technologies. Indeed, high performance is a reason to use CEP COTS, however, as indicated in Roy Schulte's slide above - it is actually a minor reason. According to Gartner, the high end is somewhere between 5-10 percent of the candidate applications, while looking at the prediction for 2012 - the use in the high end will be 8% - out of the 27% total use; note also that Roy Schulte defines the high end as 250 events per second, which is really far from the definition of "high performance", so the numbers are even lower. It seems that the market for "non high performance CEP" is much larger, and will grow faster. If that's so - where does the misconception that EP equals high performance always ? I think there are two sources - the first, the early adopters were from the capital markets industry, where some (not all !) of the candidate applications has indeed high performance characteristics. However, with the growth of the market, and use of EP software in other applications and other industries, these type of applications, while continue to grow, will not match the higher growth of non high performance applications. The other reason is that some vendors make the high performance as their main message, and trying to get the market believe that this is indeed the most important property.

So - if high performance is not the only reason to use EP COTS what are the other reasons to use EP COTS ? this is a matter for investigation, but IMHO the main one is the "high level programming" and agility - in short - the ability to reduce the Total Cost of Ownership.

I'll provide more insights about the TCO issue in a future post.