For those who read the title and grinned -- not again, discussion about what is the meaning of the term CEP, relax --- I explained in a previous posting entitled: "Is my cat CEP" , why such a discussion is futile, and I am typically consistent. BTW - when I have written that posting I did not have a cat, since than my daughter has adopted one, and he does not seem to me complex.
However, I would like to answer a more interesting question that somebody asked me recently -- what are the sources of complexity in event processing ?
In high school we have learned about "complex numbers", we liked this topic, since it was one of the most simple topics in the matriculation exam in Mathematics... Complex number is just a combination of two numbers, thus the complexity is in the structure. David Luckham also coined the term "complex events", where the complexity is also in the structure. However, there are more levels of complexity that may serve as a motivation to use COTS instead of hand-coding this functionality. What type of complexities can we observe beside the structural
complexity ?
Complexity derived from uncertainty:
- The applications specification is not known a priori and has to be discovered, example: fraud detection. This is related to the "pattern discovery" I have discussed in the previous posting.
- There are no reliable sources to obtain the desired events, or the events achieved can have uncertainty associated with them. This is a distinct complexity, since there may be the case where the application specification is well defined but the events cannot be obtained, and vice versa-- the patterns are unknown, but once discovered, the required events are easily available.
- Producer related complexities --- semantic differences among various sources, problems of time synchronization among various sources etc..
- Consumer related complexities --- similar to the producer ones, these two are, of course, orthogonal to each other, and to all other complexities.
- Interoperability complexity where various processing elements are involved.
- Complex functions requirements -- e.g. complex patterns that may involve temporal, spatial, statistical operators and combinations of them.
- Complex topology of the event processing graph, with a lot of dependencies among the various agents, which creates a complexity in validation and control.
- Complex subscription / routing decisions.
- High throughput of input events.
- High throughput of output events.
- High number of producers
- High number of consumers
- High number of event processing agents (imagine 1M agents in a single application)
- Requirement to maintain high amount of space for processing state.
- Hard real-time latency constraints.
- Compliance with QOS measurements such as threshold on average latency, threshold on percentage of events that don't comply with some latency constraint etc...
- High availability requirements.
- Dynamic, frequent changes in the logic of the event processing
- Need for programming by various types of "semi-technical" people among the business users community...
Of course, a single application may be the ultimate complex application of event processing and need ALL of these complexities, finding this application is, for sure, the dream of every researcher --- getting a lifetime of research challenges, but in reality different applications have different combinations of complexities. An application can be simple in all metrics, but have hard real time constraints, it can have very complex functionality, but no quality of service, or quantities issues. Another applications may need pattern discovery, but again the rest is simple, another combination can be relatively simple application, with complexity in quantity of producers and consumers and in semantic integration with all of them, and with the wonder of combinatorics, one can get to many more combinations....
More on complexities - later.
No comments:
Post a Comment