Today, Friday, is part of our weekend, so it is a good time to do shopping and other arrangements.
My wife and myself went to our local friendly bank to open some new account for some purpose. The lady that handles our account said that they have a new software to open an account that is extremely difficult to operate, with a lot of screens that one has to understand what is asked, and suggested she'll do it off-line and call us when ready, so we'll come to sign the papers. Once, opening an account was simple and lasted a few minutes, just signing some forms; the more sophisticated a software becomes, the more difficult it to operate, and sometimes it becomes obstacle to the business. Often, developers don't really care about the human engineering aspects. Hans Gilde wrote recently about the fact that CEP software is not smart. I agree, in several occasions I have given talks to an audience of high-school students which gives a rough introduction to AI, under the title: can a computer think ? while there some works in AI that strive to do it, today's software does cannot really think, and is not really smart. One can use the software to do things that look smart, but the wisdom is not in the software itself, it is in the way it is used. In the bank case, the software does not even look smart...
This week I had three visitors from Germany, Rainer von Ammon and two of his CITT colleagues, and we made some progress towards defining the EDBPM project that we plan to submit as EU project. They have asked me to pose in my office under my " wall of plaques" (half of them are in Hebrew, so they could not really read them...). So this is my most current picture..
One short clarification -- after my posting entitled : "event processing platforms - yes, but..."
I received some private communication claiming that there is a confusion between the terms "platforms" and "engines". The claim is that there are vendors who refer to their engines as platforms, moreover, some people refer to any run-time software as an engine. So I thought it worth clarifying how do I see the distinction:
- Event Processing Platform is a software that enables the creation of event processing network, handle the routing of events among agents, management, and other common infrastructure issues.
- Event Processing Engine is a software that enables the creation of the actual function - in the EPN term implementing agents.
This is similar to the difference between an application server and a single component.
What is the connection ---
- On one extreme, there are closed platforms, i.e. platform that can run only one type of engines, in this case the distinction becomes more fuzzy.
- On the other extreme -- there are open platforms, in this case these concepts are totally separated, a platform that can run multiple engines. The main issue about it is that there may be a collection of different languages that come with the different engines, and this may make the development of an application more difficult.
The first generation of event processing has started with engines that are stand-alone, the emergence of platforms, and making them open, are the signs of the second generation. I'll say more about the challenges of constructing the next generations -- more later.
The February issue of the journal "International Banking Systems" has an article called Event Horizon written by James Ling. I have tried to get soft copy of this article, but the journal has strict policy that only subscribers can have access to soft copies, and agreed to send me hard copy only; the hard copy arrived today and I am looking at it now. I have been interviewed for this article on my hat as EPTS chair, and also there were some other people from the EP community who have been interviewed, such as: John Morrell from Coral8, Jiles Nelson from Apama and Jeff Wooton from Aleri to name a few, and also several customers of EP technologies in the financial services sector. Some interesting points from the article:
- EPTS has been mentioned in length including quotes from its charter; it seems that EPTS is getting traction as both authority source and representative of the community
- Various of EP applications are mentioned besides algorithmic trading are: external surveillance by regulators, risk management, auditing, market depth analysis and more.
- It is mentioned that there is a confusion around the distinction between event processing and BRMS, saying that some people say this is exactly the same thing. They quote me as saying that they are complementary technologies, but did not cite my explanation, and prefer to leave the reader in the dark about this issue.
- They mention that most vendors are using the term CEP, but stick to the EP name throughout the article.
- The concluding remarks are: "EP is certainly one of those technologies that will play an important role in the future of financial services".
This is a picture of Akko (AKA Acre), an ancient city with walls, middle eastern typical market with the smells of spices, an fisherman's harbor. I am hosting now some visitors from Germany, Rainer von Ammon and his colleagues from CITT to discuss some collaboration topics including a consortium for an EU project that we are establishing with other partners. Unfortunately they chose to arrive in the most rainy period we have this year, so we could not do much sightseeing today, however, we succeeded to get two hours break in the rain to stroll around the old city of Akko.
I'll get back to the discussion about the questions I posed yesterday soon, as I would like to see if more people want to react before stating my opinion (I am in learning mode...).
Today I'll write something about Information Fusion and its relationship to event processing; I came across a recent survey article in ACM computing surveys about data fusion.
There are various kind of fusions - data fusion, information fusion and sensor fusion -- and all of them are intended to get information from distinct sources blend it together and understand what has happened. A very simple example of sensor fusion is in traffic monitoring, there is a sensor that senses the speed of a car, there is a camera that takes pictures of the car and its license plate, fusion of both can identify the fact that a certain car has violated the speed laws, this is a relatively simple case that requires some basic image processing, but it is quite easy to determine what happened. This is, of course, very simple case, and in the area of military intelligence it is much more complicated to understand what happened / happening / going to happen and some techniques are being used. The Center for Multi source information fusion in University of Buffalo maintains a site with collection of bibliography about fusion issues including tutorials and their proposals to modify the relatively old JDL model, so you can find much more information there.
So where is the confusion ? --- there are people who confuse event processing with some other different areas, somebody in IBM who saw an illustration of event processing network once tried to convince me that we are re-inventing workflows, some data management people think that event processing is just a footnote to existing query processing, everyone with a hammer looks at the rest of the world as a bunch of nails; Likewise, there are people who confuse fusion with event processing.
So what is the infusion? the fact of the matter is that information fusion and event processing are complementary technologies. The goal of fusion is to determine what happened, i.e. to determine what is the event that has occurred. Event processing is processing the event after somebody determined that the event happened it has multiple goals, the techniques are different, fusion is using conflict resolution techniques and stochastic modeling, event processing is using pattern matching, transformation, aggregation etc. Thus an event can be created using fusion techniques and then processed using an event processing system -- this is the infusion.
However -- there is also a potential synergies between these two applications - a partnership of fusion technology as a preprocessor for events and event processing can be beneficial for certain applications, this is the most obvious synergy. Another type of synergy is that techniques used in fusion can be used in event processing and vice versa, this is an interesting direction to investigate further and also investigate possible real applications for it. More on this - later.
I am sitting now in my living room, watching the heavy rain outside.Rainy; We did not get a lot of rain this winter, however when the rain comes it tends to come in the wrong times, like weekends (Friday is part of our weekend which is Friday and Saturday, Sunday is a normal work day, it is used to catch up on things since in this day our colleagues from abroad are idle and there are no conference call and other interactions).
Today I would like to concentrate on the question --- what level in the Enterprise is event processing for, I had a recent discussion with somebody who investigated the BRMS market and asked him this question about BRMS, and the answer was the mostly BRMS products concentrate in the operational level, the typical example of BRMS is assignment of rate to insurance policy, which is clearly an operational decision. What is the situation in event processing ? There is a famous analyst presentation that talked about "detecting threats and opportunities" as part of the ROI for event processing pattern matching. Let's examine what's behind this title. Sometimes there are risks in the operational level, such as security attacks, but since this presentation have concentrated in the business area and not the IT, the meaning has been seeking opportunities for the business and mitigating risks for the business, which is beyond the operational level, such detection is probably in the tactical level, but the outcome can flow to the strategic level, since there may not be an answer to specific threat or opportunity within the current strategy. On the other hand, event processing is also associated with being done on-line (what some people call "real time" or "near real time" when they are not sure if it is really "real time", which is even worse as a term).
Some interesting questions on this topic are:
1. Whether organizations are really doing tactical and strategic decisions on-line ? in the illustration on the top of the page taken from the Microsoft site
the authors believe that tactical decisions is matters of days to months, and strategic decisions are done in resolution of quarters or years. Is there benefits/feasibility to change it online ?
2. Do we need different variations of event processing for the different levels ?
3. Is the semantics of events the same among all levels ?
4. Are there different complementary technologies, and different platforms in the different levels, or can we look at a single event processing system across levels?
Today I am just posting the question, will try to address each of these questions in subsequents postings.
EPTS moved to an hyperactive phase with the launch of the six working groups. This has been also an opportunity to gain more members, as membership in the working groups is a privilige provided for EPTS members. There are several more members that belong to most of the different communities --- academic peo0le, consultants, customers and vednors (no new analysts...). I am especially glad to mention that CA has recently joined EPTS; I welcome CA as an expereinced company whose main line relates to managing infrastructure events, and knowing some of its people that have already been involved in the community, I am sure that they can make a substantial contribution to the technical society.
In fact, any new member interested to contribute to the understanding and progress of the event processing area on multiple fronts, is most welcome to join. There is a lot of action and place for contribution and innovation. Event Processing is a relatively young discipline; disciplines progress is accelerated with a community work. I am confident that the activities done today with a lot of excellent people who invest time and energy to contribute, will make a substantial impact on the universe. If you are interested to join EPTS, please follow the instructions
Somebody who looked at the public EPTS home page has asked me why there is not any trace for all the activities in this site? the answer is that on the public site we'll post final reports, but work in progress will be done internally, thus the "action" is reflected in the members Wiki.
EPTS is also starting to be recognized in the larger universe as a representative of the EP community. OMG mentioned EPTS explicitly in its RFP that deal with event meta-modeling stuff. Several articles in professional magazines, both industry oriented journals and general computing magazines are running articles about EP or CEP and cover EPTS as part of this article, I'll write more about any such article when I'll have it under my hands.
This is a slide rule, an ancient means to do arithmetic calculations easily if one has some experience in working with it. When I took the matriculation exam in mathematics many years ago, the Israeli ministry of education did not allow the use of calculator, since calculator at that time was relatively expensive, and it was considered of giving unfair advantage to those who can afford it, however they allowed the use in slide rules, so it served me well at that time. Today slide rules together with logarithmic tables and typewriter found their way to museums, but other type of rules are still with us.
Some Blogs have recently made references to the recent Forrester report with the catchy name:
Must You Choose Between Business Rules And Complex Event Processing Platforms?
The Forrester reports discusses some confusion that exists between the two terms. It is true that there is some ambiguity of the word "rules" - on one hand rule-based is a kind of programming style that can be used to express event processing patterns, and between BRMS - a collection of products with a certain functionality. Forrester also claims that to add to the confusion there are people who use (or abuse) BRMS products to do CEP applications and CEP products to do business rules applications. You can read the rest of the original report for more details. In my previous posting about state processing and event processing I have talked about the difference between the two. In fact, BRMS products are processing the current snapshot (state), while event processing is about processing of the history of transitions, different kind of techniques and optimizations are used for each.
I have also blogged recently about decision agents, talking about the fact that event processing agents (at least some of them) can be a subset of a larger whole which can be called - decision agents. And indeed, while the two type of technologies are distinct, there is also a sense to look at them with a unified view. Here I share the vision of James Taylor who talks a lot about Enterprise Decision management which consists of business rules, event processing and analytics. We'll hear much more about this concept - more later.
I'll start with a reminder that the deadline for submitting papers for DEBS 2009. The DEBS conference is the major conference for event processing related research. I would like to highlight especially the industrial track and solicit submission both from vendors and customers. Vendors may not submit a marketing oriented paper, however can send papers about architectures, technical questions, language issues, semantic issues, and engineering related topics. We especially solicit experience reports of event processing systems of any kind that are working in production, with experience insights that may be of general interest. Note that a short abstract is due next Monday, February 23rd, the deadline for submission is currently March 2nd, however extension of a few days will probably be possible.
Many years ago I have taken a course that has qualified me to be "IT project manager", and when talking about software life-cycles the instructor showed us the picture above, since then I have a Pavlovian Conditioning that the term life-cycle is associated with this picture.As a follow up to my recent posting about static and dynamic event flows, one of the related questions is whether event processing systems have the same life-cycle as other software artifacts, i.e. has to go through development and testing processes before going into production, or since it is very easy to modify such applications, this is not really needed, and changes can be done fast.
As usual in event processing there is no single answer. In some cases the event processing software is part of mission critical applications and thus any change should go through some validation and approval processes, which, alas, takes time and hurts the agility; in the other extreme there are systems in which an individual consumer defines personal alerts from events related to social networks, this application is not part of any enterprise application, and the consumer can add/modify/delete alerts every 10 minutes without upsetting anything except for himself, so there is no meaning to life-cycles and red-tapes.
Some applications may be hybrid. Part of it is a controlled logic, and part of it can be free for use to users.
