CAC Japan is the only Japanese member of EPTS, some of their team have participated in the EPTS 4th Event Processing Symposium. One of the missions they have talked upon themselves the task to bring the event processing ideas and concepts of event processing closer to the Japanese market, they have sent me the URL of their Japanese portal
and in specific they have translated the EPTS glossary to Japanese
I salute the initiative, we indeed need to reach out for more parts of the universe. I don't understand the Japanese part of the site, but trust the CAC team. Hope to have an opportunity to visit Japan some day.
The issue of translation from language to language reminded me about the still chaotic situation in the area of event representation on which I have blogged about a year ago.
Much of the existing event processing systems see the event representation as an internal issue and make their own decision about which structure should be supported: flat positional like 1-NF in relational database, tagged semi-structured (XML), or Java objects. Furthermore the "header" of an event varies. This reminds me a discussion a few years ago with some of my IBM colleagues who worked on standard for management events. They included "managed entity" and "severity" as mandatory attributes, since in their domain -- event is always a reported problem. We explained them that there are other types of events, e.g. money withdrawal. For this type of event there is no severity and no "managed entity". This has been a funny discussion since they did not really think that what we are talking about is event, however, in their domain, what they have done made perfectly sense. This leads to a thought that we may have several layers of attributes:
- Universal attributes that should be mandatory in any event --- there should be very few of them --- Marco from RuleCore has suggested in his recent Blog:
- Id - every event is unique and have an unique id.
- Type - Every event is of a specific type.
- Detection time - every event is detected at a specific time.
- Entity id - The entity that changed its state.
I agree to the first three, but not every event changes the state of a single entity, and in
some cases (e.g. derived events) it is not really important which entity changes it state, so I
would move the "Entity id" to the next category.
- Domain common attributes (which also may have several level of domain, sub-domain, sub-sub-domain etc...) -- this may include: event-location, entity that changed its state, time-zone, severity and much more.
- Event type specific attributes -- individual for each event type. Some of them may be reference to entities, some descriptive attributes.
There is even more interesting discussion about derived events - but I'll defer it to one of the next postings.
The window in the picture cannot be openned or closed without some fixing, which is typical to some of the event processing language that support in fixed window. I have been quite amused in the last couple of days to read the discussion in the CEP-Interest Group on Yahoo!, started by
a query posted by Pedro Bizarro.
If you do not subscribe on this group I am copying Pedro's query:
Hi there,A company contacted me to solve a query puzzle that they have. Theyare reading incoming tuples and want to select all tuples that arebetween two tuples. For example, assume that I want all tuples betweenthe first tuple with PARAM1=2 and the first tuple after this one withPARAM2=0. This would select all tuples with time between 3 and 8 inthe example below.TIME | PARAM1 | PARAM21 1 02 1 13 2 2 <== ON4 2 35 3 56 3 47 2 38 2 0 <== OFF9 3 1I have tried a number of event processing engines but cannot writesuch a query. (However, I can write the query on a table on Oracle 11gusing Oracle's Analytic functions with the help of a simple PL/SQLfunction.)Do you have a solution for the OnOff Window puzzle?Thanks!-pedro
You can browse and see the variety of answers given by different vendors - my intention is not to analyze the results, but look at the semantics of what is asked by Pedro. Pedro is talking about window, typically window is used in order to select a collection of events and perform some operation on them, in Pedro's case the operation is trivial - just select all of them, but we can slightly change the requirement and change it to "find if a pattern P is satisfied in this collection of events" (example of a pattern -- there are at least two events in which PARAM1 > PARAM2 for the same event) and if this pattern is detected perform some action (e.g. send me SMS).
Here we have three types of conditions:
1. Condition for opening the window
2. Condition for closing the window
3. Condition for detecting a certain pattern (in this case the result of the detection is a "situation" since it triggers action outside the event processing network.
Condition 3 is a normal type of "pattern detection" processing, however, conditions 1 and 2 have different roles -- they are used in order to decide which events will be used as input for the pattern detection operation. One of the answers was given by my colleague from IBM Haifa, Guy Sharon, who talked about the AMiT implementation of "lifespan" as a temporal context. Again, without going to the particular language, I would like to point out that the principle of "separation of concerns" --- have distinct separation between conditions to select the events that participate in an operation and the operation itself provides better clarity to the end result. Since a window is indeed a subset of the term context on which I have blogged several time before and pointed out about the importance of looking at context as a first class citizen in event processing. Context can have several dimensions -- the temporal dimension of context subsumes the notion of window, and there are other dimension (e.g. spatial - that selects events based on their location).
Bottom line: my belief is that the consumability of event processing will increase if we'll succeed to make them available to semi-technical people and have clear semantic roles to the building blocks instead of trust the developers that they will find a way to hack a bit in order to satisfy requirements -- more on this topic -- later.
Towards the year 2009, EPTS will increase its activities. Currently six working groups has been approved by a series of meetings of the EPTS steering committee extended with all the people who proposed working groups. We are going to issue soon a call for -- comments, vote and participation for the EPTS community.
First - something about the process of EPTS work. The main work will be done in working groups, the steering committee serves as a facilitator, but each working group has two co-leaders (as the proposals go now), and help the proposers devise the charter, make sure it makes sense, and meet the legal requirements (one of the properties of making EPTS as a formal organization is that there are some legal agreements between the members that need to be kept). The sec0nd phase which we are now entering is -- putting the working groups proposals in the EPTS site, in a members only section of the site, for comments, vote, and call for participation - each organizational and individual member can participate in any working group they are interested in. However, participation also means commitment for active participation. We shall hold a members' call to present all the proposals, and then the members will vote. Each negative vote will have to be augmented, and the proposal leader will answer - both objections and answers will be made public. After the members' vote, there will be final discussion in the steering committee, especially for a working groups that had objections, and final decision will be made.
The idea is to finish all this process in early January and launch the working groups for 2009.
EPTS members will get further instructions; the participation in the working group is restricted to EPTS members only, for legal reasons; however, everybody can become EPTS member (organizational member or individual member - see instructions in the EPTS website.
The six working groups that will be presented are:
(1). Glossary: We have issued version 1.1 of the glossary, but the work has not ended; this is a living document and a moving target, as the event processing area is in a relatively young age as a discipline. An agreed upon glossary is important to have common language, and has been successfully done in other disciplines.
(2). Use Cases: This working group continues from 2008 and had devised a template for the analysis of use cases, the idea is to survey a significant amount of use cases in order to classify event processing applications.
(3). Meta-modelling: OMG has issue RFPs for meta-modeling standards that have relations to event processing, in specific: Event Metamodel and Agent Metamodeland Profile.
EPTS still needs to determine about its status of engagement with OMG, according to it this can be either official response to the RFP, or input to OMG. In any event, EPTS has been recognized by OMG (and referenced in the RFP itself), and was asked to provide input. The working group will attempt to provide unified response of the EPTS community. Note that this is the pattern we are pursuing in general - EPTS will not become a standard development organization, but will assist existing organizations to develop EP related standards.
(4). Reference Architecture: In the early days of the pre-EPTS, there has been some work to collect and compare reference architectures of various vendors. We are now returning to deal with refernce architectures, this time in the form of an EPTS Working Group. This working group will propose one or more reference architectures for various cases (consistent with the evolving classification in the use cases workgroup and the evolving glossary).
(5). Interoperability Analysis: This working group will engage in study of requirements and mechanisms for interoperability - both between event processing products of various vendors, and between event processing products and various producer and consumers of event processing.
After the study, the working group will recommend to the EPTS community about next phases
(e.g. creation of additional standards, revision of current standards etc...).
(6). Languages Analysis: This working group will engage in study of existing event processing languages (both from products and from the literature) to devise (in a semantic level) a set of functions that is being used. After the study, the working group will recommend to the EPTS community about next phases (e.g. creation of a single language standard or creation of N variations for various languages or creation of a meta-lanaguage standard...).
I am personally will co-chair the languages analysis one (an area that I spent a lot of time on recently), and will follow, all others.
More working groups may be launched, however, I surveyed only those approved so far to continue to the next phase.
I believe that at the end of 2009 with the results of these working groups report, we'll advance the understanding of the event processing discipline, and will have a clear road-map for related standards...
Enough for now -- more later.
In Israel, a new law has gone into effect in December 1st, to fight Spam - it forbids sending commercial material over Email, SMS and recorded phone calls, unless the person signed up explicitly (e.g. through a website); recorded phone calls are especially bad, picking up the phone and talking to recorded message is something I am allergic to... Unfortunately, this law is in effect only in Israel, and I keep getting Spam in Turkish and Russian -- I have already blocked more that dozen spamming sources, but others keep pooping.
Anyway -- this week, my Master student, Elad Margalit, had his final exam on the thesis, his thesis dealt in event-driven control of traffic lights, and he has shown (by simulation) that when the traffic lights frequencies of green and red in individual junctions are optimized based on the actual traffic it can shorten substantially the waiting times (there may be various goal functions here). This is done by continuous monitoring of the traffic (entrance and exit from a road segment by camera). IBM Research has done a lot of work in the area of "continuous optimization" that deals with solving optimization problem continuously, there are two variations: Solve the optimization problem all the time (in our case - for each cycle of the traffic light), or monitor all the time and solve the optimization problem only when there is a significant change from the base assumptions (in our case -- when the distribution of traffic in the various direction is significantly different, otherwise leave the traffic lights policies as is), this is based on the assumption that the cost of monitoring is much lower than the cost of actual changing the system behavior; if no such difference then we can use the continuous actions.
It is interesting to note that Coral8 added in its logo the trademarked expression "Continuous Intelligence". Interesting phrase, not sure if there is indeed intelligence, but I guess that intelligence is in the eyes of beholder.
Besides the fact that there is still a sort of uncertainty among the general IT community about what is event processing (or CEP, which is the most common TLA associated with it), some discussion on the Blogsphere lead to further confusion --- e.g. claiming that CEP equals BRMS. Carole-Ann Matignon,VP, Product Management at Fair Isaac, has nice posting in her Blog, talking about this confusion, and providing what she calls simplistic view of the definitions of BPM, BRMS, and CEP. I'll go with the simplistic direction of thinking and claim that this is a mix of terms from three different domains -- application, technique and function.
Function answers the question --- what is being done ?
Technique answers the question -- how something being done ?
Application answers the question --- what is the problem being solved ?
Examples
- Business Activity Monitoring (BAM) is an application type, it solves the problem of controlling the business activities in order to optimize the business, deal with exceptions etc...
- Business Rules are type of technique --- which can be used to infer facts from other facts or rules (inference rules) , or to determine action when event occurs and condition is satisfied (ECA rules) and more (there are at least half a dozen types of rules, which are techniques to do something).
- Event Processing is really a set of functions which does what the name indicates -- process events --- processing can be filtering, transforming, enriching, routing, detect patterns, deriving and some more.
Of course there are inter-relationships among them --- some types of business rules are a non exclusive way to execute some of the functions of event processing -- e.g. ECA rules can be used for routing, inference rules can be used for event derivation and more. However, there are other techniques that can be used for each of this functions. Likewise, a BAM application may use event processing functions as part of its implementation, but may also use other techniques (e.g. being data-driven rather than event-driven and do all its processing in retrospect, looking at committed data periodically and not on events as data-in-motion). Business Rules can be used for various utilization in BAM and in BPM applications, with or without the use of event processing.
The functions of event processing is typically used for several motivations:
- Observation into business processes, business activity monitoring.
- Dynamic reaction and modification to transactions and business processes
- Diagnosis of problems and finding root-cause.
- Prediction of future problematic events that need to be eliminated or mitigated
- Dissemination of data in motion to arrive to the right person at the right time in the right granularity.
These motivations are not applications --- there are multiple applications that perform diagnosis -- system management, car repairs, medicine diagnosis, oil drilling management; there are also multiple applications that perform more than one of these -- e.g. observation and reaction or prediction and reaction. I have blogged before about the metaphor of blind people touching an elephant -- there are some people who are looking at a single line of applications, or a single implementation techniques and saying CEP is X, while another person who is thinking on another concept is saying the CEP is Y, thus a confusion is being created - the following elephant's picture illustrates it (taken from the original postings).

Bottom line: the confusion is a result of equating terms from various classes, and resolving this confusion is indeed very simple.
Yesterday I had the honor to eat dinner with the Israeli president Shimon Peres (well - not 1-1 dinner, not even in the same table, but on the table next to his); this was in an event that included many of the seniors of the Israeli High-Tech, about a project in which I am involved of helping Arabs to break into the Israeli high-Tech on which I have written before.
In Israel, the president is a symbolic head of state, while the "CEO" is the prime-minister; Shimon Peres, seen in this picture from last week where he became a knight by the UK queen, is a very impressive person, 85 years old who has a lot of energy and considered as the ultimate elder statesman. I wish that I'll have the same energy and vitality in the age of 58...
Back to event processing -- reflecting back on the "event hierarchy" discussion. Some people assume that the event processing network is a DAG (Directed Acyclic Graph) which is a structure whose handling is well-known, unfortunately, the world may be sometimes cyclic and the event processing network that represents this world is also cyclic.
Let's look at the following example -- I went to the interior ministry for re
This is, of course, an event processing system, relatively simple one. Trying to sketch the event processing network we'll have events - like:
customer arrived, clerk arrived, clerk displayed number, customer sits in front of clerk, customer gets up and leaves the clerk, clerk displays number.
Looking at this events we can realise that they create the following cycle
clerk arrived ; clerk displayed N; customer sits in front of clerk; customer leaves clerk; clerk displayed L etc... This is a cycle Clerk displayed L is an indirect descendant of Clerk displayed N... and likewise we need to continue circling...
Since event processing network deals with event types then we can say that "clerk displayed number" repeats itself, this may also be for the same clerk and same number -- same clerk, since clerk's id is fixed, same number -- since numbers can be recycled (e.g. after getting to 20, return to 1)... The support of cycles, of course, spoils the notion of hierarchy.
How do we know that cycle will not be infinite? ---- if the context is bounded then there is a natural stopper for the cycle (e.g. end of the reception hours), if the context is not bounded, then an infinite loop is possible, and there should be other means to detect and handle it, if occurs (e.g. allow limited number of cycles).
Should we forbid cycles? -- we are back to the dilemma about Russell's type theory.
we can forbid it and our language will be restricted and will not be able to express scenarios such as the one expressed in the example above, if we allow it --- we spoil the strict hierarchy, but as mentioned in the previous postings, strict hierarchy of events may not be realistic.
In a comment to my previous posting Harvey has reminded me that Israel won second place ("gold medal") in the Chess Olimpics, actually it has been number one through part of the games and then moved to number two which is also very respectable. I have not played chess since high school, but I have registered to play in an internal IBM championship in Backgammon;
A game I have not played a lot recently, so need to practice...
I am following with interest Marco's Rulecore Blogging about geo-spatial operators, I think this is in the right direction, and there is a lot of potential in spatial and even spatio-temporal patterns and I also view space as one of the main dimensions of context. However, today I would like to write about one of the other topics that Marco has been writing about -- event hirearchies, in this posting Marco suggests hirearchy level of events and rules, such as rules in level 1 consume only events of level 1 and produce event of level 2, and so forth. This reminds me a great scholar named Bertrand Russell, whose picture you can see below:
Russell has introduced a well-known paradox which in its popular form is called the "Barber Paradox" (Wikipedia claims that
which states that in a village there is a barber who shaves everybody who does not shave himself. The question is who shaves the barber? if the barber shaves himself that means that the barber is not shaved by the barber, however if the barber shaves the barber that means that the barber does not shave himself, meaning that the barber shaves himself if the barber does not shave himself and vice versa. Russell also provided solution for the paradox called "type theory" saying that we have hierarchy of terms and functions/predicates based on "types". Thus if a "person" is of type 1, Barber is a predicate of a person, which makes it term of type 2, being a term of type 2, it cannot participate in predicates on type 1, and thus the entire paradox does not happen. Type theory indeed resolves the paradox (if you wonder the "types" in programing languages are indeed derived from Russell's terms), however, the price is that the expressive power of the language is compromised, things which seem to us perfectly valid in natural language are not valid under the types theory.
Back to event processing -- let's look at the following example:
- Our location-based event processing system manages fleet of cabs and travel reservation.
- An order arrives for a travel in 30 minutes
- this is an event of type 1, it is processed and matched against taxis locations and directions to assign the right cab and makes "cab assingment" which is an event of type 2 (since it was created by an event processing agent - "rule" in Marco's terminology that processed events of type 1).
- After a short while the perspective traveller changed his mind and called to cancel the order. This is again, event of type 1.
- However, now we have to find out if the travel has already been allocated and if yes deallocate the travel and free the allocated taxi. This operation has to take two events as input -- the travel cancellation (type 1) and the travel allocation (type 2)... Oops... our system works purely on strict hierarchy of event types --- can't do that!.
This demonstrates that using strict hieratchy we'll have, like Russell's type theory, to give up expressive power. So it seems that while hierarchy is a good idea, it may also serve as a restriction... One modification is indeed to order the event types and the agents, however enable agent of type 2 to accept as input events at type 1 and 2.
With this correction there are certain benefits for using the hierarchy levels, relate to optimization possiblities that Marco have not mentioned. We'll discuss them in one of the next postings.