Weekends is catch-up times for any workaholic. In my case, I dedicate the spare time in weekend to advance in the EPIA book, which, as expected, is somewhat behind the original schedule. We are now cleaning up the main chapters of the books that deal with : event processing network, contexts and patterns. I'll dedicate some of the coming postings to book related stuff.
In chapter 6 we get a deep dive into the notion of EPN, and its various components: event processing agents, channels and global states. Consumers and producer are discussed earlier in the book. Event Processing Agent (EPA) is a major abstraction we are using. In recent posting I've discussed that the conceptual EPN may be mapped in various ways to implementations, EPA is part of the EPN, as such it is conceptual creature, and can be mapped to implementation in various ways.
The illustration on the top of this page illustrates the (logical) anatomy of an EPA. It illustrates a kind of SCA like component (SCA has now a proposed events extension, I'll discuss it another time), with input terminals and output terminals. In this cases, the EPA gets events from input terminals, each of them has a distinct type of event. Let's take a simple three event conjunction example. A virtual car dealer matches used car sellers and buyers, once there is an agreement on the transaction, it supervises that it is carried out. The used car buying transaction is a conjunction of three events (the order of them may not be important) :
Event1: The seller physical hands over the car to the buyerEvent2: The buyer transfers to the seller the agreed upon amount for the carEvent3: The department of transportation approved and records the ownership transfer -- in Israel it is a service given in post offices, the buyer and seller identify and sign, and the post office clerk has access to the transportation ministry vehicle registration system and can verify who is the registered owner, and if the car has any incumbrance
The three logical steps of EPA are: filtering, matching (if the EPA is looking for a pattern), and deriving. The filtering part may be done outside the EPA, e.g. if the EPA instance deals with a certain transaction, then the channel should route only events related to this transaction to the EPA, I'll discuss more of the notion of channel in the near future. However, there are cases that the events should be filtered by the EPA, e.g. when it involves a condition on more than one event. After the filtering part some subset of the input events survive to be the players in the EPA. If the EPA is doing also pattern matching, it creates matching sets according to the requested pattern. In the case of the used car selling, the pattern is a conjunction of these three events that relate to the same transaction, the order is up to the buyer and seller. When the pattern is matched, it means, in this case, that three events have been detected, which satisfy together the pattern. These three events create a matching set, for each EPA instance there may be zero , one or multiple matching sets during the lifespan of this EPA instance. After the matching set is created, the third phase - the derivation phase is aimed to determine what should be reported out of the results of this EPA. It can just report the composite event that contains the three events in the matching set, and it can contain any other raw or calculated value that is a function of the attributes of these three events. Furthermore, for different purposes, different derived events can be required - consumed by other EPAs or by consumers. This is the way we describe EPA, note that this description is general enough to cover EPAs from different types. The input terminals can take either individual events or sets. I'll discuss more about EPA types, channels, contexts and patterns later
The Gartner hype cycle has a notion called "peak of inflated expectations", which states that different technologies go up in the hype ladder until getting to the peak, where people think that can solve much of the universe's problems, and then, somehow people realize that this is not the case and go through frustration and disillusionment, until realizing the true value (if any !), and getting back on track, now with the right set of expectations. Two recent Blog postings show some indication that we are getting closer when talking about event processing:
Actually, both are right. Event processing may have a role in decision management, there are some applications of decision management that are pure event processing, some in which event processing has some role, but not doing the entire trick, and some that are really batch oriented data management. Likewise continuous analytics can be done in response to event (raw or derived) or just periodically. Event processing may or may not have a role in deciding when to do this analytics (e.g. optimize the traffic light setting), the optimization itself is typically not event processing per se.
I think that it is very good to observe that event processing can play a role in many areas, likewise, it is also good to be clear about its possible role, and which are the cases in which it has value, and which are the cases it hasn't, I guess that we'll have to wait for the enlightenment phase, in the hype cycle terminology until there will be more universal clarity about the role and value of event processing. More - Later.
Working now in my spare time on completing the second third of the EPIA book, so I'll have several postings related to the next three chapters of the books that are now in the "cleaning phase". The topic I'll discuss today deals with the concept of EPN (Event Processing Network) which is a major concept in our book. The approach we have taken in the book is to explain event processing through a meta-language that provide the various event processing concepts, and the event flow through a model based on event processing network. We are now also competing an editor that will enable the reader to play with the meta-language. However, this meta-language is not an executable language (at least not in this phase), and thus we also show the readers how the same application described in the meta-language is implemented using various executable languages of different language styles. The EPN described by the meta-language is a "conceptual EPN", it consist of logical EPAs, while the run-time EPN consist of run-time artifacts that implement the run-time instances of the EPAs.
The conceptual EPN can be mapped into physical implementation in various ways, as shown in this picture:
The traditional centralized implementation is that the entire EPN is being executed using a single run-time artifact, and the EPN describe the internal flow within this artifact.
When talking about distributed EPN, the EPN can be distributed according to several criteria:
- Segment partition: All the EPAs that relate to "platinum customers" are being executed by one run-time artifact, all the EPAs that relate to "gold customers" are being executed by another run-time artifact etc...
- Function partition: All the EPAs that perform a certain function are being executed by a unique run-time artifact
- Location partition: All the EPAs that relate to events created in a certain location.
These, of course, are just examples. The most distributed example, is, of course, a direct mapping of each EPA instance to run-time artifact.
The conceptual EPN is important for design and validation of the event processing application, while the run-time EPN is useful for control and management of the run-time.
More about EPNs and their components - in later postings.
These three pictures are taken from the posting : On unicorn, professor and infant, which is, by far, the most read posting on this Blog. Today is August 28, exactly two years since I started my first posting - this is posting number 314 (PI * 100)
- Event processing technology is still a "child" (maybe no more infant), in the the life-cycle of technologies. It is moving to the youth stage, but have not achieved maturity. It still has challenges in standardization, interoperability with the rest of the universe, engineering and optimization issues, and consumability.
- Event processing is not a unicorn, it is not mythological but real, and it has more than one horn, it is not strongly associated with a single application, as some people tended to believe in the past. Each technology has its early adopters, but we see now that the utilization of event processing is pervasive for each industry, no less than technologies like databases, web services and workflows.
- The blind people touching an elephant and describing it in a different way is still an excellent metaphor. People have come from multiple perspectives and see the world in a different ways. Our challenge is to get a good understanding of the entire elephant, and overcome this syndrome. This is the mission of the Dagstuhl seminar in 2010.
While the unicorn and infant posting was by far the most popular - other popular postings were some postings from 2007 entitled: On event stream processing in which I have written first time that the ESP term has virtually died, and Agnon, the dog, playing and downplaying,
which responded to the claim that I am downplaying the role of rules. Among the 2009 postings the most popular was the one about the event processing language analysis workgroup,
Some more current statistics.
- The number of distinct readers is around 35,000 over these two years, however, many of them are just one-time browsers, the loyal readers, which consist of those who entered the site at least 101 times, their number is around 2,000, if we'll add more than 1,000 more who entered the site between 51-100 times, and another 1,000 who visited between 26-50 times, I would say that there are around 2,000 loyal readers, and around 2,000 more occasional readers - not big numbers in the Internet scale, but fair enough for a Blog focused on a narrow area with a developing community.
- The countries with most readers are: USA (still more than 50% of the readers), UK, Germany, Canada, France, India, Australia, Holland, Sweden and Spain. Israel and Japan have many entries, but less readers, which say concentration of loyal readers.
- The biggest referring source, no surprise is Google, other big reference sources are: the TIBCO blog, David Luckham's CEP site, Marco's rulecore Blog, and ebizQ.net.
The Blog has been cited by the CEP Knol and has become a major reason for the fact that I was approached by a publisher to write a book.
The other Blogs of the community are either company's Blog - like the TIBCO, Aleri, Streambase and Apama Blogs which express the company's view; or those of independent people or professional bloggers. While I work for IBM, I have never made it as an IBM marketing Blog, but a personal one. I have never been asked by anybody in IBM to write or not to write about anything, and although as IBM employee I do have some restrictions, I have taken upon myself more restrictions as the EPTS chair, among them the restriction of not to write evaluations about products; I am leaving this task to others; the only time that I was asked to revise a posting was fairly recently, when I cited the Gartner Enterprise Architecture Hype Cycle report, where I was asked by Gartner to make clear distinction between what I am citing from Gartner, and what are my own opinions, so I have revised it and restructured the posting in a way the distinction between them will be clear.
Sometimes new topics emerge, and sometime topics are repeating discussions, but never a dull moment in Blogland. Will return with a professional posting s00n.
Among all topics that I mentioned in my previous "catching up" posting, I chose to talk about downloadables. The Forrester report mentioned "free downloadable" as a criterion, and one of the authors defended this criterion in a short article. Paul Vincent has written about his impressions from the summer school he participated in that some of the academic people like to use open source, so they can play with the implementation code.
There are two issues here: whether downloadable is a criterion in product evaluation, and whether for academic reasons one should use open source or commercial product downloadable ?
As for the first question: the Forrester report stated that event processing is moving from being niche technology to being part of mainstream computing. An enterprise can chose in using a specific product, either because it is being sold through some solution / application, and in this case the line of business owning the application, may select this application and get the product behind it as a by-product, this is exactly niche technology kind of behavior. Getting it as part of mainstream computing, mealing that this product is being used in enterprise architecture, and being endorsed by the IT side of the enterprise. I have spent 8 years in my life in an IT shop of a big enterprise, and from this angle, the world seems somewhat different, decisions (if not constrained by force majors) are being determined after the IT developers have studied the products from all angles. Big enterprises don't typically have problems to get any software for free evaluaton; however, where free downloadable may help is for people who want to explore products by their own initiative, whether they are developers in organizations that have not decided (yet) that they want to do formal evaluation, or by students, who are the future generation of developers. Investment in students is considered a good investment and thus many vendors provide free software for academic purposes. As I have written in the past, the publisher of the EPIA book we are writing is very keen on providing downloadable software for the reader. Since we told him that there are multiple approaches and we don't want to restrict the reader to a single approach, we have approached several vendors, who agreed to: a). provide free downloadable; b). provide a solution of (at least a subset) of the "Fast Flower Delivery" example that accompanies this book. Currently we have several vendors that already agreed to work with us on this: Aleri (representing the Stream programming style), Rulecore (representing the ECA rules programming style), Apama (representing the script programming style). We also got agreement from Esper (which is already an open source) to provide the solution of the case study in ESPER. I hope that other vendors will follow. I am sure that the readers will be benefited by this exercise (we'll put all solutions in an appendix to the book, and point at the soft copy of the solution, and downloadable from the book's website).
Now, to the question of downloadable vs. open source in teaching. I am giving students two types of programming assignments (depends on the course) -- one is to write some application using a product, the other is to write a subset of an event processing engine.
For the first type of assignment, both product downloadable and open source can be used. In the database area, I guess that the majority of the basic database courses are using My SQL, however, in the event processing area we don't have a standard language yet, and typically products are more advanced than open source.
For the second part -- writing a subset of an event processing engine -- one can take an open source as a basis and extend it, which may be one approach (common in operating systems course), my personal preference is to let students build such system from scratch, the reason is that again, due to lack of standard, the use of open source will restrict the student to use the approach taken by that open source, alas, the approach in which I am teaching event processing is somewhat different. I also think that students have to think about how to implement concepts rather than to ruminant on other food. More on teaching considerations -- later as I am prepared to teach an event processing course twice in the two semesters for two different populations, in the Information Systems Engineering program at the Technion, where I'll ask the students to use products, and in the Computer Science program, where I'll ask the students to write a mini-product.
A comment: I am moderating comments after some spam attack in the past, recently I am getting everyday a comment to this Blog in a charachter set which may be Chinese or Japanese.
I wanted to respond: לא הבנתי מה רצית ממני, but as this Blog is being handled in English, I will not post comments in other languages, I don't even know whether they are spam or real comments.
I also don't post comments from people who don't identify themselves.
Last but not least, a reminder: The EPTS 5th event processing symposium is getting closer, if you wish to participate, and have not registered yet, please do it on the conference site, the reason we need this registration ASAP is that there are quite a few of requests to register from non EPTS members, and since we limit the number of participants, we want to make sure that we are within these limits. BTW - the program on the site is not up-to-date. The updated program will hopefully be published next week.
This picture is another picture from Finland, this time from the national park Helvetinjärvi in the Tampere area (more or less), but time to move on and return to work, which I am doing rather slowly (apologies to all Emails I have not answered yet, will catch up by the weekend).
A lot of catching up in work, but now I took a small break, drink bitter coffee, and had a quick scan on some Blogs, some deserve detailed responses, but in a catching up mode I'll only provide highlights:
- Marco has written about patterns, there are two comments here - one a terminology comment, he seems to be confused by the distinction between complex event, composite event, situation, event correlation (a term I don't really use) and pattern. These terms came from different sources, and they are not exactly synonyms, I have tried in the past to look at their definitions and outline the relationships among them, the interesting question is do we want to stay with all legacy terms or settle in a fewer terms. This is a question for the community, and for the glossary work group.
- Marco also started to look at list of patterns. I'll refer to it in one of the next postings, I am now working on the chapter in the EPIA book about patterns (chapter 9). We also are looking at it within the language analysis work group.
- Paul Vincent had a few interesting postings, some of them summarize iimpressions from the summer school in Norway in which there seems to have been discussion among different approaches.
- Paul Vincent also had another posting. getting back to the CEP vs. ESP issue. I've already made a short comment to this postings, that I think that this label discussion is somewhat anachronistic, since the term ESP is virtually gone away. Besides the labels, there are indeed different approaches to approach event processing applications, and I think that we'll see more of hybrid approaches. This also deserves a longer discussion.
- Paul Vincent also wondered about downloadable as criterion, following the Forrester report, that has put it as one of its 114 criteria to compare EP products. John Rymer from Forrester has responded in ComputerWorld UK. I would like to have further discussion on the Forrester criteria in general (within the catching up I also read the full spreadsheet of comparison, and have some interesting insights), also I need to follow up on the downloadable issue. Not sure if it is a criterion about product quality, but IMHO, free downloadables are important on getting event processing from niche technology to the mainstream, which was the major theme of the Forrester report, this issue of niche vs. mainstream, also deserves a discussion by its own right.
- Hand Gilde has written about the Gartner Hype cycle report. He was not happy from the way that Gartner articulated the value of event processing, and sent the challenge to EPTS to make better articulation of the value of event processing to the general community. I agree that EPTS needs to catch the ball here, and will propose to imitate such activity. As always, I am throwing back the ball to anybody that throws it towards me, I think that Hans is an excellent candidate to be one of the key persons in such an activity, and I call him to quit his bystander position and jump into the cold water.
- Mani Chandy who uses his recent Blogs to post articles, has written about "Smarter communities". This also deserves a longer discussion, I agree that we are heading this way in variety of areas that IBM labels as the "smarter world".
- Some other references to the Forrester report which seems to catch the attention and kept (almost) everybody relatively happy. I still need to discuss some of the observations from this report. While looking at the detailed spreadsheet, I have did a quick summary of the number of customers reported by the different products. The number looks impressive, I wonder how many of these are currently running systems in production. But I'll let the use case work group chase them to obtain use cases, and ask everybody to help on this -- another topic to discuss.
Well - enough for now, back to the long list of to-dos...
This is a picture of the cottage in Finland, where we spent the last week. The cottage is totally isolated, in a small Forrest, near a lake. In the previous week we spent in another cottage, also in Finland. This morning, I have returned home (and tomorrow return to work), and here are some fresh insights from Finland:
- There are almost 200,000 lakes in Finland; comparing to Israel, in which we have two (2) lakes, that we call each of them "sea", the dead sea is one of them.
- The local drivers are surprisingly obeying speed limitations much more than any other country I have seen so far.
- Sauna is an important part of the picture, we visited a place that has been prison until 16 years ago, and found out that even prisoners had right to use the sauna once a week; I tried using the sauna once, but I never liked sweating.
- The Finnish language is not similar to other Nordic languages, it sounds to me like Hungarian spoken in Italian accent, but this may be my own imagination.
- The summer in Finland ends roughly in August 10, we discovered that some of the attractions are already closed, since the summer (i.e. the summer vacation for schools) has gone. We succeed to visit the stone age village in Saarijarvi, thanks to the lady from the Saarijarvi tourist information that coordinated to open this site especially for us.
- I have used GPS in order to navigate, it brought us to the most deserted places, however, we found out that one needs to use GPS in Finland (and elsewhere) with a grain of salt, and when it directs you to a route that does not seem to you the intuitive way, prefer your own intuition.
- The most common "fast food" in Finland is Pizza.
- Overall -- great place for peace and quiet and nature walks, good for disconnect from civilization.
And one comment on the flights back -- we came back from Finland to Israel in a combination of a Finnair and ELAL (the Finnish and Israeli airlines). Both Finland and Israel see themselves as high tech super-powers, however, the software in their airline is not impressing. The seat assignment software of Finnair, has problem to understand that people who travel together want to sit next to each other, and around me several people switched places to achieve sitting together. It seem that ELAL has not heard about modern development in distributed computing, fault tolerance and some other trends. We checked in relatively early and it was OK, however, at some point, the centralized server in Israel crashed, and they could not bring it up, and it seems that the remote station was completely helpless, they ended up making manual seat assignment, and the flight was delayed for almost an hour. Back to professional Blogging - later this week.
