On event processing as a technology and as a business

Philip Howard, an analyst who covers event processing for several years now, has posted a blog entitled: what's happening with event processing, observing that event processing is getting integrated with other areas such as: BPM, data integration and more. This is not a new phenomenon; in the EPIA book, we mention that among the event processing trends is moving from standalone to integrated even embedded, and this trend is evident with the evolution of event processing as a start-up universe, to having bigger software vendors as dominant forces. However - will event processing as technology going to disappear? I don't think so. There is common functionality among event processing utilization in various industries, applications, and hosting technologies, in all of them there are functions of filtering, event transformation, aggregation, pattern detection, and routing. It is not cost-effective to re-invent the wheel for each individual use (although there are variations). This is a similar situation to databases; database can be used for various reasons, and also be embedded with various other technologies and products (e.g. application servers, BPM, system management products, messaging systems - all use databases), while there are also variations, it is not the case that each of these areas develop database technology in an ad-hoc fashion. Thus, I see event processing continuing to evolve as a technology, and having both research and development activities that build generic event processing tools. From the business point of view, there will always be some niche for event processing stand-alone applications, but as Philip writes, most of the market will indeed be in the integrated area, this fact already reflects on the event processing technology in terms of need for standards, interoperability features, and ability to have embeddable collection of building blocks and components. More about this - later.

Some lessons learned from students' projects

Spending some time in looking at the projects of the event processing course I have taught in the spring semester. I am going to teach again this course in the winter semester, starting in late October, the first course that will be based on the final version of the EPIA book.

First - it seems that understanding the model based on the seven building blocks - was quite easy for the students that none of them had any previous knowledge of event processing. It seems that the students could study the model in a relatively short time and represent an application using this model. Of course, getting to the next level of detail in contexts and patterns require some learning, but based on some internal courses we made, I believe that with three days course that includes hands-on lab, one can study the model.

However, the students task was not only to model an application but also to implement it, and they were given the freedom to chose how. Some of the students have used existing products and some of them implemented the application in Java. I guess that here is a micro-cosmos of life, and the world is partitioned among people who like to learn software products and use them, and people who prefer to do their own development.

I have asked the group of students who preferred to write their own why they chose to do it, and their answer was that they are very experienced Java programmers, and while it was quite a substantial project to write it in Java, they felt that the project is "in control" and they are not dependent on any third party (i.e. ability to learn and use a product, possible bug handling etc...) but stay on familiar ground; I guess that this type of consideration also reflects what happens in real life. There is indeed an overhead to study a product for a single project, but some teams thoughts it is cost-effective and succeeded to implement it using a product.

It seems that both type of teams thought that modeling the application first using the model has made their life much easier when they came to implementation since it imposed order. We'll try to have some more modeling efforts of event processing applications using this model to gain more experience (and evolve the model). There are already some ideas and activities around extending the model, but I'll leave it for other time.

First review of the EPIA book on Amazon

Since the book Event Processing in Action became available, I am keep getting Emails from various people about it. The first one who has put a review on Amazon was Jim Odell, a notable person in agent technology, who recently wrote a series of articles about agents and events, hosted in Paul Vincent's TIBCO Blog. The review is entitled: "A true magnum opus". Since Manning publications is doing three reviews during the development process of the book, we got several dozens of reviews throughout the development of the book, many of them included some criticism in various aspects, and responding it required much work, in one case, a major rewrite. Some of the feedback I am getting about the book includes also some ideas for additions -- well, it is still early to think about second edition.

On P not equal NP

Last week a researcher from HP Research Lab, by the name of Vinay Deolalikar, notified the universe that he succeeded to prove that P is not equal NP, which has been an open problem for many years, many people has set to work on his more than 100 pages proof, and some claimed on finding fundamental flows, as a result Deolalikar notified that he is fixing some parts and will re-post an updated version soon. Many people are skeptic, but we'll have to wait and see.

Some observations about it:

1. The current Web 2.0 media -- Blogs, forums, social networks, is the frontier in which this entire discussion is being made, in a way there is a public peer review that is typically in scientific publications is being handled behind the scenes and don't make public. This is an interesting phenomenon. We see public reviews of books on the Amazon site, but public peer review on unpublished scientific work is rare to go public; I guess that this is currently reserved to VIP papers.
2. The scientific who did this work is from an industrial research lab and not from the academia, somewhat surprising since the image is that industrial research is interested in system oriented research and not in theory oriented research -- well, the world is not black and white. While research in industry is not in its glory period these days, it still attracts high quality scientists who prefer the frame of the private sector on the academic one.
3. People generally assume that P is not equal NP, and thus for many of the NP problems which have a need to be resolved in reality, there are devised methods and heuristics to obtain good enough solutions for most realistic scenarios. A proof that this assumption is true will have a huge theoretical value, but not a huge practical value, since it will not provide any insight of how to improve the solutions. Of course, if somebody will prove that P = NP it will have a pragmatic value, if it will show how to solve all the NP problems in polynomial time.
4. While the fate of this proof has not determined yet, I wish luck to Deolalikar, I like daring people.

On specialized graduate degree programs

Recently I came across several specialized programs, one of them is the master degree in De Paul university entitled "Master of Science in Predictive Analytics" that is sponsored by IBM; the other is a planned transatlantic master degree in collaboration between three European universities and three Canadians universities to pursue a collaborative program around BPM, in which event processing is one of the components. There are people who are strong supporters of specialized degrees, and there are also some opponents. Somewhere in the 1990-ies, I have been a member in a committee of the Israeli Higher Education Council, whose role has been to evaluate a new department (and degree programs) in one of the Israeli universities called "communication systems engineering", at that time this program has been controversial, the supporters claimed that it is the best education to a much needed profession, while the opponents claimed that it will create engineers with two narrow focus, and in general an engineer need to be more broad minded. At the end we recommended to approve this program, and it still exists, however, in some of the more conservative universities, it could not have happened.

Nowadays we have some tension between depth and breadth, and I guess that this is more of person own inclination where does the person classifies him/herself on the breadth vs. depth axis. The answer is not black and white, some roles are better occupied by people with depth and deep expertise, some roles are better occupied by people with very broad professional education. Of course, there are people who succeed to have both. Personally I prefer persons that are very good at something over persons that are mediocre in everything, moreover, one of the common mistakes of enterprises in employees development is to try and strengthen areas in which the employee is weak, in many cases, it is more effective to strengthen areas in which the employee is already strong, to challenge the employee to achieve excellence (unless the weakness is in a fundamental area that is critical for the employee's success).

Anyway, each of these programs will have to find its focus to create expertise in order to succeed.

On family vacation in Western Canada

In the picture you can see a rain-forest within the Pacific Rim natural reserve in the western part of the Vancouver Island. This area is less known then other natural reserves in the Canadian Rockies, but IMHO it is the most beautiful one, and we spent several days there at the end of our trip. We started our trip in Lake Louise, in the Canadian rockies Banff reserve, and caught by sleet storm in the middle of walking uphill (we climbed down in heavy slit that turned into rain), but besides the first trip, the weather was relatively cooperative. We continued north to Jasper, where we also had one of the nicest tracks, the Malinge canyon. There is a parking lot up the canyon, and another parking lot down, so one would expect that there will be a shuttle between the two parking lots, as climbing back is not really fun, it seems that the Canadians don't think this way - luckily I found somebody leaving the lower parking lot who agreed to drive me to the upper parking lot, so I could bring the car and take the rest of the family, in turn I also took another person who waited there for the same purpose -- and idea for a start-up!

We spent one day in the Yoho park in the way to Banff, and then continued to the second part of the trip -- flying to Vancouver, going in the ferry to Victoria and driving to Pacific rim (300 KM from Victoria), and then back to Vancouver through the northern ferry. Most of the trip consists on walks in the nature - glaciers, lakes, mountains, and in the west rain-forests. We also sailed in the Pacific rim to an island with natural hot springs, and then in the way back sailed through the ocean to see whales (we saw some). Good trip, I've returned charged with energy (and intimidately got flu, so last couple of days have not been so nice - and I am still behind on Email answering, but am recovering now).

Next trip: Hong-Kong and Singapore (for VLDB) in September.

The book: Event Processing IN ACTION - is now out

I still need to write something about the vacation in Western Canada, but returning to the office today, I have received a package of copies of the EPIA book, that was just published. The project of writing this book (in my spare time) was quite demanding, and was twice longer than the original expectation. I have talked with some colleagues who wrote books for other publishers recently, and found out that relative to their experience, Manning has exceptional quality control procedures, with three reviews by readers during the book's development, and a multi-stage production process with a lot of iteration between the authors and various people on the production team -- technical proofreader, copy editor, proofreader and the production manager.

The acknowledgements section of the book also lists many people who helped in contributing ideas and review and the Manning team; I am also grateful to David Luckham who agreed to write the Forewords section. Last but not least -- working with a partner on such a project requires the ability to agree on many details, and a lot of interaction, many of them in evenings and weekends. My partner in writing this book, Peter Niblett, has complemented me since he came from a different perspective; Peter's drive for perfection has contributed considerably to the quality of the book, Peter is also a very pleasant person to work with.

I also noticed that Manning added a section explaining who is the person on the front cover (some people asked me).

Manning maintains an Authors forum that enables communicating with the authors, this forum helped us during the book's development process to get feedback, and a lot of the comments have been adopted (with acknowledgement to the appropriate person); this forum is being kept alive.

The book itself is printed in black and white, there is also a colored ebook version, available on the Manning's book webpage. The book also included a use-case that is known as FFD (Fast Flower Delivery), and has already several implementations in different languages, with more expected, this site will be kept as live site and is being hosted by EPTS. Interestingly, the FFD case has been used within the DEBS 2010 event processing architectures tutorial in order to demonstrate the architecture notions.

The book has served as basis for both academic and IBM internal courses, and I'll be able to share teaching material with anybody interested to use the book for that capacity.

Some other follow-up ideas is to provide comprehensive authoring tool for the model described in the book, work on automatic compilation from the book's model into various event processing languages and maybe to general programming languages as well -- these can all be nice students' projects, the model described in this book can also serve as a first iteration on standard in event processing application modeling. We'll see how much of the follow-up will be materialized.