On students' projects in the event processing course

The semester has come to its end, and the event processing course in the Technion had the last meeting today; the students still has to finish the projects, and they have some time to do it, but today they have presented a preview of the project, here is a brief summary. They were given a choice to do research oriented project or implementation oriented project, those who chose implementation oriented project got some directions on the size (quantities of consumers, producers, patterns, aggregations, filters etc...), and could chose both the application they want to work on (and they'll have to test it with simulated data) and the tool they use in order to implement.

There are five teams:

• Team no. 1 -- chose to work on a research problem of interval-oriented event processing and defined a collection of operators and patterns required to support interval oriented.

The rest of the teams chose implementation projects:

• Team no. 2 -- chose to work on "smart traffic control", simulating traffic lights, traffic jams, accidents and events like concerts etc... They decided to hard-code their application in Java, since they know Java well, and it is more fun doing it this way :-).
• Team no. 3 - chose to work on "smart home" and mainly concentrate on the security aspect of home automation, they are also Java fans, but chose to use Esper in conjunction with Java.
• Team no. 4 -- chose to work on control of people who are abusing chat rooms in various ways (like using dirty words), and managing complains. They have chose to work in Streambase, and done all implementation (so far) using the graphical tool of Streambase. We again see that some people like graphical programming and some people prefer old-style imperative programming.
• Team no. 5 -- chose to work on tracking bus driver performance, both for safety issues (speed), and accuracy (skipping bus stations). They have organized late, so have not started the implementation phase yet, but they said that they wish to do the project using Apama.

The projects are due in August, and then I'll write more, if I'll have any interesting insights.

On image and unfair disadvantages

This is the logo of the "Hebrew Reali School" - a famous Israeli private school, in which I have spent 8 years of my life, my wife and myself are both products of private schools and we sent our daughters also (at some point) to study in this school. Yesterday I spent all evening there, in a ceremony to celebrate my daughter Hadas graduation from junior high (9th grade), she will stay in the same school, but move to a different location for the "upper division". The coordinator of the 9th grade classes in his speech claimed that by learning in that school the children gain advantage over other children for their future, from my experience - this is definitely true. Some people may view it as unfair advantage, but that's life -- the public system provides a low common denominator, and if I can afford getting more than that, I am not ashamed of pursuing it. I admire people that I know who send their children to public schools because of ideology and values , but I don't share their attitude.

Somewhat unrelated --- I read in Marco Sierio's Blog that quoted an article in the Atlantic magazine about the unfairness of using high-frequency-trading, Marco asserts: When this is what the public reads about CEP, then I think we might have a minor image problem.

Some quick comments about Marco's assertion:

1. I did not find the phrase CEP in the article, it has talked about high-frequency-trading. If my memory does not mislead me, Marco's company has a nice event processing software-as-a-service offering, and its typical applications are not really high-frequency-trading. This is also true for what we do in IBM in this area (there are many applications in many industries), and other companies as well. Algorithmic trading applications were indeed some of the early adopters of event processing technology, and there are some vendors that are still centered around trading applications. However, one of the issues, that is indeed a misconception that exists among some people is the event processing = algorithmic trading, but this is only one of the applications, and there are many others.
2. Furthermore, there are some other applications that are indeed aimed at the benefit of humanity as a whole, and our event processing grand challenge as noted in the EC FET FLAGSHIP presentation (that we want to realize with the help of government funds) is certainly in that direction.
3. Now getting back to the unfair advantage, I cannot say that I have deep understanding of the trading issue, so unlike some people who have opinions about everything on earth, I tend not to express opinions about something I don't understand, so I'll leave the discussion to those who understand more and avoid expressing opinions here.
4. However, I'll write in general about using technology for providing unfair advantage. When I was young, I worked in the IT shop of the Israeli Air-Force, and saw how sophisticated informations systems have shifted the balance of power in organizations, gave one organization advantage over others, and eventually changed the organizational structure and missions. I noted how some people could take advantage of technology in order to advance the organizations they managed, it was fascinating lesson. Was it unfair to the others? certainly, one can think this way; but nothing stopped the others from doing similar things, those who made high priority in investing in technology benefited in more than one way, there is nothing special in that assertion. Event processing technology can certainly provide real-time observations or automated actions in many areas, and one cannot stop the progress of technology.
5. Again, I don't know if some traders abuse the system, this is a question to regulators, and I have no knowledge and opinion about it -- but it is well known the terrorists have used Email and SMS to coordinate among themselves. Every technology can be abused in some way.

Getting to the beginning of the posting -- I am sending my children to private schools to give them advantage in life; likewise the wise organizations take advantage of sophisticated technologies to get advantage in many areas -- and I still have no opinion about the high-frequency-trading, as I don't know much about it.

On Sapiens veterans reunion

Yesterday, I have driven 1.5 hours (each direction) to participate in the Sapiens veterans reunion; I have worked in this company from 1983 to 1986, and is considered one of the old veterans. The reunion included people who worked there until 1994, some of the people I knew and some I did not. Sapiens had a product, whose initial name was DB/1 which was way ahead of its time, and enabled very high level language of database programming, that made the development of applications quite shorter (if you succeeded to become skilled in it), I discovered it while being in the Israeli Air-Force, and was part of a team that has built a complicated and substantial application in 4 months (later we got the Israeli Air-Force commander award, the first ever such award for a software system, I have it on my wall framed). While there has been some success, it did not conquer the universe in the way it could be. Which shows that amazing technology is neither a necessary nor sufficient condition to do it.

It was nice to meet many people that I have not seen for years, one of the characteristics of the Sapiens company (at least in my time, and according to the stories I've heard, also later), was the unique atmosphere of "unit pride" that I don't see in my current work place. My work over the years was greatly inspired from the years I've spent in Sapiens, and also the driving to use high level languages, since using Sapiens, I could never return to like imperative languages.

The meeting also resulted in some more LinkedIn contact, checking the count now, I have 824 contacts. My wife told me that when I'll get to 999, she'll join LinkedIn and becomes the 1000th -- so I still have some way to go.

The decision model - Larry Goldberg's talk

A few months ago I have written a review about "THE DECISION MODEL" book. Today I have hosted in the Haifa Research Lab one of the authors, Larry Goldberg, who gave us a live talk about the model. The decision model provides table oriented representation of rules, where a table designates a family of rules which share the same consequence (i.e. the "right hand side" assigns value to the same fact), tables are connected in a way that a fact that is an outcome of one table is an input for another table, this brings some order to rules, and also fits both inference rules or computational rules. I see strong benefit of using such structured way, and also possible to build an hybrid flow of "rule families" and "event processing networks". I think that there is a future there. I was involved in the past in some similar effort to provide some structure to data-driven rules, a short ACM SIGMOD RECORD paper is referenced here.

I'll have a short presentation in DEBS 2010 around the relations between event processing and business rules (in the fast abstract session).

OMG event processing symposium - replay is available now

For those who missed the original broadcast of the event processing symposium - a replay is now available. My presentation entitled "event processing - seven years from now" (Brenda Michelson asked me to talk about "event processing in 2015", but I thought that 2017 is safer...) is now on Slideshare. I am just listening to myself now, it is quite strange to hear your own voice and realize your English accent is somewhat strange. Enjoy!

Some subjective reminiscences on Amit

Catching up on Emails and community's Blog after returning home, I have discovered that Paul Vincent has added Amit to the genealogy of products that he is maintaining. There is also going to be in DEBS 2010 a paper presented about experience with Amit, written by some of our team members (I did not participate directly in this paper, since it described work on some use cases that were done in the period that I have spent outside Haifa Research Lab).

I wanted, for a long time, to write something about Amit, maybe this is the right opportunity.

What is Amit?

Amit has been a name of a research project resulted with what is called today "event processing engine". It is based on ECA rules (however, had originally been specified as extension to SQL, as done in active databases). The original specification and design have been done by Asaf Adi (who also single-handedly coded the first version) and myself in 1998. The original language has survived with some additions over the years.

What is was used for?

Amit was used as a component in some of IBM products over the years, most notably Websphere Message Broker, but also Websphere Sensor Events and some others; also it served for some customers engagements as stand-alone, both IBM internal and external customers. The applications developed through Amit spanned various industries. It was also OEM-ed to two external software vendors and served as part of their solutions.

Where is the name Amit coming from?

The official name has been - "Active Middleware Technology" and at some point, somebody in UK decided that the acronym should be written as AMiT, thus you can see that in some places it is spelled that way, however, the acronym came later -- the original name was indeed Amit, and it was called this name since Amit is an Hebrew name, and I chose Hebrew names to all my research projects (Pardes, Adi and Arad are some of the other names), this particular one is also an Indian name, and one person whose name is Amit, who was born in India told me that the meaning of the name in Hindu is "to aspire beyond limitations", actually looking Amit at Wikipedia, one can found that the Hindu meaning of the name is: Infinite or immeasurable or boundless. I liked the aspiring beyond limitations, since that when we started it we did not know where it will lead, but we had a feeling that this is something big, a feeling that the immediate environment did not share (hence the limitations). Later, I have coined the "Active Middleware Technology" acronym, and the plan has been to develop a middleware with many components, we started with a first component called: situation manager, and somehow this was the only component developed, as we went deeper in this one, thus the situation manager component became a synonym of Amit. Again -- history develops in strange way.

What is the impact and legacy of Amit?

With the acquisition of Aptsoft, in early 2008, the IBM executives decided to retire Amit, and it is now still used as a legacy in some places, but not been developed for several years now. However, the impact of Amit has been and still is noticeable. IBM has recognized the impact last year through its internal awards process, that the activity around Amit, and some other related activities was the direct reason that IBM has decided to enter the EP market.

Looking at technical impact -- since we have published detailed description of Amit in journal and conferences papers, it was read by many people in the community, and we identified ideas in various products (even names of operators) inspired to Amit, I'll not finger any specific product, though. I think that some of the ideas that we worked on (which have been developed further in the post-Amit period) like contexts and policies laid foundations for the next generation.

As the paper written about experience with Amit states -- the experience with Amit also taught us where the event processing platforms should evolve to, and triggered the post-Amit work on the EPN/EPA based event processing that we are working on in the last couple of years.

This is a very brief account on Amit, I might write some follow-up to this one telling more.

The EU FET Flagship program

Back at home. In the previous two days I have attended an EU conference in Brussels on FET (Future Emerging Technologies) Flagship. The EU is recently acting to diversify its R&D investments, and launch some new programs in addition to the traditional programs that will keep running. One of this new ways is PPP (Public/Private Partnership) that is aimed at short term more engineering oriented projects, while the FET flagship goes to the opposite direction: launch two very ambitious large 10-years projects, with around 1B Euro investment in each.

They have brought some examples of projects in that scale such as: Large Hardon Collider, The Human Genome project and some others, with varying degrees of success. The rule of the game is that 5-6 ideas will be given a year (and a funding of around 1.5M Euro each) to develop the program (including partnership around the world, funding sources -- the EU will fund only part, expect that national funding agencies will support it as well as others), and out of them 2 will be selected.

In the last couple of days I have heard a lot of talks about potential proposals, many of them will probably merge, others will disappear. The commission people are encouraging people to merge and share, they don't think it is reasonable to have more than 10-15 proposals. One of the important tests for the proposals is the grandmother test: If you can explain your grandmother in one sentence what it does in terms that your grandmother will understand, then you'll be able to sell the idea to politicians. IMHO most of the presentations I have heard fail to satisfy this property, especially all the ones which has technology in their center. The promising directions relate to Biology -- human brain, breakthrough in robotics, and socio-technical systems.

What is the relationship to the "event processing grand challenge"? --- this may be one of the vehicles to advance the grand challenges, however, since the two flagship projects will start in 2013 -- we need to find other avenues to start earlier.

Event processing is relevant to everything, the human brain is very much event-driven, robots are in many cases event-driven, social evolution is event-driven and more.

One of the challenges is to sell the contribution of event processing to these projects, in discussing with one of them, they thought that they indeed need to process events, but there is no research here, just hard dirty work of programing. I told them that our research issue is how to eliminate the hard dirty work of programming in conventional languages and provide for them something much more high-level with "easy clean work" instead. They have not thought of it this way, since we have not reached out to these communities yet, I gave the analogy that 40 years ago everybody has created home-made database, until the relational database changed the picture, this is exactly what we are doing, raising the level of abstraction.

Next --- see where this can fit and how it can intersect with the EP grand challenge, and determine how to position it (in my talk on live ecology I have equated event processing to the nervous system of a human body). My presentation on live ecology is now on slideshare.

More - later.