On the virtualization of event processing functions

There is some discussion about scale-up and scale-out for measures of systems scalability, as indicated by a recent Blog of Brenda Michelson, I would like to refer to the programming model aspects of it. Parallel computing becomes more and more a means of scalability due to hardware development and barriers in the scalability of a single processor that stem from energy consumption issues. In event processing both parallel and distributed computing will play important role, as we a large, and geographically distributed event processing networks.

The main issue in terms of programming model is that manually programming a combination of parallel programming and distributed programming is very difficult, since many considerations are playing here. The solution relies on the notion of virtualization. The event processing applications should be programmed in a conceptual level, providing both the application logic and flow, but also policies that define nonfunctional requirements, since different applications may have different important metrics. Then, given a certain distributed configuration that may also consist of multi-core machines, the conceptual model should be directly compiled into an efficient implementation based on the objectives set by the policies. This is not easy, but was already done on limited domains. The challenge is to make it work for multiple platforms. This is part of the grand challenge of "event processing anywhere" that I'll describe in more length in subsequent posts. Achieving both scale-up and scale-out in event processing require intelligence in the automatic creation of implementation, and ability to fully virtualize all functional and non-functional requirements. More - later.

On the holocaust and the use of negative thinking

It is Holocaust memorial day here, a year ago I have written in this Blog a story about my father and King Alfred. This story brought me a lot of reactions. This year, the holocaust memorial day for me is in the shadow of my recent research about the family history, which I have neglected to do until now. The picture above is the famous gate to the Auschwitz concentration camp. I never had uncles or first cousins from my father side. Unlike myself, my father has not been an only son, he had seven brothers and sisters, five of them ended their lives in a single day in this place, the other two ended their lives even before getting to Auschwitz in Ghetto Lodz. Until two weeks ago I did not know most of their names, ,my father rarely talked about his family. My family history research revealed me all the names. The Nazis were very thorough and documents everything, today one can find a lot of information on the Web, though some effort is needed. My father was the second of the eight children and the only survivor. His elder brother Yehuda, died in the Ghetto in the age of 19, another of his brother, Chanoch, died in the Ghetto at the ages of 12. In that day in Auschwitz, my aunts and uncles whom I never met, were poisoned in the Gas chambers, like the people in this picture who march to their death.

The names are: Chana who was 18 years old, Eliezer - 12, Roza - 11, Rivka -10, and Miriam - 8 years old.

I would like to take this opportunity and connect the holocaust to another thread of off-topic posts that gained a lot of reactions, the one that talked on positive and negative thinking. There are people with negative thinking who just waste their energy on looking at the world in a negative light, these people are often annoying, but the more dangerous people are those who use negative thinking in order to get support for their agenda. It is always easy to unite people behind negative message, behind common threats, whether existing or imaginary. People are getting support to their political agenda by turning people against groups of people, the negative sentiment is unfortunately very effective way to get people's support. This is what enabled the extermination of millions of people, in what is known as the holocaust. This is also a warning for the future.

OMG event processing symposium and other event processing conferences

I was recently invited to give a talk in the OMG event processing symposium that will take place in Washington DC (actually, Arlington VA) in May 24-25, 2010. I will give a talk about -- event processing seven years from now --- talking about where the event processing area goes in the future, this will be a week following the Dagstuhl seminar on event processing, and some of the conclusions may be reported there. The OMG new event processing consortium that will be kicking off in that event is another indication to the importance that the IT community places on the event processing, and the visibility it gains. This is a customers' oriented conference, whose aim is to educate the general community about event processing.

We also announced that EPTS will be doing its annual conference in November, co-located with the Gartner Enterprise Architecture Foundations Seminar, in Los Angeles. In this conference EPTS (possibly in collaboration with OMG) will grant two kind of awards: event processing innovative foundation awards (for research work) and event processing innovative application award. Stay tuned for call for nominations soon.

More on family trees

Still holiday week, and I am spending some of it on investigating my family history, got to over 200 leaves in the family tree, and keep growing, on one branch of the tree I already got to the 18th century. This is a picture of a second cousin of my father, her name is Pola, and she was perished in the Holocaust in the age of 23, found her picture, since her father who survived the war, documented her, nobody can mistake her to be part of our family, she has striking resemblance to some of the people I know. This is an interesting process, and I am renewing connections with relatives that I've not talked with in years. Back to "event processing thinking" after the holiday in Tuesday.

On family trees

Taking advantage of the holiday week to do some shopping, since trying to move the hour in my watch to "summer clock" (what is known as: Daylight Saving Time), I've discovered that the watch is getting older, and it is not easy to tune it, I decided it is time to buy myself a new watch, my previous watch was digital with two different time zones, two stopper watches, alarm, and some more options. I decided to go back to basic and return to the good old analog watch, and chosen a locally manufactured one shown in the picture. Speaking about "back to the basics", I have also participated yesterday in an extended family meeting, one of my relatives decided to kick off a project of investigating the family tree, find unknown relatives in various places, and understand how some relatives whose relations is unclear are related. Creating family trees is kind of a fashion. The idea is to do some investigation (and roles have been distributed between us), and do an extended family event (yesterday only part of the people were there) with presentation about the family history. I have never been into investigating family histories, but the younger generation seems interested, so we'll give it a try... and will try to renew connection with some more distant relatives (e.g. the large Canadian branch of the family), can be interesting... The organizer also took pictures and already posted some of them on his website, the second raw has pictures of me, my wife and three of my four daughters (the eldest one did not join the meeting).

More on event processing agents

There are various agents types of agents in reality, like the one shown above, likewise there are various agents types in computing, as I've discussed long time ago in this Blog, but since that time the thinking was somewhat evolved. Recently, Jim Odell, a long time agents advocate, has been hosted by the TIBCO CEP Blog, and advocated the use of agent technology in event processing, providing scalability as main motivation. In the Event Processing in Action book we are making EPA (Event Processing Agent), a term coined by David Luckham, as the most notable building block in our model. The term agent is used as software agent, and not necessarily agent in the AI sense. We are using the term agent as a meta level, where in the run-time level there are agent instances that can be implemented in various ways. Event Processing Agents in event processing are event-driven in the sense that they are taking one or more events as input, perform some processing on these events, and derive one or more events.

EPA can filter events, transform events, detect event patterns or do any combination of the above. Event processing agents are typically (but not always) associated with context, thus context related operations determine when an EPA instance is open or close, assume that the context is a temporal sliding window of non overlapping periods of 2 hours, and there is an EPA associated with this context, then every 2 hours, an instance of this EPA terminates and another instance is initiated. In essence EPA interacts with events in various ways:

• An EPA receive events as input
• An EPA processes events
• An EPA may query historical events
• An EPA derives events as output
• Through context -- an EPA instance may be initiated or terminated by events

Each EPA instance is autonomous in the sense that it does not communicate with any other EPA instance, and thus can be implemented by different run-time artifact, which indeed can enable scalability. However, there can be various grouping of EPA instances to run-time artifacts, where the two extremes are: run-time artifact for every EPA instance, and a single run-time artifact that contains all the EPA instances within the application.

Benefits of using EPA are -- simplicity of the model, modularity, and as said flexibility in implementation that may support various scalability and performance objective functions. I'll write more about EPA - later.

Some media events

I have written last week about participating in an entrepreneurship panel, where there has been a discussion on whether innovation in big companies is counted or not, it turns out that the echo of this discussion got to the national press in Israel, although you have to read Hebrew in order to understand what is written. Well - for those who do, they did not quote me accurately, and even did not spell my name correctly... go figure.

And from past media event to future one, in Thursday I am participating in an IBM Webinar given on the ebizQ platform. The Webinar is a blend of the IBM "business agility now" message and its relations to event processing, and some selected topics from the EPIA book. The Webinar is planned for Thursday, March 25th at noon USA EDT. You are all invited.