An interesting comment to my previous posting on smart cities, said as follows: Even if I am a big supporter of event based systems, I have a question on this topic. Can you explain me, who is the owner of the event based routing, filtering integration services?I hope, that you do not think about weather the government, or some industrial companies, cause this would empower them to rule the world.There need to be a self organizing way like e.g. the Internet, but getting there will be even a larger competition than introducing the web, because of already competing companies in that market segment.I have written before on the big brother when talking about the previous NY governor whose felling out of power began when a computer program indicated him as suspicious in money laundering. This reminds my of somebody who told me that he is afraid that information is gathered in computerized systems and used for other purposes, that person lived in NY, and moved at some point to live with his girl-friend in New Jersey. He did not want to change his NY address for various reasons (among them not to bother changing his driver licence). He said that he never used EZPASS to pay for toll roads, since somebody could conclude from the fact that he drives every working day in the "garden state parkway" northbound, that he actually lives in NJ.
So the question is -- using all the smarter planet services will have the possibility to gather much information about the individual, who owns this information and what can this information be used to. Governments typically do not own infrastructures, but they have powers to make laws that will compel infrastructure owners to provide information, they do it today. Actually Internet service providers has a potential to know a lot about us. The same goes for events, if we'll have smarter planets, there will be a lot of events about individual person floating around, and if somebody will be able to join events from various sources, this somebody have a potential to know much about us. I guess that here there will be a need to have some legal structure that will prevent infrastructure suppliers as well as governments to abuse this information, like the bank secrecy laws in Switzerland.
During my last visit in the USA I have purchased and now reading the book "the last oracle" (nothing to do with the software company that bears this name), one of the thrillers that are based on some history and span around the world, exposing some scheme that will bring disaster on the universe. It seems that this type of books became popular recently.
Anyway, I wanted to come back to the issue of states that I have recently blogged about. Actually the term state in event processing are somewhat overloaded. A state may refer to any of the following:
- The internal state of a single agent -- when trying to detect a pattern (say - E1 and E2) there is a need to keep the state, e.g. the fact that E1 already arrived but E2 did not. This can be kept either by keeping the input event as an object, keeping another object somehow derived from the original event, or keeping some internal data structure that helps in the detection.
- An event store where events are kept for further consumption by other agents -- this is a global (or bounded) state.
- Reference data that is being used by event processing agents for enrichment, this data is not maintained by the event processing system, but can be considered as part of the event processing state, since the event processing results may depend upon values of this data.
- Global variables (whether persisted or used on a shared memory) that are being used as a global state across event processing agents.
- Context -- on which I have blogged several times, which may have temporal, spatial, segmentation and external state (which is indeed a global variable) dimensions.
All of them are in a way part of the system state, they have different roles, though.
BTW -- we have revised the building blocks that we are using in the EPIA book, by replacing event derivation (which is actually part of the event processing agent) with global state.
Among the types of states we mentioned before -- context is a first class citizen in the model, internal state is hidden inside the implementation and is not explicitly modelled (although some languages allow also to model the internal state explicitly) and global state contains all the rest (reference data, event store, global variables). More - Later.
The IBM Institute of Business Value has recently published its vision of smarter cities, which is part of the "Smarter Planet" vision on which I have written before. As noted before, the smarter planet vision is based on the fact that the world will be instrumented, interconnected and intelligent. Event processing is the backbone of smarter cities. As smarter cities is been labeled "system of systems", the integration and communication between the different systems is done using events, and the event processing filters, transforms and routes the event among systems. It also crates derived events from one or more systems that may be used by other systems. Event processing technologies have two roles: In the macro level, it is a glue between the systems and is part of the intelligent part of each individual system. I'll write more about some individual systems inside the system of systems. I am trying to get a smarter cities keynote talk in the 5th EPTS event processing symposium --- stay tuned, details about the symposium (Trento, September 21-23) are coming soon.
I am now in my hotel room in Tarrytown, NY; one of the nearby towns is known as "Village of Sleepy Hollow", I always wondered about the origin of this name, and who wants to say -- I live in sleepy hollow.. I have seen some other strange name. For example, there is a USA small town called - "bird in hand".
I have noticed that Pedro Bizarro has posted the DEBS use case tutorial provided by the EPTS use case work groups on slideshare in form of 7 presentations. Enjoy!
It is interesting to note the "lessons learned" presentation, it summarizing ten lessons.
- Architecture
- Lesson 1: State-based event processing
- Lesson 2: Incident objects
- Lesson 3: Integration with other data management systems
- Lesson 4: System of systems
- Languages
- Lesson 5: Query languages
- Lesson 6: Classification and rules groups
- Lesson 7: Customization
- Glossary
- Lesson 8: Changes to glossary
- Use Cases
- Lesson 9: Changes to questionnaire
Lesson 10: Better instructions to describe a use case
I agree with the state-based lesson; I am advocating the notion of context which can include various types of state and segmentation, and I view it as a first class citizen in the model. This is both architecture and language issue.
What they call "incident object" seems to be an object that describes derived event that has some internal structure which can be updated by programs. This seems to be quite trivial notion that events need to have some structure, and be able to participate in processing other events, may be retained over time; I think that this is true for any event, as event may also mean "event object". About the question whether it can be updated by programs, this is a matter of taste, since they want to keep these objects in temporal databases, temporal databases has adopted an "append only" model, where changes create other objects, and there is some reasoning behind it. I may dedicate a postings to this topic.
Integration with other data management systems -- given that Dieter Gawlick is co-leading this workgroup, one can expect that there is a database-centric view of event processing, his views on this topics are well known. I agree that databases can be integrated with event processing systems for various reasons (state keeping, reference, meta-data holding, high availability...), but event processing may also be integrated with a messaging system, with business process management system, with a BAM system, and with sensor networks, so I would extend it to integration with other parts of the enterprise architecture.
System of systems has to do with modularization and some implementation issue (e.g. bounds of local and global states). In my view EPN can be defined recursively such that an EPA can represent an EPN, thus we can express "system of systems". I am not sure that this requires any conceptual difference.
Query languages -- I am not sure I got the exact point here, but it seems that all event processing styles require to express conditions such as predicates for various uses. I think this is already happening. The issue whether internal state can be queried is different, and not relate to the language issue.
Classification / customization-- This is again one of the context dimensions, as context can classify populations according to classification predicates. Languages that are more syntactic oriented like the SQL derivatives hide classification inside SQL Where, but where context is supported as a primitive concept then classification is an explicit construct; this also allows customization. The use of vocabulary, as been used in rule languages, can be helpful here.
Glossary -- I'll leave the glossary discussion to the glossary workgroup for a while.
Use cases description --- the goal of this entire exercise was to "debug" the use cases template; I hope that the use case workgroup will finalize the template soon, and move to the next phase -- getting a substantial number of using cases and draw conclusions about their types, requirements etc--- it will be a great help to the community if we'll succeed to classify the applications to a finite, small, number of classes, where in each of them we'll be able to specify important functional and non-functional requirements. More on this -later.
Tarrytown, NY. I have left Nashville this morning and arrived to Westchester county, NY, to do some IBM meetings on the way back from DEBS. I think that the nicest analogy I got from the conference came from Alex Buchmann keynote address, he used some other pictures, but provided the analogy that while regular programming is like drinking with a straw, this is good when the data is standing, while the data is moving, like in event processing, using the same kind of thinking is similar to use a straw to drink from a waterfall. I like this example. It requires some education to understand that event driven is a different paradigm of thinking, and this impact the way we should think about the programming model as well as the execution model of these types of applications. This also links to the discussion on education that will create a critical mass of developers who understand this paradigm. It is somewhat different, but it is not rocket science, with some education, I believe that it is doable.
I'll end with some travel anecdotes: Today I rented a car in the airport, got to the car and did not find a car key, just a remote control; it took me two minutes to understand that the remote control serves also as a car key, there is a hole in which it has to be inserted to, and when it is there, there is a on/off button which turns the engine on or off. I showed it to one of the people I've talked with in the place I visited and he told me that he has even more sophisticated electronic key that he does not need to insert it anywhere, it is enough that it is in his wallet that the car identifies it and enables to turn on the engine.
Another anecdote that always puzzles me is hotels' superstitions regarding the number 13. My Nashville hotel was semi superstitious, it did not have 13th floor, jumped from 12 to 14, however I have stayed in room 413. Now I am in a different hotel, this time in room 412. For curiosity I have looked for room 413, and did not find it, it seems that the level of superstition here is higher... I never understood this superstition, but this is not the only thing that I don't understand... Go figure.
I am still in the DEBS conference venue, in a time gap between the conference sessions and the conference dinner that will be held here in an hour or so. This is the time to provide some more comments about DEBS 2009. The conference will continue tomorrow, but I'll skip the last day, going to NY for some IBM meetings. Spending whole three days in a conference is a heavy toll on my busy schedule, but it is fun. The next conference will be held in Cambridge, UK, and the call for papers (above) has been distributed. Today Alex Buchmann gave the second keynote address about some event processing applications and research challenges, I hope that Alex will make his slides available to the public. Also today there have been sessions of demos and posters, mostly by Ph.D. students, mostly concentrate on the transport area, pub/sub, messaging etc... We need more Ph.D. students on the event processing topics. Next there has been a fast abstract session that present work in progress in 10 minutes. I have presented a work in progress on spatial aspects of event processing, and will write more in the future about this activity.
Back to yesterday, the industry track was divided into two parts: industrial reports and a panel. In the industrial reports there was one, presented by Florian Springer, that the audience will remember for long time, not so much because the content of the talk, but since Florian made a point about importance of standards, by showing real-life examples for standards, and the example he chose was condoms, which, according to Florian, have a standard - they are all in the same size. People kept talking about it.
I have moderated the panel about academia/industry relations in event processing, and posed the following questions (see slide below)
The first question was, whether the academia should work on incremental stuff related to current technology, or on disruptive technologies, making current technologies obsolete, which was translated to the question --- do we want to clone John Bates to develop new technologies that will make the technology created by the original John Bates as obsolete. John Bates, who was in the audience, reacted by saying that his wife will not like the idea of cloning him, since she thinks that one of this type is more than enough. The panelists were two persons from academia, and two from industry. One of the industrial people was Richard Tibbetts, Streambase CTO, who showed the famous Gartner hype-cycle picture, claiming that the event processing area is already approaching the plateau. I think that he is a bit optimistic, in my opinion, we are not even at the peak of inflated expectations... 
There was also some discussion about teaching event processing courses, and the fact that today it takes a lot of time to get a university graduate being effective in developing event processing applications, since they need to be taught to think in a different way. There have been some more talks, but I am tired from writing, so that's all for now.... Tomorrow early morning -- flying to NYC.
Yesterday evening the "open business meeting" of DEBS took place. This meeting is intended to draw lessons from the organizers of this year's conference, and obtain decisions about the next years' conferences. I participated in the business meeting two years ago in Toronto, where Mani Chandy and myself have challenged the DEBS organizers to extend the scope of the conference, and this seems to happen, the conference is shifting its focus from being mainly a "pub/sub" conference, to a conference that covers the entire aspects of event processing (I view pub/sub as an aspect). Last year in Rome, we suggested to host DEBS 2010 in Haifa, however, the decision was to do it in UK, and give us a chance to host DEBS 2011. Indeed, the next DEBS will take place in Cambridge, UK. In the meeting yesterday the decision about DEBS 2011 organization was to be decided. A day before, Arno Jacobsen, who is the person behind the DEBS conferences (chair of DEBS steering committee) addressed me and said that they still want me to organize DEBS 2011, but in the New York area instead of Haifa. He mentioned two reasons - the first one is that they want to rotate the conference between Europe and North-America, so in 2011, it should be held in North-America again, and the other reason is that people may be afraid to come to Israel, which will reduce the number of participants. This is an issue, since in Nashville we have a substantially less participants than we had last year in Rome, and they thought that NY is easily accessible place both to Americans and Europeans. I think that the reason that there are less participants this time is a combination of the economic situation, which travel restrictions in many places (especially in industry), and the location, while the conference is in a nice setting, it needs connection flights even for some Americans. Another reason is that I think that most of the community in Europe is larger than the north american community, also a couple of participants had to cancel due to visa issues. We don't know what will be the economic atmosphere in 2011, but hope it will be changed. About the fact that the potential to have more participants in NY than in Haifa, I tend to agree. About the argument that people may be afraid to come to Israel, I stated my opinion publicly in the business meeting yesterday that this is a distorted perception that is a result of hostile media coverage in the western world. Personally I feel safer in Haifa than in Nashville and certainly safer than in London, New York, or Paris. Having said that, I am aware of the fact, that it is difficult to fight perceptions, see my posting (on another matter) about historic truth vs. archaeological truth
Bottom line -- together with my colleagues in IBM Watson Research Center, I agreed to organize DEBS 2o11, this will take place in one of the IBM sites in the NY areas, we still need to determine the exact location. Stay tuned.... Meanwhile, we still have to finish the DEBS 2009 and survive the next 2 years...
