Over the years I faced a lot of skeptic people, for the entire idea of event processing. In the early days when I explained people what event processing is, one of the common reactions was: this is not really practical, you would never be able to get hold of the events you need for this application. This has some truth in it, the producer and consumer parts of event processing are often the most time and cost consuming parts of an event processing systems, However, the availability of events increase with time, since many of the current trends, like the IBM smarter planet, are based on the assumption that events of many types and many sources are available. The picture above the the quasi-logo of the "Internet of Things", a phrase I always thought as somewhat funny. The idea is that any "thing" - car, building, cell phone emits events to an Internet scale infrastructure - which gained the name IoT. There is now a lot of work around IoT,
Research and Markets recently issued a report about IoT, called it a potential game changer. The report also mentions event processing as one of the ingredients of IoT. Here - the relationships comes from both directions, IoT infrastructure is a source of many events that can be used of many applications with event processing functionality as part of them. The IoT infrastructure itself requires event processing for filtering, transforming, and reducing the number of events by reporting derived events based on trends, or situations of interest. We are now looking of some potential use cases for our proactive computing project, and the relations to IoT is clear. While IoT typically works on the producer (sensor) side, there is also a consumer (actuator) side to it, I'll write more about sensor and actuators examples later.
Yesterday I spent the evening in the theater, watching "All my sons" again (in Hebrew). The first time that I watched this play was in the age of 19, I have tagged this play as the best theater play I have ever seen and the actors who played there as the best actors I have seen on stage. These two assertions have not changed over the years, it is still the best play I've ever seen, and the actors who played first time I saw it are indeed the best actors I've seen on stage. The actors played yesterday are good, but not in the same league of the two giant actors from the past.
"All my sons" is now in Israel plays to the popular sentiment. It talks about taking the greed, and making the "business' and profits in the center of the universe. In this play it takes it to extreme. shipping damaged airplane parts to "save the business" and causing many people to die is considered as extreme behavior, but the principle of putting profit in the center, is one of the cornerstones of the western culture. Israel is now experiencing mass demonstrations with the slogan "the people demand social justice". In Saturday night there was the biggest demonstration that the country has experienced. This wave challenges the western culture thinking, talking about "swinish capitalism", where cost of living for much of the population is too high since a relatively small group of people who are very rich, control most of the branches of the Israeli economy, with monopolies or oligopolies, and keep the cost high.
Maybe in a social utopia, big corporations would say: "we earn a lot of money, we can give up some profit in order to raise the standard of life for the common people, either by offering some cheap products if they are in the retail business, or sell directly to common people, or if they sell to such businesses, do it indirectly, by reducing price and in return ask them to reduce prices of some products to the end consumer, or if the customer is a government, use the reduction to offer more social services",
Of course, social utopia has nothing to do with reality, the companies work in the real world of Wall Street analysts, executive bonuses linked to accounting metrics, and stock prices. I wonder how many companies that would face the Joe Keller's dilemma from "All my sons" -- do something morally wrong, or risk losing the business, would resist the temptation. No wonder that Arthur Miller's play was accepted in unease in the USA in the 1940-ies, and earned him the title of communist. But I think it is good as a food for thought.
The event processing area keeps evolving and there are more terms being used. The EPTS glossary workgroup, co-chaired by Roy Schulte and David Luckham has published a new version of the EPTS glossary. It contains both old and new terms.
As a follow up to the transactions vs. events posting, I would like to discuss some interactions between event processing and events. ACID (Atomicity, Consistency, Isolation, Durability) are the properties of classic transaction systems that guarantee serializibility.
Some people think that event processing and transactions live on two separate universes, as event driven architecture advocates asynchronous mode and decoupling. Note that there are transactional asynchronous systems (e.g. messaging with guaranteed delivery).
One of my past Master students, Boris Shulman, has investigated the issues of interactions between transactions and events. There was no paper published from his thesis, he has done the thesis while working for IBM, but then went to work in a start-up and did not have time to complete a paper, and the topic was not on top of my priorities either, but recently I am thinking about it more.
In the thesis he pointed out several different motivations to look at the combinations of events and transactions:
- A monitoring system that monitors a transaction system, events are emitted from the running transaction systems, in a way, emitting such events before transaction commit, violates the "isolation" requirement of the ACID. However, at some cases there are timing constraints, since the monitored event need to be processed without delay. This gets complicated when the transaction aborts, and all transactions operations rollback. In this case the monitored event conceptually never happened.
- Event processing is done as part of an actual transaction, the producer and consumer are part of a transaction, and the event processing is doing some processing that happen to fall into the functionality it support, but it is part of transaction. Note that this does not view event processing as a stand-alone functionality, but as a functionality that can be embedded in other systems that can be transactional. In the past when we did the Amit nodes inside IBM Websphere Message Broker, it worked exactly in that mode, event processing as a specific node within a transactional message flow.
- The event processing network (or a certain path in it) may have a requirement to behave as a single unit of work. This is true when an EPA reads/write data, but also the raw events and derived events may issue transactional behavior.
There might be other cases, but these cases indicate some cases for interactions between transactions and events. The transactions that relate to events, may not be pure ACID transactions, I'll discuss other model of transactions that might be appropriate - later.
Paul Vincent posted on the TIBCO Blog some thoughts on transactions and events. His story was mainly about bookings in a sleeping train that should have been in the same cabin, but the system did not have any way to do it in a single transaction, In DEBS 2011 one of the keynote speakers Johannes Gehrke talked about similar things, of not being able to coordinate travel reservation of two co-workers who want to fly together (but still want to have independent reservations for the rest of the trip). This is a symptom of the "old world" vs. "new world" phenomenon. The "old world" in IT is the world of OLTP, which is centered around ACID transactions (maintaining: atomicity, consistency, isolation and durability). The "new world" is a world of mobile apps, social media, NOSQL, Web 2.0 and more - this world has started from scratch, the "cool" geeks who created it don't want anything to do with the nerds that work in the old world (slang I got from watching American TV programs, starting from "happy days"), Anyway -- it seems that some relations do exist between the old and new worlds:
- New world applications should use some old world techniques like transactions, but this is not necessarily ACID transactions.
- New world applications should sometimes talk with old world applications
- Old world applications should support some of the new world characteristics -- and have to evolve.
All these three relations do exist, I'll write in the future about each of these topics.
Last but not least -- back to the title of transactions vs. events. Event processing seem to have some of the characteristics of the new world, however, there are some synergies again between transactions and events.
One of my earliest posts on this blog, in 2007 had the title: "Event processing and transactions - real, real time, and real time enterprise", where I discussed briefly the idea of event processing within transactional system. Since then there was some progress in the thinking - I need to revisit this issue on the blog.
So made some promises on future posts on this Blog - and now back to the new world of looking at what is new on Twitter, Facebook and LinkedIn... More -later
According to Wikipedia, Siddhi is translated into: perfection, accomplishment or unusual skill. In the picture we can see the eight primary Siddhis. Siddhi is also a name of an open source event processing engine, recently advertised. From the basic description it is aimed to support event processing as stand alone applications, and experiment with optimization algorithm for the run-time engine. Siddhi came from University of Moratuwa in Sri Lanka, a university who claims an ambition mission: "to be the most globally recognized knowledge enterprise in Asia". I like ambitious goals, and probably making open sources like this can contribute to the recognition, of course, if this will get traction.
I know of event processing open source and products coming from the USA, UK, Germany, Austria, Sweden, Israel, Australia, and I probably miss some (will be happy to update this list) So now Sri Lanka to the list, We are starting to see more activities in Asia in recent years, and hope to see more.
In the tutorial I've given in VLDB 2010, one of the first slides was a rhetorical questions - see above.
There are four opinions: some people think these are aliases, some people view stream processing as a subset of event processing that deal with ordered events, some people view event processing as a subset of stream processing, saying that event stream is one type of stream, and there are also other type of data streams such as voice stream, video streams, and there are also people who think that these two are actually totally different concepts, relating to different types of applications. There is something of true in each of them, looking at some interpretations, but IMHO none of the above is really true,
Curt Monash decided to renew the old terminology discussion on his Blog, Taking the "stream" approach which is favored by the database people which look at "data streams" as data in motion, and view events as type of data that does not need any real special handling.
The difference of opinions and terminology stems from the fact that some people are thinking about apples and some about oranges.
What is the apple? - let's take as an example the S4 from Yahoo Labs, in the Blog post I referenced here I mentioned that S4 is a platform for doing "programming in the large" for stream processing, what does it mean? -- it supports a data flow graph, where streams are flowing on the graph's edges, and the processing logic is embedded in the graph's nodes. How is this logic implemented? this is not part of the model, each developer can use the platform and implement the nodes, the platform takes care of the flow, and some non-functional properties (distribution, fault tolerance, cluster management, scalability in some aspects etc..).
It is a pure programming-in-the-large framework. There are others like that, in this case the model is blind to the type of stream, and the stream can indeed be video stream, voice stream etc.. I would call such a framework as "stream processing".
What is the orange? -- if we look at the abstract model of event processing, the way we defined it in the EPIA book, it is a model that is centered around the programming-in-the-small, with language primitives that related to the semantics of events: mainly the notion of context (when? where? to whom?) of events and patterns over multiple event occurrence. The orange does not sound at all like the apple.
Can something be both apple and orange? -- the answer is positive, while event processing can be implemented using various "programming in the large" models, we advocate the "event flow" one, and the "event processing network" can be mapped to the data-flow graph model of streams. So it is possible, but not necessarily to implement event processing as a kind of stream processing. It turns out that there are some benefits to do it, and we see that indeed this seems to become a dominant way for "programming-in-the-large", while the programming in the small is still based on the semantics of events.
The intersection is not an overload, in stream processing one can add a node dealing with audio processing, but the event processing language might be of little value, likewise, there are implementations of event processing that are based on other programming-in-the-large models (such as: logic programming framework) and not on the stream model.
When looking at current state-of-the-art, we see that many of them indeed lie in the intersection of both, thus each of the sides can classify them its own way. The fact that most classify them as event processing may show where the market thinks that the value is.
