On interactions between transaction and events

As a follow up to the transactions vs. events posting,  I would like to discuss some interactions between event processing and events.  ACID  (Atomicity, Consistency, Isolation, Durability) are the properties of classic transaction systems that guarantee serializibility. 


 Some people think that event processing and transactions live on two separate universes,  as event driven architecture advocates asynchronous mode and decoupling.   Note that there are transactional asynchronous systems (e.g. messaging with guaranteed delivery).    


One of my past Master students, Boris Shulman, has investigated the issues of interactions between transactions and events. There was no paper published from his thesis,  he has done the thesis while working for IBM, but then went to work in a start-up and did not have time to complete a paper, and the topic was not on top of my priorities either,  but recently I am thinking about it more. 


In the thesis he pointed out several different motivations to look at the combinations of events and transactions:

1. A monitoring system that monitors a transaction system,  events are emitted from the running transaction systems,  in a way, emitting such events before transaction commit, violates the "isolation" requirement of the ACID.   However,  at some cases there are timing constraints, since the monitored event need to be processed without delay.   This gets complicated when the transaction aborts, and all transactions operations rollback.   In this case the monitored event conceptually never happened. 
2. Event processing is done as part of an actual transaction,   the producer and consumer are part of a transaction, and the event processing is doing some processing that happen to fall into the functionality it support, but it is part of transaction.   Note that this does not view event processing as a stand-alone functionality, but as a functionality that can be embedded in other systems that can be transactional.  In the past when we did the Amit nodes inside IBM Websphere Message Broker, it worked exactly in that mode,  event processing as a specific node within a transactional message flow.
3. The event processing network (or a certain path in it) may have a requirement to behave as a single unit of work.   This is true when an EPA reads/write data, but also the raw events and derived events may issue transactional behavior.     

There might be other cases,  but these cases indicate some cases for interactions between transactions and events.   The transactions that relate to events, may not be  pure ACID transactions, I'll discuss other model of transactions that might be appropriate - later. 

On Parallel processing and Event Processing - Take I

My travels in the universe have taken me to Brussels (somehow, I have never been in Belgium), for meetings regarding an EU project we intend to propose for the next call, I'll Blog about this proposal when it will be more mature. As we are working for a while on parallel processing related to event processing, I am looking at some of the related work. I came across an interesting article in Gregor Hohpe's Blog, stating some of the assumptions behind the parallel programming model of cloud computing. Gregor defines these assumptions as the "new ACID", relating to the old ACID properties of transaction correctness (Atomicity, Consistency, Isolation, Durability). Gregor provides four new ACID properties for the cloud computing: Associative, Commutative, Idempotent, Distributed. Gregor explains the notions:
While the original ACID properties are all about correctness and precision, the new ones are about flexibility and robustness. Being associative means that a chain of operations can be performed in any order, the classical example being (A + B) + C = A + (B + C) for the associative operator ‘+’. Likewise, we remember commutative from junior high math as A + B = B + A, meaning an operator can be applied to a pair of operands in either direction. Finally, a function is idempotent if its repeated application does not change the result, i.e. f(x) = f(f(x)).

Having these properties enable paralleling the operations and collecting the results, the principle behind Google's MapReduce. In order to take advantage of this way of parallelism for event processing functions (or agents), we need to check whether these properties are satisfied. In event processing -- the temporal dimension and the fact that some of the patterns are not associative, some are not commutative, and some are not idempotent, may issue a challenge. This does not say that the parallelism idea is not valid for event processing applications, it is just saying that the parallelism is influenced from the semantics of the event processing network. I'll drill down into this issue in future postings.