CICS event processing improved version

IBM CICS is an example for smart producer of event processing system, it does not do event processing inline, but instruments CICS transactions to emit events, and works in a loosely coupled mode with any event processing engine that can read its emitted events.  CICS TS 4.2  released recently has several improvements in the CICS event producing capabilities.  Among these improvement are:

• Including the event emission to be part of the transaction, by doing the event emission as part of the commit process.  Note that since it is loosely coupled with the event processing itself, this does not becomes atomic unit with the event processing itself, I have recently written about the relationships between transactions and events, and identified this area as one that need to be investigated more. 
• Change management inside the event instrumentation in CICS with appropriate tools
• Inclusion of system events inside the CICS instrumentation (e.g. connection/disconnection to databases, transactions aborts etc..). 

Since the strength of a chain s typically equivalent to the strength of the weakest link,  in many cases the producer is the weakest link, and the amount of work required to emit the right events and the right time is often much larger than the rest of the system.   Smart event producers like CICS making this weakest link much stronger.

On interactions between transaction and events

As a follow up to the transactions vs. events posting,  I would like to discuss some interactions between event processing and events.  ACID  (Atomicity, Consistency, Isolation, Durability) are the properties of classic transaction systems that guarantee serializibility. 


 Some people think that event processing and transactions live on two separate universes,  as event driven architecture advocates asynchronous mode and decoupling.   Note that there are transactional asynchronous systems (e.g. messaging with guaranteed delivery).    


One of my past Master students, Boris Shulman, has investigated the issues of interactions between transactions and events. There was no paper published from his thesis,  he has done the thesis while working for IBM, but then went to work in a start-up and did not have time to complete a paper, and the topic was not on top of my priorities either,  but recently I am thinking about it more. 


In the thesis he pointed out several different motivations to look at the combinations of events and transactions:

1. A monitoring system that monitors a transaction system,  events are emitted from the running transaction systems,  in a way, emitting such events before transaction commit, violates the "isolation" requirement of the ACID.   However,  at some cases there are timing constraints, since the monitored event need to be processed without delay.   This gets complicated when the transaction aborts, and all transactions operations rollback.   In this case the monitored event conceptually never happened. 
2. Event processing is done as part of an actual transaction,   the producer and consumer are part of a transaction, and the event processing is doing some processing that happen to fall into the functionality it support, but it is part of transaction.   Note that this does not view event processing as a stand-alone functionality, but as a functionality that can be embedded in other systems that can be transactional.  In the past when we did the Amit nodes inside IBM Websphere Message Broker, it worked exactly in that mode,  event processing as a specific node within a transactional message flow.
3. The event processing network (or a certain path in it) may have a requirement to behave as a single unit of work.   This is true when an EPA reads/write data, but also the raw events and derived events may issue transactional behavior.     

There might be other cases,  but these cases indicate some cases for interactions between transactions and events.   The transactions that relate to events, may not be  pure ACID transactions, I'll discuss other model of transactions that might be appropriate - later. 

Transactions and events: old world vs. new world

Paul Vincent posted on the TIBCO Blog some thoughts on transactions and events.  His story was mainly about bookings in a sleeping train that should have been in the same cabin, but the system did not have any way to do it in a single transaction,  In DEBS 2011 one of the keynote speakers Johannes Gehrke talked about similar things, of not being able to coordinate travel reservation of two co-workers who want to fly together (but still want to have independent reservations for the rest of the trip).   This is a symptom of the "old world" vs. "new world" phenomenon.    The "old world" in IT is the world of OLTP, which is centered around ACID transactions (maintaining: atomicity, consistency, isolation and durability).   The "new world" is a world of mobile apps, social media, NOSQL, Web 2.0 and more - this world has started from scratch,  the "cool" geeks who created it don't want anything to do with the nerds that work in the old world (slang I got from watching American TV programs, starting from "happy days"),    Anyway -- it seems that some relations do exist between the old and new worlds:  

1. New world applications should use some old world techniques like transactions, but this is not necessarily ACID transactions.
2. New world applications should sometimes talk with old world applications
3. Old world applications should support some of the new world characteristics -- and have to evolve.

All these three relations do exist, I'll write in the future about each of these topics.    


Last but not least -- back to the title of transactions vs. events.   Event processing seem to have some of the characteristics of the new world, however, there are some synergies again between transactions and events. 
One of my earliest posts on this blog, in 2007 had the title: "Event processing and transactions - real, real time, and real time enterprise", where I discussed briefly the idea of event processing within transactional system.   Since then there was some progress in the thinking - I need to revisit this issue on the blog.  
So made some promises on future posts on this Blog - and now back to the new world of  looking at what is new on Twitter, Facebook and LinkedIn...   More -later

On Event Processing Network and Transaction Processing

It is a holiday period, time in which we have four holidays during three weeks, and is quite a lazy time here, with many people taking vacations (like the second part of December - beginning of January in the countries with christian majority), due to holidays and some other events I'll see my office in the coming week only on Tuesday, but working a bit from home now...

In an IBM internal Email exchange this week with a person who does not really understand event processing, this person has seen some illustration of EPN (Event Processing Network) and wondered -- this seems like regular transaction processing ? what is the difference ?

Indeed - from bird's eye view everything looks like directed graph, such as the one shown in the top of this page, both transactional flow and EPN as well as many other things are expressed using a directed graph, however there is a major difference in the semantics of the graph.

In order to refer to a concrete example, let's take an EPN example taken from an application of remote patient monitoring.






The semantics of EPN means that a node in a graph creates events and then these events are consumed by other nodes in the graph, for example the "enrich" node takes a blood pressure reading and enrich it with indication whether the patient is diabetic, thus creates a derived event; this derived event is consumed by the node that is looking for pattern to alert physician. Without going to the application's details too much - we may also state that unlike a control flow, the pattern detection node does not start its execution when all of its predecessors have finished, since the pattern may look at multiple blood pressure measurements of the same patient, it may exist for longer period relative to the enrich node that is created and measurements anytime that there is a blood pressure reading of a new patient, so the graph does not show the control flow, moreover, these two nodes don't know each other and communicate through a router (channel) node. So there are some differences between event processing network and transactional flow:

1. The EPN graph does not represent control flow, but event flow.
2. In a control flow graph, typically the relationship between predecessor and successor nodes are "finish to start" (either "meets" or "after" in the Allen's operators that I'll discuss in a seperate discussion) which means that the predecessor's node must terminate in order for the successor node to start, in EPN, this may not be the case.
3. EPN does not necessarily be atomic (one node in the EPN may fail, but others continue - no "atomic commitment protocol" (e.g. 2PC) is applied
4. It also may not be isolated -- a node can emit events, while still continue working; even if it fails later - its emitted events may still be valid, if atomicity is not required.
5. EPN can be restricted to behave in a transactional way - this is an interesting observation, as transaction support violates the decoupling principle, however there are cases in which it is required (again, deserves some more discussion). More - Later.