This is a blog describing some thoughts about issues related to event processing and thoughts related to my current role. It is written by Opher Etzion and reflects the author's own opinions
Saturday, October 22, 2011
On use of event processing in routing
Eric Roch, chief technologist of Perficient posted a question about the use of event processing for message routing rather than ESB, the router pattern is taken from the enterprise integration patterns by Gregor Hohpe and Bobby Woolf that we cite several times in the EPIA book. Roch mentions several reasons to use event processing, some of it relates to non-functional characteristics such as low latency, and in-memory management.
I think that there is some continuum that relates to the routing requirements - some of the routing decisions are quite simple, they are based on a predicate, typically on a value of a single attribute; some are more complex, and the routing is determined according to a pattern that involves multiple messages, in other cases the routing decision might involved a structure like decision-tree or decision-table, yet in other cases the routing is determined according to probabilistic reasons.
The better solution is not to have ESB and EP systems in isolation, and chose in each case what should be used, instead it is better to use a single programming model that includes all types of routing, and autonomic translation to implementation in various building blocks, some of them are taken from event processing and some from ESB or traditional messaging systems, this is the approach I prefer for event processing in general, not stand-alone, but synergistic with other technologies. In 2008 I had a posting about event processing and ESB, arguing that ESB is a natural carrier of event processing within a SOA architecture, and indeed I think that we'll see more and more ESB products that are enriched by event processing capabilities.
Thursday, October 20, 2011
Pura vida
Pura vida - means "pure life", is an expression that is being used in Costa Rica as: good morning, hello, response to the question "what's up?", general statement that life is good, thank you - and some others. It is the motto behind life in Costa Rica. We spent in Costa Rica 10 days (1 less than we planned), and here are some impressions:
- Costa Rica is an excellent place for a trip in the nature - it has nice rain forests, rivers and volcanoes, as well beaches of the two oceans.
- Animals -- you see them everywhere in nature. There are various tours in foot, car and boat to see animals.
- The Parador hotel we stayed in the last 3 days of the trip is the best hotel I've ever been to (and I traveled all over the world). The picture above is the view from the hotel room, it was elected several times as best hotel in central america (including Mexico), and best spa (I got there the best massage since my visit in Thailand in 2007), t it is located within a forest on a mountain overlooking the pacific ocean in Manual Antonio, it has various animals on site - a hawler monkey provided wake-up call in one of the days in 5am, with extremely loud voice, on the tree in front of the hotel room; a raccoon entered the daughters' room, when they did not close the balcony's door, and scared them a bit, other raccoons visited us at the dinner table, and were scared away by the hotel staff. An iguana caught some sun in the middle of the mini-golf course, and frogs have settled at night in the swimming pool.
- The local people in Costa Rica eat the same food - breakfast, lunch, dinner -- rice and beans and tropical fruits (in each they might have some supplements). At the end we got tired from this food and escaped to the sushi menu at the Parador.
- Rain -- Costa Rica has a lot of rain, especially in this season. It has a four day heavy rain storm at the pacific side, we had there in the last two of these in Monteverde, but we did most activities in the rain; luckily when we got to the beach the rain storm was over.
- The road infrastructure cannot cope with the quantity of rain, thus many of the road have lot of holes, some of the main roads are not paved, and they call travelling in them as the "Costa Rican Massage".
- Tourism has become the biggest industry in Costa Rica, over the last 25 years, which directed the education system towards this goal. Most people can speak English.
- The amount of Israelies we met was incredible given the distance and cost to get there, it seems that it is now quite fashionable among Israelies who already explored Europe and USA to visit Costa Rica.
- Iberia is the worst airline company I have ever used --- in all aspects (including food), the worst thing was that they did not let us get on the connection flight from Madrid to San Jose, Costa Rica, though we got to the gate 15 minutes before the scheduled flights, due to the fact that our flight from Israel was late 1 hour, and they decided they'll not have enough time to load the luggage, though we were 20 passengers coming on that flight, and the fact that they have 1 flight per day to Costa Rica, they did not agree to make any effort to get us on the flight. They told us that since ELAL was responsible for the late flight - it is not their problem (we purchased the ticket from Iberia, and used Iberia ticket with code share). We had to spend 24 hours in Madrid before getting on the next flight -- and lost the first segment of the trip (which is non-refundable as anything else in Costa-Rica). I'll check if it is possible to sue Iberia for damages. The attitude of their agents in Madrid was also very unfriendly - a big difference between the nice people of Costa Rica everywhere, and those of Spain).
- And on positive aspect --- Pura vida --- is a motto of positive thinking!
Now it is a lazy holiday -- back to work early next week, and already a business trip to Europe!
Wednesday, October 5, 2011
Touring the universe again; current destination: Costa Rica
In a few hours we are planned to start the long journey to Costa Rica, the chosen destination of our next Family vacation, during the holiday period (here in Israel). Will be out of touch for the next couple of weeks.
Monday, October 3, 2011
More on Internet of Things and event processing for personal systems
This illustration is taken from a recent Blog posting by Phil Windley, the CTO of Kynetx, whom I met in DEBS'11. Phil writes about IoT as a source of personal event processing, and about "personal event bus", in which any consumer can subscribe to events and process them individually.
I have provided some background about IoT recently and about its potential to be game changer; the personal systems is an interesting aspect. Event processing models and systems of the first generation have been developed in mind of business oriented systems. However, I have stated before that I see the big potential of event processing outside the traditional corporate IT, and provided robotics and biological computing as some areas of interest. Consumer applications is definitely an area with a lot of potential, however the challenges to do it is beyond collecting and subscribing to events, it is the ability to compose event processing application by consumers, which requires progress in the abstraction level of composing such applications. This is definitely and avenue to pursue.
Saturday, October 1, 2011
On context and punctuation
This illustration is taken from the EPIA book and shows the notion of context as a first class construct that groups together events that need to be processed together, where each of its instances is processed as a single unit, context may have temporal aspects, segmentation aspects, spatial aspects, and state-oriented aspects, and the actual context instance may be combined of one or more aspects.
Recently I had some discussion with somebody from the data stream community who was in the opinion that the stream oriented way to do it is more simple since they are based on standard thinking (i.e. SQL).
In the stream research literature, the temporal context is divided between windows and punctuation, all the other aspects are somehow expressed by the group-by construct. I guess that windows and group-by constructs are familiar to many people, punctuation is less familiar, thus I'll explain the idea briefly.
A good source to learn about punctuation is Peter Tucker's site, from where the illustration below is copied:
Peter Tucker has done his PhD (under the supervision of Dave Maier) on data stream punctuation; the idea of punctuation is to enable creating sub-streams, since a stream by definition is infinite. The notion of windows in classical data stream models consists of variations of sliding windows, either by event count, or time, however, in reality there is a need to explicitly end stream, i.e. when the bid ends. The definition of punctuation in the database encyclopedia (the value was written by Maier and Tucker) says:
In other words, this is a "dummy event" put into the stream to denote the end of a sub-stream.
Now the question is whether using windows + punctuation + group-by is indeed simple. The relational model's claim to fame is its simplicity, however, from early days of the relational model, it has been realized that there are semantic anomalies, and all the normalization rules came to resolve those anomalies, yet, the overall model has been simple. However, this is not really true for the extension of the relational model for streams, while the relational model has a single entity: relation, the extension has multiple entities: relation, stream, window. Punctuation is a kind of trying to add semantics as a kind of logical patch.
IMHO, the context model is semantically cleaner. It has a single semantic notion for all the constructs that determine how to group event together (in the stream terminology, create sub-streams). In the context model there are no dummy event; the notion of temporal context (window) can also support event intervals - time windows that start by event and end by event; the events that start and end the window are real events, and can be used for other purposes. Moreover, looking at the definition of punctuation above, it defines the end of a sub-stream, but assumes that the next bid starts when a new bid event arrives. However, if we look at a situation where bids have not only "end bid" events, but also "start bid" events, then we would like to ignore bids that arrive when no bid is open, and this has no natural representation, and has to be implemented with tricks, so we also need the equivalent of "start sub-stream punctuation". So, we view this as kind of context based on , and not as kind of event within the stream with a special semantics.
The model we presented in EPIA (which we keep evolve) to be a semantic model that provides abstractions above the current implementations of both event and stream processing systems; several product owners have already told us that they use concepts from the book as inspiration to next versions of their products, and I guess that full implementation that is based on this model as native is yet to come.
Friday, September 30, 2011
On the four Vs of big data
In my briefing to the EU guys about the "data challenge", I have talked about IBM's view on "big data", recently Arvind Krishna, the IBM General Manager of the Information Management division, talked in the Almaden centennial colloquium about the 4Vs of big data. The first 3 Vs have been discussed before:
- Volume
- Velocity
- Variety
The regular slides are talking about volume (for data in rest) and velocity (for data in motion), but I think that we need velocity to process sometimes also data in rest (e.g. Watson), and we need sometimes also to process high volume of moving data; the variety stands for poly-structured data (structured, semi-structured, unstructured).
The veracity --- deals with uncertain/imprecise data. In the past there was an assumption that this is not an issue, since it would be possible to cleanse the data before using it, however, this is not always the case. In some cases, due to the need of velocity in moving data, it is not possible to get rid of the uncertainty, and there is a need to process data with uncertainty. This is of course true when talking about events, uncertainty in event processing is a major issue still need to be conquered. Indeed among the four Vs, the veracity is the one which is least investigated so far. This is one of the areas we investigate, and I'll write more about it in later posts.
Wednesday, September 28, 2011
CICS event processing improved version
IBM CICS is an example for smart producer of event processing system, it does not do event processing inline, but instruments CICS transactions to emit events, and works in a loosely coupled mode with any event processing engine that can read its emitted events. CICS TS 4.2 released recently has several improvements in the CICS event producing capabilities. Among these improvement are:
- Including the event emission to be part of the transaction, by doing the event emission as part of the commit process. Note that since it is loosely coupled with the event processing itself, this does not becomes atomic unit with the event processing itself, I have recently written about the relationships between transactions and events, and identified this area as one that need to be investigated more.
- Change management inside the event instrumentation in CICS with appropriate tools
- Inclusion of system events inside the CICS instrumentation (e.g. connection/disconnection to databases, transactions aborts etc..).
Since the strength of a chain s typically equivalent to the strength of the weakest link, in many cases the producer is the weakest link, and the amount of work required to emit the right events and the right time is often much larger than the rest of the system. Smart event producers like CICS making this weakest link much stronger.
Subscribe to:
Posts (Atom)









