Killing elephants - is MapReduce dying?

My English teacher in the last grade of high school had an interesting taste in literature, and taught us the story on  "Shooting and Elephant" by Orwell.   I was not a very good student in English and forgot about it until reading Colin Clark's Blog posting entitled : "It's time to kill the elephant".   From time to time there are various people claiming that various things are dead or dying.   Some of the readers may still remember the discussion about whether SOA is dead.   Recently the Forbes Blog has announced the death of ERP.  Colin's contribution to the hunt is the observation that MapReduce is dying (or should be dying) and the batch processing should be replace by more real-time processing.  His evidence is that Google is dumping MapReduce and using Colossus for its search technology.   While this fact is certainly true, I think that there are still many types of analytic procedures that are done off-line using batch processes, so while the use of real-time analytics will substantially increase given supporting infrastructure, I am not sure that batch will die soon (the same goes for SOA and ERP)...   Old soldiers never die - they just fade away (s-l-o-w-l-y).

Some footnotes to recent blogs

Today's experience relates to the remote control which opens the doors of my car (well - mine does not have a panic button), for some strange reason I have noticed that it resides on the floor of my office decomposed to basic components, one screw has broken or something... I picked up all parts, held them tightly together and went straight to the contact person of the leasing company in IBM Haifa Labs, he looked at the broken device and said -- these days we are not allowed to replace them if they work, so he took the following working tools (see below) -

and now I have a remote control with cello tape around it, it turns out that the atmosphere of "reducing expenses" in our leasing company gets to the tiny details.... I also heard that some almost forgotten professions like shoemaker who knows how to fix shoes got alive these days...

Anyway today I am in reactive mode -- since several of the recent Blogs deserve some footnotes:

A footnote to the Blog with the provocative title: SOA is dead; long live services
by Anne Thomas Mannes (along with the many other responses)- I once heard a good talk about the many interpretations of SOA, which means different things to different persons, so it may be dead for some, and alive for others... anyway, good ideas live more than marketing TLAs that come and go with fashions. As an analog, The TLA CEP may survive or change with other fashionable term that will have some other blend of technologies, however, the more interesting thing is not the marketing term, but the substance behind it. If it solid and have value to customers, it will survive and prosper, and I believe that event processing (as a discipline - see: my previous posting on EP as a discipline) is one.

A footnote to Mark Palmer's Blog -
Speaking on event processing - it was interesting to read Streambase's report by Mark Palmer stating that Q4 was the best quarter ever for Streambase, which is a pure play event processing vendor. While this may or may not be an indication for a more general phenomenon, as I've started this posting, everybody is trying to reduce costs these days, and one of the ways to do it , automate processes that are event-driven by nature (e.g.automated exception detection and handling), getting alerts on cases where there is a potential expenses leakage (auditing), compliance with regulations that corporates see as "tax" on their operations, and would like to invest as little resources as they can, straight through processing and other activities associated with cost reduction are event-driven in nature, and thus can benefit from use event processing technologies, thus, the positive correlation between troubled times and growth in the use of event processing software may not be surprising, again, this may or may not explain Streambase's report, I am not familiar with the details.

A foontnote to Marc Adler's Blog - Marc cites a study about the influence of Blogs on purchasing. I am amazed every time to see the power of Blogs... from my personal experience, I am getting a lot of private communication based on my Blog, including some surprising offers, will write about it one day; one the RFPs that I got from our sales team to advice on, was traced by them to be copied from one of the area Blogs... customers started to see Blogs as authority, and this can be of course dangerous since not everybody who Blogs about something is really an authority on the area he or she Blogs on and get into the trap --- that's all for now -more alter.

On Enterprise Service Bus and Event Processing

This is one of the variations of Enterprise Service Bus (ESB) illustrations that I have taken from an article by one of my IBM colleagues. The topic of today is -- what are the relations between ESB and Event Processing ?

• An event processing functionality that runs Event Processing Agents in various sources, requires a that will take care of the routing and execution in a distributed environment. There are three alternatives here:

1. No native platform -- an engine that can run in multiple platforms (thus need to be integrated to each platform that it runs in by adapters etc...).
2. Dedicated event processing platform -- the event processing part has a dedicated platform that provides the infrastructure for the event processing functions.
3. Event Processing is built as part of an already existing platform.

All of these variations exist in the market today, and there are pros and cons for each of them, smaller vendors may prefer the first alternative as my friend Marco noted in his Blog.

When getting to the third alternative, if the environment is a SOA environment, then the ESB is a natural place in the SOA middleware to be the principle carrier of event processing functionality:

• It provides messaging infrastructure and routing capabilities
• It provides mediations like - validation, transformation and enrichment that can be reused for event processing (have a large intersection with the "mediated event processing" functions)
• It supports distributed environment.

While the principle usage of ESB in SOA has been to mediate between consumer and producer of services, being a carrier for event processing is now considered as a step in the evolution of ESBs.

This does not say that ESB is the ONLY place in which event processing functionality can run, which brings to a discussion about the Event Processing Conceptual Model- which I'll deal in a subsequent posting.

The ESB gets into the picture in alternative

On Trifecta and Event Processing

Reading the educated blog of my friend Tim Bass
I had to admit that I don't have any clue what a "Trifecta" is, and rushed to wikipedia

for help to find out that in horse races - it is a bet on the first three horses. I must confess to my ignorance - I have never watched a horse race, neither have I gambled on horses. I have made a major technological gamble though, around 10 years ago, to bet around "complex event processing" (without using this name and without knowing on the three others who have worked in parallel on the same topics - will have at some point postings about the history), and after suffering some frustration periods, it seems that the reality also slowly getting there, and I still hope it will be the "next big thing" - but we have some work to do in order to get there -- again, a topic for another posting. Now the question posed both by Tim Bass, and by Joe McKendrick

is - whether CEP is depended on SOA, and in turn - does it require the coupling of SOA and BPM - which is by itself a problematic concept.

My answer is -- CEP is an horse that is indepdent in any other horse, donkey, mule or tiger. As I noted in earlier postings, SOA and EDA are orthogonal - SOA is about modularization or componentization of the enterprise, and hence it IT systems, introducing modules as "services". Now let's talk about the relations between SOA, CEP and BPM.

(1). Services in SOA can be producers and consumers for "Event Processing Agents"; as a producer - the service is instrumented to emit events, as a consumer it can be notified or triggered by an event; Services are one type of consumers and producers but not the only type - every application can be consumer or producer, thus there is a loose coupling.

(2). In SOA environment there is an ESB (Enterprise Service Bus), whose functions have some overlap with event processing (especially mediated event processing) - can be a natural host for the "event processing network" (I'll write more on ESBs in another posting). However, again Event Processing Agents can also run outside ESB.

(3). Workflows (BPM) are special type of consumers and producers. In the SOA world, BPM orchestrate services, and thus, BPM can emit events about the status of the workflow, while as a consumer it can add/modify/delete one or more workflow instances. Some of the Event Processing applications (e.g. some types of BAM applications) are based on business processing - however, even in SOA environment this is only one type of consumer and producer, and there are other types (e.g. RTE applications which process events in-line and not in observation mode).

Bottom line --- Event Processing can have different interactions with SOA, and when IBM's announcements in this area will be available you'll realize that there are different entry points. Event processing can also work in legacy and non-SOA environment.

So today I have learned about horse races and promised some more postings (not promising when) -- more, later.

IBM WebSphere CTO sees CEP as SOA's 'next big thing'

In Israel there are sometimes news items that are forbidden by the military censorship, but if broadcasted by foreign sources - one can quote the foreign source and broadcast it in Israel.

I have not referred in this blog to IBM plans in this area, since they have not announced yet by the proper channel, but I may quote the interview with IBM Websphere CTO, Jerry Cuomo

in which he stated that he sees CEP as SOA's next big thing -- and promised both new technology and additions to existing ones. Stay tuned for more information on IBM and Event Processing.

On CoDA - Context-Driven Architecture

Yefim Natis from Gartner, the same analyst who is bringing us the XTP vision
is also responsible for the brand new vision CoDA = Context-Driven Architecture; which is one or two steps further from the XTP era. In the previous posts about contexts: (1) ; (2)
I have argued about the use of context as a first-class-citizen in event processing; and indeed people act and think within contexts, and the use of explicit context can get the computing closer to the human thinking, and also has computational benefits (like indexed vs. sequential search in the early days of file systems). Both Yefim and myself did not provide formal definition of context when talking about it, and I still have a "to do" to think about it. One comment though about the "Architecture" - some people think that SOA and EDA are contradicting terms, since there can be only one "big A" - meaning architecture. As I have already mentioned - in the posting about EDA and SOA that they represent different aspects of computing, thus they are orthogonal (but can co-exist). The question is - whether context is yet another dimension that can co-exist with the other aspects ? again - still need to think about it. Thus, I'll return to the context concept soon.

More on EDA is EDA and SOA is SOA

Hello from Haifa again. My good friend Tim Bass has written a blog entitle EDA is EDA and SOA is SOA. I totally agree that EDA and SOA are not the same, more than that, I don't even think that they are talking about the same thing. Many people equate SOA with the synchronous "request-reply" style of interaction, but this is only one way to implement SOA, since SOA is not about interaction style, but about modularization or componentization of the enterprise, and hence it IT systems, introducing them as "services", the interaction style is a secondary issue, services can interact both in synchronous and asynchronous ways, and there is nothing inherent in the concept of services that say anything about the interaction type. EDA is indeed about asynchronous push interaction style, and this can be interaction between services, where we get both SOA and EDA, or interaction between other software artifacts, in enterprises that don't implement SOA. Implementation of SOA and EDA both make sense for other reasons and to solve other problems, and they are completely orthogonal. In enterprises that implement SOA there is a sense to use a combination of both, thus, it is important to position them in a way that they are complementary and not contradictory, since there are some misconceptions about it (exactly since some people interpret SOA = request / response). I may dedicate another blog to in-depth discussion about this two concepts. A side point - since this blog deals with "event processing", event processing is not implemented on top of EDA -- there are some cases in which event processing has "request-response" functions such as: getting events in PULL from the producer, which may be done periodically or on-demand, or event processing network that is implemented as part of transaction, and thus does not fulfill the loosely coupled principle of SOA. Thus "event processing" is not a pure EDA.... confused ?
more - later.