Is Business Intelligence dead?

Neil Raden posted recently an article entitled " BI is dead! long live BI".  This adds to the trend of declaring some technology or trend as dead:    Earlier this year I wrote a post in this blog, entitled: "Is mapReduce dying?" in which I also indicated some references to claims such as  "SOA is dead" or "ERP is dead". Actually nothing in the IT industry dies quite easily,   I guess that COBOL, which is about my age, is still alive  today, and may outlive me.   What Neil claims is that the success of BI was much less than the hype created around it, and its adoption was around 10-20% in large organization depends on the survey.   This is consistent with other opinions,  James Standen reported  on a survey that found that the the actual adoption is lot less than the percentage claimed  by BI vendors are. The picture at the top of this page is borrowed from his post "Business Intelligence adoption low and falling".     

Neil claims that BI does have a future, but its future is in doing it as part of a decision management framework, not as a stand-along service consumed by people.   When talking about decision management, it has a role to analyze historical data, but doing it within a context of a specific decision.  This can be used to provide input to decision or to reinforce or refute assumptions.  

I think that this is an interesting perspective and in a way has some similarity to event processing which also transitions from stand alone technology to part of a bigger game.   I have discussed it in the past, and still stand by the conclusion that stand alone event processing will stay as a niche, but the mainstream will be part of bigger games.  I have investigated recently the current list of "bigger games"and will write on it in one of the future posts,  however, I see the same observation as valid for BI,  current BI will remain a niche, while the mainstream will move towards embedded BI.   More -later. 

On the Rabbi and the limited appeal

An old Jewish folk story is about husband and wife who came to a Jewish Rabbi for marriage counseling, the Rabbi listened to the husband and said: you are right,  then listened to the wife and said again: you are right. After they left his assistant asked him: "Rabbi - why did you say to both of them that they were right, it is not possible that both of them are right", and the Rabbi answered again - "you are also right".    I recalled this story after reading Mark Palmer's Blog entitled:  "how broad is the appeal of CEP".   The story starts with the CEO of SAS  claiming that CEP has limited appeal continuing with the rebuttal of Progress and TIBCO, saying that the SAS guy underestimates the potential and there are many applications in multiple industries.   Mark Palmer has a counter-opinion saying that relative to BI, EP is still small, but has a potential to grow,  and it is hot within a single industry - capital markets, with relatively modest interest in other industries - Chad Gadya!     


So why does it remind me the Rabbi's story?  -- since each of them is right in something.
EP is indeed a smaller market relative to BI, and much younger and less mature.  Some of the BI people regard EP as a "non issue" since one can do anything with BI tools, or database queries, and the only reason to use EP is when there are latency and throughput requirements that cannot be satisfied.  However, as noted many times, the bigger appeal of EP is the abstractions that enable to develop and maintain applications of that type more easily and substantially reduce the TCO.     The observation that EP market is  much smaller than the BI market today is certainly true,  maybe the fact that the SAS guy bothered to react on it is a sign that they start feel some presence in their territory.    BTW -- I don't view BI and EP as competitive technologies, each of them are destined to do different things, some applications need both.


Next --  inside the  EP house,  it seems that there is some agreement, about the potentially bright future, and some disagreement about the present and  even more about the scope.    I guess that appeal is in the eyes of the beholder,  but let me make one observation:  when Streambase is looking at the universe it looks at the market for a stand-alone event processing product; when some of the bigger companies look at the universe, they look at event processing capabilities as part of a larger enterprise computing infrastructure.   When we did, a few years ago, the business evaluation work for IBM, towards the decision to enter this market, we have identified both of them as opportunities, the stand-alone one has its own existence, but is more limited in scope,  for the "embedded" one -- possibilities are much higher, but more work is needed to make event processing capabilities as part of facets of enterprise computing infrastructure -- decision support platforms, business process management platforms, messaging oriented middleware,  sensors and actuators frameworks and  even business analytics framework.   These two view points of the universe provide different perspectives to the beholders, and so each of them is right from his own perspective.      More - later.  

On event processing as a technology and as a business

Philip Howard, an analyst who covers event processing for several years now, has posted a blog entitled: what's happening with event processing, observing that event processing is getting integrated with other areas such as: BPM, data integration and more. This is not a new phenomenon; in the EPIA book, we mention that among the event processing trends is moving from standalone to integrated even embedded, and this trend is evident with the evolution of event processing as a start-up universe, to having bigger software vendors as dominant forces. However - will event processing as technology going to disappear? I don't think so. There is common functionality among event processing utilization in various industries, applications, and hosting technologies, in all of them there are functions of filtering, event transformation, aggregation, pattern detection, and routing. It is not cost-effective to re-invent the wheel for each individual use (although there are variations). This is a similar situation to databases; database can be used for various reasons, and also be embedded with various other technologies and products (e.g. application servers, BPM, system management products, messaging systems - all use databases), while there are also variations, it is not the case that each of these areas develop database technology in an ad-hoc fashion. Thus, I see event processing continuing to evolve as a technology, and having both research and development activities that build generic event processing tools. From the business point of view, there will always be some niche for event processing stand-alone applications, but as Philip writes, most of the market will indeed be in the integrated area, this fact already reflects on the event processing technology in terms of need for standards, interoperability features, and ability to have embeddable collection of building blocks and components. More about this - later.

Event processing - stand-alone, part of a bigger game, or both?

Following my previous posting, somebody told me that I was vague about whether I think that event processing as stand-alone technology is good or bad. Well -- when I was a student I took "image processing" course, and we worked on a graphical screens with 256 gray levels, since then I realized that even in "black and white" picture there are a lot of gray area Thus, I cannot classify it as "good" or "bad". But I'll provide some observations:

1. The event processing area started as "stand alone" engines, this has been obvious, since it started by start-ups and not by big companies as part of other frameworks
2. There is a gradual shift in the market from stand alone event processing solutions to event processing capabilities embedded in larger frameworks, when bigger companies got into the picture, and this trend has intensified.
3. "Stand alone" products may have to implement functions that "embedded" products can use existing functions in the original framework, such as: routing, transformation, filtering...
4. Unlike some software components that may need tight integration, event processing work in loose coupling relative to other components -- sending and receiving events --- thus this supports the possibility for having stand-alone EP.
5. However, there are no interoperability standards which requires to provide adapters for each producer and consumer, which makes stand-alone EP more difficult, relative to a single framework -- the level of difficulty is a functions of the quantity and diversity of producers and consumers. Enterprise integration framework may already include variety of adapters that the embedded EP can get "for free".
6. Event processing is in many cases part of a bigger application, and in this case, there is a benefit of having a single programming model for the bigger application, and not using different programming models/languages/user interfaces for the various part of the system, this also goes against stand-alone EP; In cases where the system is pure EP, this consideration may not be valid.
7. Stand-alone EP may support heterogeneous components -- e.g. work with DBMS from one vendor, messaging system from another vendor, and connect to BPM system from third vendor, while embedded EP is typically homogeneous, since it all comes from one vendor. This may be true, though today there are a lot of cross-adapters among various components that enable framework to support other components (say DBMS) from other vendors, especially where there are standard interfaces.

Is there a bottom line here? --- I guess that the gray area is that there is some segment of the market in which stand-alone EP can live, but I also think that the trend of moving from stand-alone to embedded will continue to exist.