Some other input related to student's work, is the posting of Sascha Retter, based on his diploma thesis (the term discloses the origin in Germany). The observation is about critical success factors of event processing systems - integration with workflow management system, and modeling tools. There is some truth in both observations, but I would like to take broader view.
There is a market for stand-alone event processing systems, but we have observed that the larger market is event processing embedded in other systems, workflow management systems, or in the more modern name "Business Process Management" (BPM) is one of these systems, but this is not the only one, there are both other systems in which event processing can be embedded inside, e.g. analytic frameworks, sensor networks and more. It may also be embedded within packaged applications - like trading platforms, supply chain management and much more.
Modeling tool are a vehicle for usability, and indeed usability and ease of use has been identified as a critical success factor. This is now a well recognized fact.
There are other critical success factors for various applications, like non-functional requirements (performance). I also believe that standards are critical success factor for the entire area.
The EPTS use case survey provides more insights into properties and preferences of event processing customers.
Interfaces is one of the main next frontiers for event processing systems. In the picture you can see an illustration of brain-computer interface which is the most advanced interface that is mentioned in a recent human-computer interface survey, this survey starts from command line, moves through mouse and keyboard and gets to gesture detection, and up to the mostly futuristic brain interface. My daughter Daphna got for her recent birthday XBOX machine with Kinect, which is capable of voice and gesture recognition and provide advanced game experience.
Getting to event processing, the human computer interface is crucial in order to raise the abstraction level and enable to extend the type of developers and users, currently the development interfaces are geared mostly towards programmers, and are in essence close in nature to the implementation abstractions, a separation between the implementation abstractions and the development abstractions is one of the major challenges, in general in software, but an important one in event processing, since realizing the event processing potential entails the ability to make application development accessible to other communities such as the spreadsheet programming community (e.g. business analysts). I have written in the past about this requirement, and will probably deal with it more in 2011.
Talking about interfaces, there is another type of interfaces the application programming interfaces (API), this is important since one of the major utilization of event processing is as embedded capabilities inside applications/middleware/solutions. Thus, APIs that are appropriate for multiple uses are gaining importance for the interoperability and for making EP easily embeddable. API design has become an emerging area, a blog posting by Shanley Kane from Apigee (and API management provider) provides trends and predictions for APIs. It is interesting to note that the first prediction is entitled "APIs go real-time big-time" (in the source all words are capitalized, but I have adopted Manning's writing style while writing the EPIA book which is against over-capitalizing). This prediction talks about the popularity for APIs of push oriented pub/sub systems. This is elementary event processing, APIs for more sophisticated event processing is the next step.
Standard APIs for EP will enable plugging EP components that are doing aggregation, translation and pattern matching components as part of an "event-driven Web" (AKA event-processing fabric) that was declared as the grand challenge in the EP Dagstuhl seminar. I'll write more about this grand challenge soon.
In any event, both human computer interfaces and application programming interfaces are major part of the work needed towards the next generation of event processing systems.
An old Jewish folk story is about husband and wife who came to a Jewish Rabbi for marriage counseling, the Rabbi listened to the husband and said: you are right, then listened to the wife and said again: you are right. After they left his assistant asked him: "Rabbi - why did you say to both of them that they were right, it is not possible that both of them are right", and the Rabbi answered again - "you are also right". I recalled this story after reading Mark Palmer's Blog entitled: "how broad is the appeal of CEP". The story starts with the CEO of SAS claiming that CEP has limited appeal continuing with the rebuttal of Progress and TIBCO, saying that the SAS guy underestimates the potential and there are many applications in multiple industries. Mark Palmer has a counter-opinion saying that relative to BI, EP is still small, but has a potential to grow, and it is hot within a single industry - capital markets, with relatively modest interest in other industries - Chad Gadya!
So why does it remind me the Rabbi's story? -- since each of them is right in something.
EP is indeed a smaller market relative to BI, and much younger and less mature. Some of the BI people regard EP as a "non issue" since one can do anything with BI tools, or database queries, and the only reason to use EP is when there are latency and throughput requirements that cannot be satisfied. However, as noted many times, the bigger appeal of EP is the abstractions that enable to develop and maintain applications of that type more easily and substantially reduce the TCO. The observation that EP market is much smaller than the BI market today is certainly true, maybe the fact that the SAS guy bothered to react on it is a sign that they start feel some presence in their territory. BTW -- I don't view BI and EP as competitive technologies, each of them are destined to do different things, some applications need both.
Next -- inside the EP house, it seems that there is some agreement, about the potentially bright future, and some disagreement about the present and even more about the scope. I guess that appeal is in the eyes of the beholder, but let me make one observation: when Streambase is looking at the universe it looks at the market for a stand-alone event processing product; when some of the bigger companies look at the universe, they look at event processing capabilities as part of a larger enterprise computing infrastructure. When we did, a few years ago, the business evaluation work for IBM, towards the decision to enter this market, we have identified both of them as opportunities, the stand-alone one has its own existence, but is more limited in scope, for the "embedded" one -- possibilities are much higher, but more work is needed to make event processing capabilities as part of facets of enterprise computing infrastructure -- decision support platforms, business process management platforms, messaging oriented middleware, sensors and actuators frameworks and even business analytics framework. These two view points of the universe provide different perspectives to the beholders, and so each of them is right from his own perspective. More - later.
Philip Howard, an analyst who covers event processing for several years now, has posted a blog entitled: what's happening with event processing, observing that event processing is getting integrated with other areas such as: BPM, data integration and more. This is not a new phenomenon; in the EPIA book, we mention that among the event processing trends is moving from standalone to integrated even embedded, and this trend is evident with the evolution of event processing as a start-up universe, to having bigger software vendors as dominant forces. However - will event processing as technology going to disappear? I don't think so. There is common functionality among event processing utilization in various industries, applications, and hosting technologies, in all of them there are functions of filtering, event transformation, aggregation, pattern detection, and routing. It is not cost-effective to re-invent the wheel for each individual use (although there are variations). This is a similar situation to databases; database can be used for various reasons, and also be embedded with various other technologies and products (e.g. application servers, BPM, system management products, messaging systems - all use databases), while there are also variations, it is not the case that each of these areas develop database technology in an ad-hoc fashion. Thus, I see event processing continuing to evolve as a technology, and having both research and development activities that build generic event processing tools. From the business point of view, there will always be some niche for event processing stand-alone applications, but as Philip writes, most of the market will indeed be in the integrated area, this fact already reflects on the event processing technology in terms of need for standards, interoperability features, and ability to have embeddable collection of building blocks and components. More about this - later.
Following my previous posting, somebody told me that I was vague about whether I think that event processing as stand-alone technology is good or bad. Well -- when I was a student I took "image processing" course, and we worked on a graphical screens with 256 gray levels, since then I realized that even in "black and white" picture there are a lot of gray area Thus, I cannot classify it as "good" or "bad". But I'll provide some observations:
- The event processing area started as "stand alone" engines, this has been obvious, since it started by start-ups and not by big companies as part of other frameworks
- There is a gradual shift in the market from stand alone event processing solutions to event processing capabilities embedded in larger frameworks, when bigger companies got into the picture, and this trend has intensified.
- "Stand alone" products may have to implement functions that "embedded" products can use existing functions in the original framework, such as: routing, transformation, filtering...
- Unlike some software components that may need tight integration, event processing work in loose coupling relative to other components -- sending and receiving events --- thus this supports the possibility for having stand-alone EP.
- However, there are no interoperability standards which requires to provide adapters for each producer and consumer, which makes stand-alone EP more difficult, relative to a single framework -- the level of difficulty is a functions of the quantity and diversity of producers and consumers. Enterprise integration framework may already include variety of adapters that the embedded EP can get "for free".
- Event processing is in many cases part of a bigger application, and in this case, there is a benefit of having a single programming model for the bigger application, and not using different programming models/languages/user interfaces for the various part of the system, this also goes against stand-alone EP; In cases where the system is pure EP, this consideration may not be valid.
- Stand-alone EP may support heterogeneous components -- e.g. work with DBMS from one vendor, messaging system from another vendor, and connect to BPM system from third vendor, while embedded EP is typically homogeneous, since it all comes from one vendor. This may be true, though today there are a lot of cross-adapters among various components that enable framework to support other components (say DBMS) from other vendors, especially where there are standard interfaces.
Is there a bottom line here? --- I guess that the gray area is that there is some segment of the market in which stand-alone EP can live, but I also think that the trend of moving from stand-alone to embedded will continue to exist.
