The Event driven world - a layered approach

Sensors and social media are becoming the major sources of data, while the traditional organizational data does not grow in the same rate.    Many information system both in the enterprise level and in the individual level are being based on streaming data - these are all old news.   I have recently written about event server as the 21st application server, following TIBCO's  statements.   Progress Software in its recent statement also talked about cloud-based middleware that is geared towards real-time analytics, within its new re-structuring.     

Currently, products and solutions in this area provide a mix of different things.   I envision that we'll settle in three tier architecture, with different products over the food chain that might be more specialized: 

 The event application server layer:

  This will provide services like:  

• sensor and actuator framework, including adapters, data normalization, de-duplication and more.
• routing services -  pub/sub, channel management .
• state services -- access to external stores (databases, files), global state, local state, state machine services 
• meta-data repository services -- representation of events, relationships among events, and other meta-data entities.
• event flow services -- ability to devise event flows ("event processing networks") from the "programming in the large" point of view (not the implementation of specific agents), this will provide API for event producing, event routing and event consuming. 
• Tuning and scalability services -- ability to tune applications by using various ways of parallelism and distribution, fast routing,run-time scheduling and load balancing,
•  tracking and provenance services
• context services 
•  management services
• visualization --- dashboard and visual analytics for feedback services. 
• recoverability services
• high availability services
• security and privacy services

and maybe more...

The application service layer will be based on event-driven  agent-oriented architecture, but will also support request-driven access to various components.  This will support large quantity of lightweight agents, and the flow among them.  While this is quite different from the way most people think about programming, it will become one of the pervasive programming models -- so it is a good investment for the future career of professionals to understand it.

The agent building environment:   

The second layer is the layer that assists in constructing event processing agents.    One way of using the event application server is to use the API  and implement the logic within traditional programming languages, this will require standards like JDBC to access events and event operators in the same way that we currently issue SQL queries inside programming languages.    The other way of doing it is to use agent building tools that generate code for agents,  these might have benefits of resusability, optimized implementation, and reduction in cost.  

This setting will allow hybrid application -- an application may consist of filter agent that is written in Java, using the assertion operator API,  aggregation agent taken from a vendor that specializes in optimized statistical functions, and pattern matching agent taken from another vendor who specializes in some type of patterns (e.g. spatio-temporal patterns).

The application layer:

Applications will be constructed on top of the first and second layer.   It may be all developed using the second layer supplied by a single vendor, multiple vendors, home-made, or any combinations.  

In some cases the applications will be developed by a specific user, in other cases it will be developed by independent software vendors specializing in a certain domain. 

In this food chain we'll be able to see different vendors -- those who specialize in the application server layer  and compete on the quality of services provided there; vendors who specialize in the agent building environment and provide filtering, transformation, aggregation, pattern matching, some of them maybe specialized for specific application type or domain; the third type of vendors will be application providers, which typically require subject matter expertise.

Some of the big vendors will attempt to be active in all three layers or create partnership.