Where does the edBPM term come from?

Writing earlier today about a German friend,  another German friend, Rainer von Ammon has written on the complexevents forum about the source of the term edBPM,  Rainer is also the one who has drawn the nice illustration above showing the reference model, Rainer is the person promoting this term, and organized several workshops around the concept of pairing event processing and BPM technologies.     Rainer wrote that the term came from Gartner and quoted me.   This is almost true,  the original term that Gartner used is "event-based BPM"  and I have slightly modified it to "event-driven BPM" when I asked Rainer to write a value about it in the Database encyclopedia (I have been the editor of the event processing related terms).  
Here is the Gartner's original slide from 2005.  

This is the original slide,  source:  "Event-driven applications make event-driven businesses work better", a presentation by Roy Schulte from Gartner in 2005. 


As you see Gartner classified the event processing functions into: simple event processing, mediated event processing, event-based EPM, and complex event processing.    Roy Schulte has later realized that "event-based BPM" is orthogonal dimension to the three others, and changed the positioning.     So this is the source for all history lovers. 

On Alex Buchmann's 60th birthday book

 Alex Buchmann is an old friend, we first met when both of us were 20 years younger, and worked on active databases.  Alex is a little bit older than me, and recently celebrated his 60th birthday.  I could not travel to the ceremony in Darmstadt, but as a gift, contributed to the book, which includes a collection of papers edited by Alex's students (or ex-students).  Today the mail has brought me a copy of this book, with personal inscription from Alex.  The book is called "From Active Data Management to Event-Based Systems and More".   

More detailed about the book can be obtained in the Springer LNCS site.    The book includes a paper entitled "Spatial perspectives of event processing" co-authored with Nir Zolotorevsky.   Browsing the book I see that I am in a very good company, some of the other authors are: Jean Bacon and Ken Moody, Mani Chandy, Umesh Dayal, Tamer Ozsu,  Gerhard Weikum, and many others. 


When I summarized the year 2009 in this Blog, I have written that the quote of the year is taken from Alex's keynote address in DEBS 2009, stating is using regular database techniques for event-based systems is like trying to drink the water in a waterfall using a straw.    Alex also does not like the term "event processing", claiming that "processing" sounds like "data processing" which is an archaic term, and prefers to talk about event-based systems, as shown in the title of the book.


I wish Alex many more years of  good health, fruitful work and fun.

Revisiting EPN

This illustration, taken from the EPIA book, and drawn by Peter Niblett,  is a portion of the EPN that describes the "Fast Flower Delivery" example that accompanies this book.   In an internal discussion today somebody raised the question, why do we need EPN at all,  and not using the alternative that has been used in Amit, and other places:  each EPA subscribes to an event type, whenever an event from this event type is detected, the appropriate EPA listens to it and processes it,  and all the event flow is implicit and the person defining the system does not need to worry about it.


Since this question is actually a good question,  I wanted to share my response.  There are two main reasons why we have shifted in the thinking to the EPN model:  efficiency and usability.  


  I'll start with the usability, experience shows (and this observation is true also to inference based systems) that people feel more comfortable in ability to control the flow rather then having implicit flows, they understand better what it does, can better debug and validate it, and trust such systems more.   Note that EPN is not a workflow, it does not represent control flow, it represent event streaming flow (in a way similar to data flow, with some semantic distinctions).  


The other reason is efficiency.    If an EPA subscribes to event type then either an EPA has to process and filter out a substantial amount of irrelevant events, or the amount of event types might successfully be increased.   Imagine the following scenario:   An event of type ET1 arrives,  first it meets a filter that filters out much of the event using some assertion, and then there are various EPAs that process only the filtered-in events,  one of this EPAs is enrichment, adding some information from a database,  and then the enriched event is being sent to an aggregator for further processing.      If we use the "event type" subscription, there are two choices:  first -- create event type ET2 for the filtered-in events, identical to ET1, and create derived event of type ET2 for each filtered-in event of type ET1,  then create event type ET3 for the enriched event with added enriched attribute, and then indeed each EPA subscribes to a single event type.  The second choice is to use ET1 for all three cases, but add indication (using some derived attribute) which variation of ET1 it is, and filter inside the aggregator to have only the right type of ET1.  Both are inefficient, the first one due to the need to manage much more event types, the second is that much more events are transmitted to each EPA to filter out, and the order also becomes important here.   


The explicit EPN resolves it by the fact that each EPA sends it output to a channel and the channel can route according to source, type, assertion etc...   -  thus a specific  output terminal of a channel is really the topic which EPA subscribes to.     Note that all the possibilities mentioned before are just special cases of EPN and if one insists, such EPN can be constructed, in the extreme case, one can construct EPN with a single channel that routes every event to every EPA to decide whether it wants to use it or not,  but I would not recommend it as a good design pattern.     More - later.

On Hadoop and event processing

The region which I live in did not have much luck recently, first the big fire on the Carmel ridge, that lasted for three and half days until it got under control, and now a major storm, with winds running in velocity of >  100 KM/H  and a lot of rain.   These two pictures, taken from the Israeli news Internet sites, were taken in Haifa yesterday.   The storm is now over and some nicer days are ahead of us.


Back to professional issues -- Alex Alves (who represents Oracle in the EPTS Steering committee among other things) wrote a nice posting in his blog explaining the Hadoop programming model, if you are still not familiar with it, it provides good explanation. 


Hadoop is batch oriented and provides kind of imperative programming model, but can be wrapped and concealed by higher level language.     I am working now with a graduate student who investigates the usability of the map-reduce model for some of the event processing functions (e.g. aggregation).   I am curious  to see the analysis of this work.   More - later

On ACM Distinguished Speaker Program

Today the ACM Distinguished Speaker program announced my inclusion in the list of "ACM Distinguished Speakers".   The program is described in the ACM DSP site,  while this is a big honor, especially looking at list of speakers that include some real giants,  it is not a recognition program, but a program that has a mission stated as:  The DSP is an outreach program if ACM that brings distinguished speakers from academia, industry and government to give presentations to ACM chapters, members and the greater IT community. 


The outreach mission means that by accepting the nomination to ACM Distinguished Speaker, I commit to 
travel and give talks by request of local ACM chapters worldwide, there are some ground rules that can be found on the site,  e.g. to justify international travel (fully funded by ACM) there should be an accumulated audience of 300 people, so it typically entails multiple talks during a single trip.     I am in the opinion that I should spend some of my time in sharing knowledge with the greater community, this is the reason I am teaching, providing long tutorials in various conferences, and wrote (together with Peter Niblett) the book "Event Processing in Action".    Thus, accepting this nomination is another link in the chain, and I'll try to do my best to satisfy requests, especially from places in the world which don't get a lot of talks on the event processing area.


My speaker page shows four proposed talks, three of them deal in event processing:

•  A short tutorial that serves as introduction to event processing -- summarizing the material in the EPIA book.
• A talk about the research challenges that exist and a "call for action" to the research community in a way to move the event processing area towards its next generations
• A talk about proactive computing, one of the extension directions of event processing, on which I concentrate recently.

The fourth talk is on more general theme:   Computer Science Research in Industry -- some history and different models of how it operates.


I hope that it will be both useful to the audience and fun. 

On the 4Ds -- past version and the proactive version

The climate in Israel this year is quite strange, it is December, and today I still saw people going in the street with short dress, the summer just did not go away.  However, the forecast for the next three days, starting tonight is of a major winter storm (which here means a lot of rain, not snow) and much colder weather, so getting the winter clothes ready.   


Today I've read a blog posting by Jeff Adkins,  one of the people with most practical knowledge about event processing, who relatively recently joined IBM  GBS (Global Business Services).  Jeff blogged about the 4Ds-- detect, derive, decide, do.    These four are part of smart systems that sense and respond, what is known as reactive system.  I knew that this looked familiar, so went back to my archive and found that seven years ago we  were engaged with a project called "active integration"  (the actual application was in the insurance area, but we have generalized the concept), roughly what was known by Gartner as "real-time enterprise".    Here is the original flow from that project:

  
I don't think it has been original invention, it was a variation of concepts from control theory, but it looks very similar to the 4Ds, with a more detailed granularity.


The first D: Detect spread into two phases on our model: sense and detect, where the sense dealt with instrumentation and sensing of raw events, and detect with a detection of the meaningful situations by pattern detection.


The second D: Derive is the same:  created a derived event (and sometimes also derived data) as a result of this detection


The third D:  Decide is partitioned to three phases:  Analyze - determine the possible alternatives and recommend,  Collaborate - in case of "human in the loop" within the decision, and Decide -- apply some decision procedure to select among the alternatives (simulation, analytic methods, predetermined rules).


The fourth D: Do we called "effect", since it had to effect a running system.  


The loop indicates that the after effecting the system, it feedbacks through its instrumentation mechanism and the sense phase is looking again for things that needs reaction.


I agree that the 4D is much more catchy then our more detailed drawing.


And one comment:  when we did this work, we worked on REACTIVE system - something has occurred, we detect it, and then do something to repair.   This drawing is also a good description of our current project that deal with PROACTIVE systems, but the semantics is somewhat different:  the detection is of predicted undesired states instead of situations that require reaction since something already happened.   

Some Blog statistics - December 2010

This year I have not posted the annual statistics about this Blog readership, so taking advantage of the vacation to do it, along with going to movies, musical on stage, and bowling with my daughters.   The Blog now showing on the bottom some of the most popular postings, however, it started recording only several months ago, thus the more accurate statistics are accumulated in Google  Analytics, where this blog is tracked from September 2007.   Starting with the quantities:  There are around 1800 regular readers that are reading each posting in this blog, and additional 3600 who get into the blog from time to time (once every two-three weeks),  There are also  people who entered the blog less frequently, some of them only one time, the total number of this blog visitors is around 65000 people.  The geographic distribution is also interesting, the map is mostly painted in several green variants, so what stands out are the white space, countries from which there was no reader so far.   Europe has not white spot, America has one white spot: Suriname.   Asia has three white spots:  North Korea, Turkmenistan and Tajikistan.   Africa still has only partial coverage, all countries in the north part and south part of the continent are green, but the middle is mostly white.    This may be an indication that the Internet infrastructure in these countries still needs to go some way, or that the content of my blog does not appeal to people in these countries.

As far as the  ten countries with most views, these are:  1). USA; 2). UK; 3). Germany; 4). Canada; 5). Israel; 6). India; 7). France;  8). Japan;  9). Sweden and 10). Australia.  The readers come from 178 countries.

In a city view ten cities with most views, there are: 1). London; 2). NYC; 3). Paris; 4). Karlsruhe;  5). Haifa; 6). Tokyo; 7). Singapore; 8). Bangalore; 9). Göteborg; 10). Vienna. 

About 10.5% of the page views were direct requests, around 25% results of searches, and the rest, references by various websites. 

The most popular posting, is still ,by far, the one entitled "On unicorn, professor and elephant",  which answers a claim that everything done until today in the event processing area is just a hype and worth nothing. Since the time it was written two years ago, there were many proof points the the EP area has value to customers in various industries, and the assertion that it is still an infant, and some vendors do over-hype it is also still valid. 

The second most popular posting, is the one entitled: "On simple event and simple event processing".   This is an early posting.  In the past we used a terminology of: simple event processing (filtering and routing), mediated event processing (aggregation, transformation, composition)  and complex event processing (pattern matching).  However, I stopped using these terms since it got people more confused, due to the fact that different people have different associations with the terms simple and complex, especially the ambiguousness of  complex event processing,  that is interpreted by some as (complex event) processing and by some as complex (event processing).   I also tend to use composite event instead of complex event when talking about event that is composed of events.

The third most popular posting, is the one entitled: "On Enterprise Service Bus and Event Processing"   which is also an early posting,  this also states that event processing capabilities should be part of enterprise computing infrastructure, where ESB is a natural place to be a center point for it.  Since that time EP capabilities became even more pervasive among various technologies.

Somehow related to this is the most popular among the 2010 postings entitled: "Consolidation and pure play in the EP market".  This deals with the fact that most of the EP vendors today are big software vendors that consolidated EP within their products, while there is still a niche for pure play vendors.

While, as the blog title indicates, most of the blog postings deal with event processing, there are several off-topic postings that won a lot of responses, such as the one on positive thinking, and the one in which I described things that I heard from my father about the holocaust.   My last posting on accountability belongs to this family.

This year I spent less time on blogging, so the quantity of postings is less than either 2008 or 2009,  but I intend to catch up.    

The book "Event Processing in Action" is in someway descendant on this blog,  the publishers read the blog before approaching me to write the book,  but of course, there is more emphasis on rigor and quality within the book, the blog is "quick and dirty".      

End of summary -- next posting will go back to professional stuff