A personal invitation to contribute to DEBS 2011

Today I have sent the following note on various mailing lists, so you may have already got it, but if not you are welcome to read my personal invitation to contribute to DEBS 2011:



ACM DEBS (Distributed Event-Based Systems) is the premier event of those that are active or interested in  topics such as; event processing, stream processing, middleware infrastructures, models, applications and best
practices.  The conference, with many new tracks,  will take place in Yorktown Heights (near New York City) on July 11-15, 2011.


You are personally invited to contribute to any of the various tracks and participate in this exciting event.


ACM DEBS is in cooperation with EPTS, and there will be an EPTS session
within this conference.


There are plenty of opportunities to contribute for different types of
interest and participants: 

•  The research track is the way to present quality state-of-the-art    research in all related areas. If you are a researcher in any related    area,  this is a primary citation source for researchers in these area,    thus you should consider it as a target for your publication. 
•  The experience report track:  If you are implementer or user of    event-based system, has an interesting experience report to share about    product, solution or application, and don't have time to write papers,   the new experience report track is aimed for you. The submission   required is just a one page abstract.
•  The industry paper track:   if you engage in industrial oriented   research and implementation and have the time and energy to write a full   paper describing your work or experience, the industrial paper track is your venue
• Tutorials:  If you have done work of survey or want to share in-depth  knowledge with a broad audience,  the tutorial program is your opportunity to offer either  is your  opportunity to share short (1.5 hours) or long (3 hours) tutorials. The anticipated audience will consist of  additional interested people from industry as well as the traditional DEBS audience.
•  Posters and demos:  If you have any work in progress you wish to show and get feedback on, or a cool demo,  this is an opportunity to show case, discuss and get feedback from all participants.
•  The DEBS challenge:  In January 31 the DEBS challenge will be published,   this will be a specification of an application that implementers of   event processing languages, systems and platforms will be invited to show the way it is solved within their solution.   If you implemented  open source product or research prototype this will be your opportunity  to participate in the  competitive track open to non-commercial users,   where the competition will be based on ease of development, and win an  award.   If you are a commercial vendor or want just to show your   solution without the competition part, this is your opportunity to show your approach and view other approaches side by side,  This track is    expected to have much interest within the user community. 
• The DEBS Gong Show:  The gong show will consist of very short presentations about  visionary and outrageous ideas towards the next generation of event-based systems.   Selection will be done by    submitting a short abstract.   The audience will vote for the best idea.   If you have far-reaching idea about either technology or novel use this is your opportunity to share, get feedback, and even win an award.
• The DEBS PhD workshop:  If you are a PhD student in a relevant area,  either in early stages or late stages, this is your opportunity to share ideas with peers and get feedback in a relaxed atmosphere with senior people from the research community that you don't have access to on a regular basis,  if you supervise PhD students you are kindly asked to encourage their participation.

As you can see --  plenty of ways to get visibility to your work, you are encouraged to submit your contribution to one or more of these tracks. Besides the interesting tracks there will be four  distinguished keynote speakers  from the user's community, infrastructure vendor's community, and
academic community. More information  about the various tracks and keynote speakers can be
found in the conference's website:  


Have a happy new year!

Some blog postings for year end

We don't really celebrate Christmas here, but it is the end of the year, and we need to finish our vacation days, and since our colleagues from overseas are now in vacation,  it is a good time to do it, so it is a good time to do catch up on things that are not part of the regular work.   Looking at some recent Blogs,  I borrowed the above picture from Scot Fingerhut's Blog entitled:  " How Santa Uses CEP - Elf Productivity, Real-time Naughty And Nice Rating" .    BTW - I found another Christmas related postings in OMG facts claiming that Jesus was not really borne in December,  

Back to the professional Blogs, my IBM Haifa Research Lab colleague Yishai Feldman started a Blog entitled "The Dusty Deck"  with punched card on the top -  I belong to the generation who really punched cards, I guess that today punched card belongs to museums along with typewriter and sliding rule. 

Yishai is a local software engineering guru,  who deals with analysis of programs, so I'll follow his Blog with interest.

More in the Blogland,;

The Prova Blog reports on a new version of the Prova open source event processing system that was released this week. 

Colin Clark brings us predictions about 2011.   About event processing he predicts that it will become part of a larger horizontal offering.  I think it is already happening,  as I stated before there is still a niche for the "stand-alone" event processing,  the major market is in having event processing capabilities pervasive over computing offerings.

Last but not least, Rainer von Ammon summarized the workshop he organized recently in conjunction with ServiceWare 2010.   

S4 from Yahoo Labs

Yahoo Labs recently announced an open source called  S4:  Simple Scalable Streaming System, available as open source.  A paper describing S4 is available, as well as a site from which one can download the software (alpha version). 
This is a distributed platform whose model is a kind of event processing network; the stream dataflow model where each node is  "processing element", similar to IBM Infosphere Streams;  The paper states that it was inspired by the MapReduce model, but I still need to read the details.  It is interesting to follow the branch of distributed event processing platforms.  It seems that this is a "programming in the large" model, where each processing element is written in Java (at least according to the paper's examples) it does not have its native language. 

Who is the developer of event processing applications?

One of the topics is is frequently discussed recently is -- who should be the developer of event processing applications?  a computer programmer (the top picture) or a business analyst - the bottom picture taken from a site called "business analysts mentor",  shows what are the business analyst skill.   Will a future list will also include - event processing development?    


In the BRMS area, one of the claims is that business analysts can develop, maintain, modify and manage rules.  
There is still need in programmers for setting up the data, connect it, deploy it etc...


The desire to have business analysts develop event processing applications exists in the industry,  there are surveys indicating that most users want it,  and it combines with a general trend in enterprise computing.


Getting to event processing,  there are still challenges in getting business analysts developing event processing applications,  the challenge stems from the fact that the possibilities in event processing are quite wide, and thus there are two main options:

1. Restrict the expressive power and provide the business analyst an "easy" sub-language.  This may be enough for some classes of applications, but not enough for others
2. Provide assisting tools for business analysts to cope with the entire capabilities of the event processing language (e.g. patterns, policies).   This requires both the right abstractions (intention language) and also tools to validate that the system is working properly.     

Why it is not easy?    first -- the semantics of the event processing various functions needs understanding, while it is not a technical difficulty, it requires understanding and training,  second -- understanding the flow, i.e. the interactions among the various functions adds to the complexity.  
It is doable,  but requires more work on setting up the right tools.     More on this  - later. 

Where does the edBPM term come from?

Writing earlier today about a German friend,  another German friend, Rainer von Ammon has written on the complexevents forum about the source of the term edBPM,  Rainer is also the one who has drawn the nice illustration above showing the reference model, Rainer is the person promoting this term, and organized several workshops around the concept of pairing event processing and BPM technologies.     Rainer wrote that the term came from Gartner and quoted me.   This is almost true,  the original term that Gartner used is "event-based BPM"  and I have slightly modified it to "event-driven BPM" when I asked Rainer to write a value about it in the Database encyclopedia (I have been the editor of the event processing related terms).  
Here is the Gartner's original slide from 2005.  

This is the original slide,  source:  "Event-driven applications make event-driven businesses work better", a presentation by Roy Schulte from Gartner in 2005. 


As you see Gartner classified the event processing functions into: simple event processing, mediated event processing, event-based EPM, and complex event processing.    Roy Schulte has later realized that "event-based BPM" is orthogonal dimension to the three others, and changed the positioning.     So this is the source for all history lovers. 

On Alex Buchmann's 60th birthday book

 Alex Buchmann is an old friend, we first met when both of us were 20 years younger, and worked on active databases.  Alex is a little bit older than me, and recently celebrated his 60th birthday.  I could not travel to the ceremony in Darmstadt, but as a gift, contributed to the book, which includes a collection of papers edited by Alex's students (or ex-students).  Today the mail has brought me a copy of this book, with personal inscription from Alex.  The book is called "From Active Data Management to Event-Based Systems and More".   

More detailed about the book can be obtained in the Springer LNCS site.    The book includes a paper entitled "Spatial perspectives of event processing" co-authored with Nir Zolotorevsky.   Browsing the book I see that I am in a very good company, some of the other authors are: Jean Bacon and Ken Moody, Mani Chandy, Umesh Dayal, Tamer Ozsu,  Gerhard Weikum, and many others. 


When I summarized the year 2009 in this Blog, I have written that the quote of the year is taken from Alex's keynote address in DEBS 2009, stating is using regular database techniques for event-based systems is like trying to drink the water in a waterfall using a straw.    Alex also does not like the term "event processing", claiming that "processing" sounds like "data processing" which is an archaic term, and prefers to talk about event-based systems, as shown in the title of the book.


I wish Alex many more years of  good health, fruitful work and fun.

Revisiting EPN

This illustration, taken from the EPIA book, and drawn by Peter Niblett,  is a portion of the EPN that describes the "Fast Flower Delivery" example that accompanies this book.   In an internal discussion today somebody raised the question, why do we need EPN at all,  and not using the alternative that has been used in Amit, and other places:  each EPA subscribes to an event type, whenever an event from this event type is detected, the appropriate EPA listens to it and processes it,  and all the event flow is implicit and the person defining the system does not need to worry about it.


Since this question is actually a good question,  I wanted to share my response.  There are two main reasons why we have shifted in the thinking to the EPN model:  efficiency and usability.  


  I'll start with the usability, experience shows (and this observation is true also to inference based systems) that people feel more comfortable in ability to control the flow rather then having implicit flows, they understand better what it does, can better debug and validate it, and trust such systems more.   Note that EPN is not a workflow, it does not represent control flow, it represent event streaming flow (in a way similar to data flow, with some semantic distinctions).  


The other reason is efficiency.    If an EPA subscribes to event type then either an EPA has to process and filter out a substantial amount of irrelevant events, or the amount of event types might successfully be increased.   Imagine the following scenario:   An event of type ET1 arrives,  first it meets a filter that filters out much of the event using some assertion, and then there are various EPAs that process only the filtered-in events,  one of this EPAs is enrichment, adding some information from a database,  and then the enriched event is being sent to an aggregator for further processing.      If we use the "event type" subscription, there are two choices:  first -- create event type ET2 for the filtered-in events, identical to ET1, and create derived event of type ET2 for each filtered-in event of type ET1,  then create event type ET3 for the enriched event with added enriched attribute, and then indeed each EPA subscribes to a single event type.  The second choice is to use ET1 for all three cases, but add indication (using some derived attribute) which variation of ET1 it is, and filter inside the aggregator to have only the right type of ET1.  Both are inefficient, the first one due to the need to manage much more event types, the second is that much more events are transmitted to each EPA to filter out, and the order also becomes important here.   


The explicit EPN resolves it by the fact that each EPA sends it output to a channel and the channel can route according to source, type, assertion etc...   -  thus a specific  output terminal of a channel is really the topic which EPA subscribes to.     Note that all the possibilities mentioned before are just special cases of EPN and if one insists, such EPN can be constructed, in the extreme case, one can construct EPN with a single channel that routes every event to every EPA to decide whether it wants to use it or not,  but I would not recommend it as a good design pattern.     More - later.

On Hadoop and event processing

The region which I live in did not have much luck recently, first the big fire on the Carmel ridge, that lasted for three and half days until it got under control, and now a major storm, with winds running in velocity of >  100 KM/H  and a lot of rain.   These two pictures, taken from the Israeli news Internet sites, were taken in Haifa yesterday.   The storm is now over and some nicer days are ahead of us.


Back to professional issues -- Alex Alves (who represents Oracle in the EPTS Steering committee among other things) wrote a nice posting in his blog explaining the Hadoop programming model, if you are still not familiar with it, it provides good explanation. 


Hadoop is batch oriented and provides kind of imperative programming model, but can be wrapped and concealed by higher level language.     I am working now with a graduate student who investigates the usability of the map-reduce model for some of the event processing functions (e.g. aggregation).   I am curious  to see the analysis of this work.   More - later

On ACM Distinguished Speaker Program

Today the ACM Distinguished Speaker program announced my inclusion in the list of "ACM Distinguished Speakers".   The program is described in the ACM DSP site,  while this is a big honor, especially looking at list of speakers that include some real giants,  it is not a recognition program, but a program that has a mission stated as:  The DSP is an outreach program if ACM that brings distinguished speakers from academia, industry and government to give presentations to ACM chapters, members and the greater IT community. 


The outreach mission means that by accepting the nomination to ACM Distinguished Speaker, I commit to 
travel and give talks by request of local ACM chapters worldwide, there are some ground rules that can be found on the site,  e.g. to justify international travel (fully funded by ACM) there should be an accumulated audience of 300 people, so it typically entails multiple talks during a single trip.     I am in the opinion that I should spend some of my time in sharing knowledge with the greater community, this is the reason I am teaching, providing long tutorials in various conferences, and wrote (together with Peter Niblett) the book "Event Processing in Action".    Thus, accepting this nomination is another link in the chain, and I'll try to do my best to satisfy requests, especially from places in the world which don't get a lot of talks on the event processing area.


My speaker page shows four proposed talks, three of them deal in event processing:

•  A short tutorial that serves as introduction to event processing -- summarizing the material in the EPIA book.
• A talk about the research challenges that exist and a "call for action" to the research community in a way to move the event processing area towards its next generations
• A talk about proactive computing, one of the extension directions of event processing, on which I concentrate recently.

The fourth talk is on more general theme:   Computer Science Research in Industry -- some history and different models of how it operates.


I hope that it will be both useful to the audience and fun. 

On the 4Ds -- past version and the proactive version

The climate in Israel this year is quite strange, it is December, and today I still saw people going in the street with short dress, the summer just did not go away.  However, the forecast for the next three days, starting tonight is of a major winter storm (which here means a lot of rain, not snow) and much colder weather, so getting the winter clothes ready.   


Today I've read a blog posting by Jeff Adkins,  one of the people with most practical knowledge about event processing, who relatively recently joined IBM  GBS (Global Business Services).  Jeff blogged about the 4Ds-- detect, derive, decide, do.    These four are part of smart systems that sense and respond, what is known as reactive system.  I knew that this looked familiar, so went back to my archive and found that seven years ago we  were engaged with a project called "active integration"  (the actual application was in the insurance area, but we have generalized the concept), roughly what was known by Gartner as "real-time enterprise".    Here is the original flow from that project:

  
I don't think it has been original invention, it was a variation of concepts from control theory, but it looks very similar to the 4Ds, with a more detailed granularity.


The first D: Detect spread into two phases on our model: sense and detect, where the sense dealt with instrumentation and sensing of raw events, and detect with a detection of the meaningful situations by pattern detection.


The second D: Derive is the same:  created a derived event (and sometimes also derived data) as a result of this detection


The third D:  Decide is partitioned to three phases:  Analyze - determine the possible alternatives and recommend,  Collaborate - in case of "human in the loop" within the decision, and Decide -- apply some decision procedure to select among the alternatives (simulation, analytic methods, predetermined rules).


The fourth D: Do we called "effect", since it had to effect a running system.  


The loop indicates that the after effecting the system, it feedbacks through its instrumentation mechanism and the sense phase is looking again for things that needs reaction.


I agree that the 4D is much more catchy then our more detailed drawing.


And one comment:  when we did this work, we worked on REACTIVE system - something has occurred, we detect it, and then do something to repair.   This drawing is also a good description of our current project that deal with PROACTIVE systems, but the semantics is somewhat different:  the detection is of predicted undesired states instead of situations that require reaction since something already happened.   

Some Blog statistics - December 2010

This year I have not posted the annual statistics about this Blog readership, so taking advantage of the vacation to do it, along with going to movies, musical on stage, and bowling with my daughters.   The Blog now showing on the bottom some of the most popular postings, however, it started recording only several months ago, thus the more accurate statistics are accumulated in Google  Analytics, where this blog is tracked from September 2007.   Starting with the quantities:  There are around 1800 regular readers that are reading each posting in this blog, and additional 3600 who get into the blog from time to time (once every two-three weeks),  There are also  people who entered the blog less frequently, some of them only one time, the total number of this blog visitors is around 65000 people.  The geographic distribution is also interesting, the map is mostly painted in several green variants, so what stands out are the white space, countries from which there was no reader so far.   Europe has not white spot, America has one white spot: Suriname.   Asia has three white spots:  North Korea, Turkmenistan and Tajikistan.   Africa still has only partial coverage, all countries in the north part and south part of the continent are green, but the middle is mostly white.    This may be an indication that the Internet infrastructure in these countries still needs to go some way, or that the content of my blog does not appeal to people in these countries.

As far as the  ten countries with most views, these are:  1). USA; 2). UK; 3). Germany; 4). Canada; 5). Israel; 6). India; 7). France;  8). Japan;  9). Sweden and 10). Australia.  The readers come from 178 countries.

In a city view ten cities with most views, there are: 1). London; 2). NYC; 3). Paris; 4). Karlsruhe;  5). Haifa; 6). Tokyo; 7). Singapore; 8). Bangalore; 9). Göteborg; 10). Vienna. 

About 10.5% of the page views were direct requests, around 25% results of searches, and the rest, references by various websites. 

The most popular posting, is still ,by far, the one entitled "On unicorn, professor and elephant",  which answers a claim that everything done until today in the event processing area is just a hype and worth nothing. Since the time it was written two years ago, there were many proof points the the EP area has value to customers in various industries, and the assertion that it is still an infant, and some vendors do over-hype it is also still valid. 

The second most popular posting, is the one entitled: "On simple event and simple event processing".   This is an early posting.  In the past we used a terminology of: simple event processing (filtering and routing), mediated event processing (aggregation, transformation, composition)  and complex event processing (pattern matching).  However, I stopped using these terms since it got people more confused, due to the fact that different people have different associations with the terms simple and complex, especially the ambiguousness of  complex event processing,  that is interpreted by some as (complex event) processing and by some as complex (event processing).   I also tend to use composite event instead of complex event when talking about event that is composed of events.

The third most popular posting, is the one entitled: "On Enterprise Service Bus and Event Processing"   which is also an early posting,  this also states that event processing capabilities should be part of enterprise computing infrastructure, where ESB is a natural place to be a center point for it.  Since that time EP capabilities became even more pervasive among various technologies.

Somehow related to this is the most popular among the 2010 postings entitled: "Consolidation and pure play in the EP market".  This deals with the fact that most of the EP vendors today are big software vendors that consolidated EP within their products, while there is still a niche for pure play vendors.

While, as the blog title indicates, most of the blog postings deal with event processing, there are several off-topic postings that won a lot of responses, such as the one on positive thinking, and the one in which I described things that I heard from my father about the holocaust.   My last posting on accountability belongs to this family.

This year I spent less time on blogging, so the quantity of postings is less than either 2008 or 2009,  but I intend to catch up.    

The book "Event Processing in Action" is in someway descendant on this blog,  the publishers read the blog before approaching me to write the book,  but of course, there is more emphasis on rigor and quality within the book, the blog is "quick and dirty".      

End of summary -- next posting will go back to professional stuff

On accountability

We on the Carmel ridge are still in the after-shock of the big fire,  a good article in English describing it was written by Professor Fania Oz from Haifa University.   The rain that arrived yesterday (a little bit too late) has washed away some of the smell,  but the feeling here is bitter, also due to the politicians behavior.   Even during the four days of fire fighting, the politicians became busy in sending the blames to others.   The current political culture is that nobody is accountable for anything.   Imagine that in the business world, a corporate would have suffered a major disaster, and its executives would keep blaming each other, and nobody would think he is accountable.     While I don't tend to write about politics in this Blog, I think that in general, people who takes responsibilities need to be accountable,  and when a major disaster in a corporate happens, the CEO should draw the conclusions, otherwise the board of directors spell out these conclusions. I wish this was also part of our political culture, but where is exactly the country's board of directors?  

On intention language - take one

I am in a one week vacation now (not related to the big fire in the Carmel mountain),  had some vacation days to finish before the end of the year, and thought that the week of Hannukah is a good timing.   When I lived in the USA I surprisingly found out that Hannukah became a major Jewish holiday, but this was done due to its proximity to Christmas so that the Jewish children would not be deprived of the holiday season spirit, and the gifts.   In Israel, Hannukah is a minor holiday, people are working (unless they take vacation), and it is a week vacation in schools, thus the timing of my own vacation, spending time with my daughters. 

Driving one of my daughters today got me a good example why an "intention language" is needed if we want "business users" to program event processing systems.  I have a Bluetooth support in my Toyota Verso car, and with some effort I succeeded to program my mobile phone to connect to it, so if I am getting a call while driving I don't have to touch the phone, but can answer using the car's buttons, and hear the voice on the car's radio system.   

Today,  I realized that I need to reset the phone, something was not working properly, and I closed and opened it again -- did it while parking, I am not playing with the phone while driving!   when the phone came to live again it was working properly, but did not connect to the car's Bluetooth.  I realized that the connection is event-driven, assuming that the phone is operating when one enters the car, and the car is the one who is starting, so I tried to validated this theory by turning the car's engine off, and then turning it on again, and as I assumed, this did the trick.   


What I really want to have is an intention language:  whenever the car engine and the phone are both working, the phone should be connected to the car's Bluetooth.   


The way it really works is:   When the car is starting and the phone is already working, connect it to Bluetooth.


In most cases there is no difference between the two, but in case where the car is already working, it really does not.


When writing things in event-driven way -- it sometimes difficult to capture all possible cases, so we need and intention language which should be then translated to an event-driven one.    This is not an easy task, but one of the big challenges ahead.  I wonder if there are any relevant work, not just for simple cases like this one.
More - later.  

Event processing as blasphemy

The fire in the Carmel mountain is still on  for the third day, although it is somewhat reduced due to the fire fighting aircraft from all over the world - the most notable one from Russia, but also from other countries; it is good to see that when a disaster happens, many countries in the world are getting to help, Israel has also a record of helping other countries while disaster occurs.   I guess that the fire will be overcome eventually, but this does not change the very bitter feeling that the population here has about the incompetency of the government, where a series of faults came together to bring this disaster.   For everybody who sent me worried Emails from all over the universe:  the fire did not get into the city of Haifa, so have not been in real danger, the site of IBM Haifa Research Lab, is relatively close to the fire area, as it is located on the Haifa University campus.  The campus was confiscated to be the headquarter of the fire fighting forces, so it is still blocked.   While in Haifa we were not in danger, people in some villages lost their home to the fire, and 42 people were killed when a bus was caught in the fire.   

Now for this posting's topic -- you probably wonder what's event processing has to do with blasphemy or religion at all?  I think that I have written before about all of these, but will put it within a single perspective. When I was young I had several friends who moved through the process of becoming religious (what the Christians call "borne again", Jewish people are using a different term), I watched this process with interest, and has many discussions with them (well, they tried to convince me that they saw the light and I just have to look more carefully to discover it).   One thing that I have learned about religious people is that it is useless to argue with them, since their beliefs are based on axioms, and once you identified this fact, one cannot argue over axioms, since this is the nature of an axiom.    Likewise, there are many professional religions, I have seen religious wars in other areas of computing, and this is not really a new phenomenon, just different gods.

Here are three religions for which event processing serve as blasphemy to that specific religions.

Religion one:   The data-centric religion.

The religion's belief:   the world is data centric, everything can be done within database tools. The is a small niche which requires high scalability, but it can also be dealt using database techniques,

The blasphemy:   events processing is a distinct discipline; it has some unique characteristics. 

Religion two:  The programming model religion

The religion's belief:   all functions need to be expressed using the programming languages we know and love.

The blasphemy:   event processing has various languages abstractions that are not part of the regular languages. 

Religion three:  The "true CEP" religion

The religion belief:  The term CEP was coined to cope with application of types of intrusion detection; any person who did not work directly on intrusion detection applications is not qualified as a priest for the religion, thus cannot really deal with event processing. 

The blasphemy:  Anybody using the term "CEP" for any other application type is a blasphemer, any technique that tries to address any other event processing application is simply irrelevant  (comment:  I don't really  tend to use the term CEP, but some of the vendors  indeed use it).

As said,  the prophets (and disciples) of these religions believe in them in an emotional way, and there is no use arguing with them, so the best way is just to expose the axioms they believe in and let people think whether they believe in these axioms or not.     One of the motivations of the EPTS use case survey is to find out about the  usage of event processing today;   since it is generally agreed that the event processing area barely scratched the surface of its potential, an equally important issue is to identify what are the gaps in th state of the art  that are required in order to achieve it, and this is another major activity of the community that will be discussed within the event processing manifesto and   other related activities -- more about these topics - later. 

Fire in the Carmel mountain

 The biggest fire in the history of Israel is now taking place on the Carmel Mountain, where I live.
It started in late morning, now it is evening and there are many branches of the fire. 

In the lunch break we went out for a short walk in the Haifa University campus, where our lab is located, and saw the sun looking like this - a result of  the fire. 

. 

This is another picture of the fire, taken from the Ynet site,  an Israeli Internet news site.  


The damage is huge, much of the Carmel forests have been burned, dozens of people killed, mostly those who came to evacuate other people.   Some villages and neighborhoods in nearby towns have been evacuated, and it seems that the local firefighters don't have means to control the fire, and have approached other countries for help, well - I guess the government has other priorities than to invest in firefighting equipment.    Meanwhile, where we are located it seems to be safe, but everybody is in the mercy of the wind directions...   more - later. 

More on ACM SIG

Some people who are not familiar with ACM asked me by Email (and one comment on the Blog) to elaborate on the idea of ACM SIG for event processing, what is ACM SIG, and what will be the benefits of creating one?


The main answer, IMHO, is to get recognition that "event processing" is a discipline, and to have the logistic support for publications (periodic newsletter and scientific journal) and conferences, ACM also manages the  budget for SIGs, whose income are memberships and conferences' revenues, a capability we don't have in EPTS.   Note that DEBS is already ACM conference, currently sponsored by two SIGs:  SIGSOFT (the software engineering SIG) and SIGMOD (the database SIG).  Complete list of current SIGs is on the ACM site.   This kind of recognition will help grow the research community in this area and can help in accelerating the development of this area.  The main benefit will be to the research community, whose members have initiated this idea.     


You can look at a recent proposal about establishing "social computing SIG" on the ACM site to get the flavor.

Towards an ACM SIG for Event Processing?

One of the ideas that have been discussed several times is to apply for ACM SIG (Special Interest Group) dedicated to event processing; the idea was discussed again last week in the EPTS steering committee monthly call.    The decision has been to apply to the EPTS members and the community at large and check the interest in devising this EP SIG.   This is not intended to replace EPTS but to have an established home for the research oriented activities of the communities.   In parallel to sending a call for interest expression to the EPTS members, I am also applying to the other readers of this blog;  if you wish to support this initiative, or think it is premature, or just a bad idea -- you can take the poll in the right-hand side bar of this blog.  

On the multi billion dollars, the accounting value and and the economic value

Came back today from a short vacation in the Grand Vista Boutique hotel in Kfar Yuval, at (almost)  the most northern (and east) part of Israel.    Looking at some related blogs, I found Joe Mckendrick's blog mirroring Colin Clark  saying that the "CEP vendors" failed to produce the multi billion dollars revenues they promised to their investors this is due to two facts:  1). end users cannot understand it; 2). it does not provide the IT people a tool to build a complete application.


I am sure that Colin, who has been a  key person in one of the first generation startups in this area is right in his perspective.   However, coming from the big blue, I have a slightly different perspective.  


I'll start from the last point, since this is a key to the rest:  the claim is that the "CEP vendors" did not offer a tool to build a complete application.  This is of course true; there are applications (indeed in capital markets but also in other industries) that have applications that are mostly pure event processing applications, however, this is not the major part of the market; the major part of the applications market is such that require event processing capabilities within an enterprise computing platform, that also includes BPM, BRMS, ESB and other stuff.  This is the approach taken by the bigger companies that got into the event processing game.  It is true that a start-up cannot do it alone, and start-ups that remain "pure event processing"  companies are confined to the niches of stand-alone EP applications (in which the potential was also not reached, so there is certainly a place for products in this niche).    As for the multi billion dollars,  I don't think that there is any analyst who predicted multi billion dollars for that market in 2010,  it takes time to develop a market, and as a stand alone EP products, may take longer time to get there.    When we did the business evaluation of the EP market around 3 years ago, we got to the conclusion that for the market of  "EP capabilities within an enterprise computing platform", one can look at accounting value or on economic value and they would yield different results.   Let me give a simple example in order to show what I am talking about:  let's say that a big enterprise is doing a software deal with the value of $15M for its enterprise computing infrastructure.   This may include many software components licenses.  Let say that the deal is a bundled deal, but if we'll try to isolate the value of the EP capabilities license, we'll get to  $500K.   However, in the RFP the event processing capabilities are required part, and a vendor that does not have the required capabilities is disqualified for this bid, then this vendor, losing the bid did not loose a $500K deal, but lost $15M deal.   This is the distinction between accounting value and economic value.   We see more and more RFP that either put event processing capabilities as mandatory, or give some weight to it as explicit criterion, this is done, even if the intended use of EP is planned in 2-3 years in the enterprise IT investment plan.     I think that looking at economic value (e.g. deals that have EP components inside) we might already passed the 1B dollars mark,  but this arithmetic, of course,  makes sense to vendors who own enterprise computing platforms and not to start-ups that has to earn the actual accounting value of stand-alone EP applications (or sell it as embedded solution).    About the "not appealing to end users"  - I'll write a separate posting about it; some of the analysis from the user's point of view was done as part of chapter 1 of the "event processing manifesto" (a follow-up to the Dagstuhl seminar on event processing) which is planned to be published in January 2011, and presented in March 2011 within the virtual EPTS symposium.   More about this topic - later. 

Some EPTS related news and call for application survey

With several more members that joined EPTS (= event processing technical society) recently, the number of members is getting closer to 100 (I think we are in 95 now).  There are several recent activities associated with EPTS:

1. The Dagstuhl seminar  on event processing held in May 201 which is affiliated with EPTS and the follow-up activities have resulted in a complete draft of the "event processing manifesto" now up for review among the team.  I'll write about it in a separate posting.
2. EPTS is going to hold its next symposium as a virtual symposium, meaning both speakers and audience will be able to participate over the cyberspace and the talks will be recorded.  The plan is now to hold it in March 2011 in conjunction with the OMG technical meeting,  I'll write more about the agenda, when it will be finalized (probably next week).
3. The EPTS use cases working group, lead by Pedro Bizarro and Matthew Cooper has posted a use case survey, which should take 15 minutes to fill,  please contribute by responding to this survey. 

OMG workshop on real-time systems

Within the OMG technical meeting in March 2011 in Washington DC there will be a workshop on real-time systems,  the call for presentation and tutorials is linked here, with the deadline of November 26.   We plan also to do a virtual EPTS symposium during this event, to enable a broad audience to listen live or to a recording, more details - later. 

The human body as event producer

I realized that I have not written anything in this blog, in the last week, well - there are weeks in which one is in a passive mode, anyway, I am at home, ready  to go to the office (Sunday is a working day in Israel, our weekend is Friday-Saturday), but wait for the technician from the phone company, all our phone lines are dead since Thursday afternoon.   Since each member of the family has cellphone, it is not a big damage, the most notable damage is that the wireless internet at home is linked to the phone (ADSL),  however, we have a second home LAN (wired) which is based on the cable TV, since none of the internet connections are reliable, they also serve as a backup for each other.   The phone company was a government owned company that was privatized, but retained a strong employee union as a legacy, and as such, its service is not that good, so we'll see when the technician will bother to come.


Anyway -- the picture above is of stream of events going out directly from the human brain, it is taken from a recent article in Scientific American.   There is a lot of recent work about brain-computer interface, the brain itself is, of course, an event processor, it receives events through the senses,  makes some decisions, and acting with some actuators.  There are also a lot of events being created in the human body  that has benefit to be processed elsewhere, either due to fact that we want to have "extension" to the brain to help sort out the events and detect situation that the human brain may miss, or that the human body is just one source, and there are other sources for events that this particular human is unaware of.     


I see a lot of future in taking what was done in event processing and utilize it outside enterprise computing, and I'll investigate this theme more in the next few blog postings. 

On the Rabbi and the limited appeal

An old Jewish folk story is about husband and wife who came to a Jewish Rabbi for marriage counseling, the Rabbi listened to the husband and said: you are right,  then listened to the wife and said again: you are right. After they left his assistant asked him: "Rabbi - why did you say to both of them that they were right, it is not possible that both of them are right", and the Rabbi answered again - "you are also right".    I recalled this story after reading Mark Palmer's Blog entitled:  "how broad is the appeal of CEP".   The story starts with the CEO of SAS  claiming that CEP has limited appeal continuing with the rebuttal of Progress and TIBCO, saying that the SAS guy underestimates the potential and there are many applications in multiple industries.   Mark Palmer has a counter-opinion saying that relative to BI, EP is still small, but has a potential to grow,  and it is hot within a single industry - capital markets, with relatively modest interest in other industries - Chad Gadya!     


So why does it remind me the Rabbi's story?  -- since each of them is right in something.
EP is indeed a smaller market relative to BI, and much younger and less mature.  Some of the BI people regard EP as a "non issue" since one can do anything with BI tools, or database queries, and the only reason to use EP is when there are latency and throughput requirements that cannot be satisfied.  However, as noted many times, the bigger appeal of EP is the abstractions that enable to develop and maintain applications of that type more easily and substantially reduce the TCO.     The observation that EP market is  much smaller than the BI market today is certainly true,  maybe the fact that the SAS guy bothered to react on it is a sign that they start feel some presence in their territory.    BTW -- I don't view BI and EP as competitive technologies, each of them are destined to do different things, some applications need both.


Next --  inside the  EP house,  it seems that there is some agreement, about the potentially bright future, and some disagreement about the present and  even more about the scope.    I guess that appeal is in the eyes of the beholder,  but let me make one observation:  when Streambase is looking at the universe it looks at the market for a stand-alone event processing product; when some of the bigger companies look at the universe, they look at event processing capabilities as part of a larger enterprise computing infrastructure.   When we did, a few years ago, the business evaluation work for IBM, towards the decision to enter this market, we have identified both of them as opportunities, the stand-alone one has its own existence, but is more limited in scope,  for the "embedded" one -- possibilities are much higher, but more work is needed to make event processing capabilities as part of facets of enterprise computing infrastructure -- decision support platforms, business process management platforms, messaging oriented middleware,  sensors and actuators frameworks and  even business analytics framework.   These two view points of the universe provide different perspectives to the beholders, and so each of them is right from his own perspective.      More - later.  

Back to temporal databases

In 1998 I have edited a book of articles about temporal databases (together with Sushil Jajodia, and Sury Sripada), this followed a Dagstuhl seminar we held in 1997 about temporal databases, an area that was hot at that time in the research community, and somewhat cooled off.   Today a Master student I supervised took her final exam on "final work" (which is less than a thesis, a track that require to take more credit work), and did an implementation of a temporal database model from a paper in this book that was co-authored by Arie Segev, Avi Gal and myself.   This is somewhat more expressive model than the TSQL based models, and had its own interesting featured like: ability to freeze and unfreeze data, ability to distinguish between modification and revision, ability to deal with simultaneous value.  In fact some of these ideas found themselves into our work on event processing (e.g. policies when there are repeating events that may match the same pattern).  


Temporal databases as an area started in the Israeli army. Kobi Ben-Zvi who went from the Israeli army to do PhD in UCLA has invented the area, by formalizing the terms, and there has been a lot of work later in the research communities in the 1990-ies.    There was even big fight about how to extend the SQL standard to support temporal databases between two parties,  I don't really remember the details, in the book you can find the position of the two sides of this battle, as the Dagstuhl seminar was one of the battle fields.  The end result is that it never became part of the SQL standard, partly because of the fights, and more importantly since at that time the DBMS vendors have higher priorities on their mind -- e.g. Web related stuff, XML data etc..   There are some features, but it did not get fully into the mainstream of databases, although there are quite a few of specialized implementations.     One of the future directions of event processing will involve getting back to temporal databases as an infrastructure,   which is the area of retrospective event processing, I'll write more about it in the future.