On big data, small things and events that matter

In a recent post in the Harvard Business Review Blog entitled: "Big Data Doesn't Work if You Ignore the Small Things that Matter" ,  Robert Plant argues that in some cases organization invest a lot in "big data" projects trying to get insights around their strategy, while failing to notice the small things, like customers leaving due to bad service.   Indeed big data and analytics are now fashionable and somewhat over-hyped.  There is also some belief, fueled by the buzz that it solves all the problems of the universe, as argued by Sethu Raman in his DEBS'12 keynote address.   Events are playing both in the big data game, but also in the small data game, trying to observe a current happening, such as time-out on service, long queues etc..., when it relates to service, and other phenomena in other domains.  Sometimes the small things are the most critical.

I'll write more about big data and statistical reasoning in a subsequent post.

More on the semantic overloading of derived events

I am recently getting back to the time in which I have dealt with semantic data models, and now I am trying to view current event-driven applications in that way, thus the semantic overloading is one of the interesting first issues that emerge.  I'll write more about semantic modeling of event processing later, but right now I'll concentrate in the semantic overloading of derived events.   There are various definitions of the term "event", but in all of them event represents a VERB in the natural language.   Looking at what we defined as derived events, it seems that some of the derived events we are looking at can indeed be described by a verb in the natural language, while others are really described by nouns.    Thus my current thinking is to have the semantic notion of DERIVATION, but the derivation can yield different concepts:

Events - when indeed the derived conclusion is that something (virtually) happened.

Entity facts - when the derived conclusion is a value of some fact

Messages - when the derived conclusion is some observation that has to be notified to some actor. 

Examples from the Fast Flower Delivery use case that we used in the EPIA book.  

The automatic assignment creates a real event -- can be expressed by the verb ASSIGN

The timeout pattern "pickup alert" which means that a pickup was not done on time --- this is an observation that is notified to somebody.  It is therefore a message that can be expressed by NOTIFICATION

The driver-ranking calculated as a function of assignment count, is actually a fact related to driver, driver-ranking is a noun, thus it is a derived fact.

More - later. 

On family vacation in Malta

I have not disappeared, spent most of the previous week in a family vacation in Malta.   Malta is a small country consists of several islands in the  Mediterranean sea  The climate is similar to the one we have in Israel.  The local language is also a Semitic language like Hebrew, it is actually a descendant of the Arabic language mostly. Like any other Mediterranean people - they are nice, friendly, and take their time.  We lived 5 minutes walk from the capital city of Valletta, which is a small city surrounded by walls and looks somewhat similar to the old city of Jerusalem.  In the evening it looks like a deserted place, nobody is walking in the street, and besides a few restaurants it looks like a ghost town. 

Some highlights:

There is a prehistoric site which has limited visitation (10 per hours), we made reservation 2 months ahead to get inside. 

Malta is a catholic country, and have a lot of ancient churches, we have visited some of them. 

We decided to rely on public transportation, which is cheap but time consuming, some of the lines have low frequency --- not really a good choice.

We took a day trip to Gozo, the northern island which has nice beaches.  

The Maltese people like to celebrate - we have watched too separate celebrations, one of them in the honor of the national fish, called Lampuki,   the other in Valleta  is the Valleta white night.  I am not sure what the celebration is about, but it was crowded and noisy, which is a contrast to the regular silent in Valleta.

Hope that my daughters will post pictures on Facebook soon. 

In general:  nice place for vacation.

I'll continue with professional blogging, hopefully tomorrow.

The semantic overload of derived events

The term "derived events" is frequently used in the event processing terminology.  Here is an example taken from the Gigaspaces Blog of using the term "derived event".    Recent discussion with somebody who is just learning the event processing area, made me realized that we overload the term "derived event".  On one hand we define event as "something that happens", and say that a derived event is an event, thus one may assume that derived events also happen.    However, the way this term is typically used has some semantic overloading.   There are actually different types of derived events.

One type of derived event is really an event that we did not observe directly, but concluded that it happened by observing other events.  Such cases is the case of fraud detection, money laundering detection and system problem diagnostics.   

Another type of derived event is a notification.   We are doing a calculation based on events and the result is notified to some person of application.  For example: derived event that calculates the highway fees, based on exit and entry events on the highway, and rate calculated by load on the highway.   This is a derived event, however - it does not really stands for something that happens, the happening here is a notification to the driver how much the fees are,  there are many derived events of this type.

The question is whether we should make distinction between the two cases. From semantic point of view they are clearly distinct.    From execution model and language point of view -- they are indistinguishable; both take events as input, apply some assertions and functions over a collection of event, and create a structured message sent on some channel.    From semantic point of view there is a difference,  the question is from pragmatic point of view, is this distinction important for somebody that takes any role in the life-cycle of the application.   More -later.

On Google Intelligence Events

I am using Google Analytics (quite infrequently) to view the activity on this Blog, but for those using websites for commercial purposes, or as a social media vehicle, tracking activity on website is a very valuable tools, especially in our metrics-driven universe. The illustration above which seems as a typical sense and respond cycle is taken from an article in seemingly odd location;   the social media sun recently reported on a feature in Google Analytics called "Google Intelligence Events" (although I think this feature is not new).    The article claimed that this is "a simple feature called complex event processing", falling into the "complex" word trap,  and in other cases it asserts that "Google's intelligence events is only a basic use of event processing engine".

In fact, what this tool is capable of doing is issuing alerts of two types:

Automatic alerts:  indicating significant changes of traffic to the website

Custom alerts:     threshold crossing of a certain indication (e.g. the traffic from Singapore was down 20% from previous day).  This is a threshold over all tracked variables (source, demographics, bounce rate etc..). 

As far as event processing is going -- it is indeed limited capabilities, mainly threshold oriented comparing two set of events (based on time).   However, it shows that event processing has got into the web analytics world and there is a potential of doing more in this space.  More on this -later.    

Event is a relative term

Following recent discussion with Jeff Adkins about the semantics of event,  one of the observations has been that event is a relative term.  First - different observers can describe the same event in different terms, and indeed different observers may not agree about - what really happened, or where it happened, or when it happened, or all the above, kind of "Rashhomon effect".    It may also be relative in the sense that different views may look at different properties of the event,  for example one view will only look at a customer enters the store, another view will also look at demographic properties of the customer -- age group, gender.
The relativism effect can mean that:  event meta-data may have different views with different structure, and event instance may have several corresponding instances according to the observers.  

The question is -- how to consolidate different observers?  this may involve modal  semantics.  I'll write about it at later phase.  

Jeff also recommended me to read a philosophy book called :"The ontology of mind: events, processes and states" by Helen Steward.  I have ordered this book and will write a review about it upon reading it.  

While you have slept - more about the big brother

Back to Chris Taylor's - this time he wrote a guest blog in the Forbes blog entitled "While you slept" dealing with issues of privacy.   I have written before about the big brother aspect on using sensors within the smarter planet. This is other aspect -- a lot of information are flowing on us from various systems, social networks, and various systems.   A smart program can look at what we published in blogs, twitter, facebook and others and determine on our political opinions, religious beliefs and others.  It seems that the combination of sensors, activities on social media, and other systems make us give us our privacy and many people don't seem to be bothered -- they allow everybody to see their pictures on facebook, or read what they write there.  There are people who report in Twitter on what they are doing 30 times a day including all their happy and frustrating minutes.  It seems that there is a growing section of the population who are giving up their privacy from their own will, and other people who are not aware that their privacy is been invaded.   I guess that this is one of the characteristics of the current web generation - losing privacy.   Of course, event processing can help in drawing conclusions  about a person.   Chris ends his posting in call for companies to be aware and set up policies and strategy and for government to regulate.