On the duration of an event

I have neglected the blogging for a while, returned from my trip in Asia, planning for my next business trip to USA (I am travelling too much, I hope for a non-travelling period after that, but one can never know), and also took some days off for the Succot holiday.    Yesterday I traveled with most of my family to Tel-Aviv, to "Beit Hatfutsot", which stands for "Diaspora house", and documents the life of Jewish community over the history in many countries.    Here is an artifact from the exhibition:

There was also an exhibition of Andy Worhal painting notable Jewish persons,  one of the pictures is of Golda Meir, the only Israeli Woman prime-minister (time for the second one?) 

The VLDB conference also uploaded pictures from the conference, so here are two pictures - one from my tutorial, and the second showing me in the first raw (it was not really the first raw, but it was the first captured by the camera) listening to the keynote talk:

While I have been away there were some Blog posts by Paul Vincent that worth focusing upon, I have already commented briefly to this one, but want to have longer reaction about the issue of event duration that was raised by Paul.   


In most of the models events are considered as instantaneous, occurring within a single time point,  the temporal database glossary from 1998 puts "instantaneous" as part of the definition of event, the rationale is of looking on event as transition between two states, and transition in most models takes zero time,    A few years ago when we started the discussions about terms, I've pointed out the temporal glossary as a source for event definition, and David Luckham issued a strong objection to that definition, claiming that no event is really instantaneous,  even simple events like the "aircraft is landing" takes more than zero time, while events that are composed of other events - "complex events" - like the 1929 crisis (now we can talk about the 2008 crisis) is compose of many events and occurred over an interval.   


This is, of course, true, yet it is more convenient from computational point of view to deal with discrete time points than in intervals, furthermore, some systems have detection time semantics, looking at the time-stamp in which the event entered the system, rather than the time it occurred, this is the reason that we find time point semantics in most systems.


We can look at the following cases:

1.  The event really occurs within a time point,  e.g. time series of sensor measurements, or stock quotes.  There is indeed an interval among two successive events, but this relates to the state between the intervals and not the events themselves.
2. The event occurs within an interval, but the granularity of our time computation is bigger than the interval, thus we can approximate the interval to a time point.   Example:  the granularity we are interested is an hour, thus even if an event occurs within several minutes, we can still approximate it to the closest hour. 
3. The event occurs within an interval, and it is important to process it with an interval semantics, since we would like to see it relationship to another time interval (e.g. temporal context).
4. The event occurs in an unknown time-point that is bounded by an interval, there is some probability (e.g. uniform distribution) that it happened in any point of time within the interval.  In VLDB there has been a paper by Yaneli Diao and her students entitled: Recognizing Patterns in Streams with Imprecise Timestamps    Note that in this paper there are also some references to interval based semantics (of type 3).
5. Derived events are another type of events whose temporal semantics may be tuned.   For example:  the derived event "frustrated customer"  is being derived when a customer approaches a call center the third time about the same topic,  the question is whether the customer is frustrated only when approaches the third time, or the customer is frustrated over all the time since the frustrating event occurred until it is fixed. Furthermore,  derived event may also indicate an event that will happen in a future interval.   I'll write more about this issue in the future.

Bottom line:  the event processing systems of the next generation should support both time point and interval semantics along with uncertainties (Paul also had posting about "fuzzy patterns" on which I'll write in the future).

On Singapore and some thoughts about the western democracy

Back home after a week in Singapore (VLDB conference + a few days of being a tourist, using public transportation).    Here is the Merlion - Singapore's symbol:

and here is myself in the Singapore's zoo which resides inside  a beautiful rain-forest.

I also learned about Singapore culture and unique political structure, Singapore is a combination of people who are by origin Chinese, Indian and Malayan - and all these languages are spoken, but the common language is English.  I have read in the local newspaper brought to my hotel room every morning an interview with Lee Kuan Yew,  the person behind Singapore, one of the question addressed to this 87 years old statesman is about the democracy style.  Singapore does not have democracy in the western style, what Lee called in the interview "competition based politics", and the power is held by a single group of people throughout Singapore's lifetime so far, but the results are very impressive -- it has the largest growth rate in the world in 2010, the economic is good, the quality of life is high and people are generally happy, the government is one of the less corrupted and most transparent in the world.   This adds to the skepticism I have developed over the years towards the western style politics, of course, there is always a danger that people in power may abuse it, but this is true no matter how they got to power. I am not sure that the current system in which people elect their leaders according to their TV appeal really brings the right kind of leaders.  Come to think of it, in the corporate world, a corporate does not elect it CEO that way either.  Maybe we should think of a country leadership like corporate leadership,  with board of directors which represents the interests of the stakeholders (the citizens), and a CEO that manages the country and being nominated by the board of directors (that can also send the CEO home anytime),  this is not really the same as a parliament system - of course, there are many details to sort.     We are educated that the western style democracy is an axiom, and that any deviation of it is dictatorship and bad for the people,  Singapore is a counter-example, while I will not claim that everything is bright in the Singapore system, the world does not consist of black and white, I was very impressed by what has been done in this island without natural resources over a relatively short period of time,  I think that the west has something to learn from Asia in more than one sense.    More - later 

On VLDB 2010 and the event processing tutorial

Still in Singapore,  after the vacation, this is VLDB time,  VLDB is one of the largest databases conferences, however databases is a large heterogeneous area, and I wonder if there is a single person who can understand all  talks.   The keynote talk by Divesh Saristava from AT&T Research talked about stream warehouses, or event stores in my language.  There were also some event processing related demos, and a stream research track, with one interesting talk that compared the semantics of Coral8 CCL language to this of Streambase and got to the (not surprising) conclusion that their semantics is different and similar queries would yield different results, then it tried to come up with a framework to generalize the two types of semantics, and they extend it to other languages.   I think that this is in line of the work we are doing on common model, and will follow up with them (they are from ETH Zurich) about collaboration on that one.


Today I have delivered a tutorial under the title "event processing - past, present and future",  much of it follows the EPIA book.    Since this is a database conference I opened in showing various opinions about the relations between event processing and data stream management, which is the name used in the database community, the various opinions are:

1.  They are aliases -- a stream is just a collection of events, likewise, an event is just a member in a stream, and the functionality is the same.
2. Stream management is a subset of event processing -- there are different ways to do event processing, streams is one of them
3. Event processing is a subset of stream management -- event streams is just one type of stream, but there are voice stream, video stream and more streams
4. Event processing and stream management are distinct and there is no overlapping between them.

As I have heard all four opinions about it,  I'll let you judge which is the right one. Hint:  option 4 is totally false, there is some truth in options 1-3, depending on the viewpoint.


Anyway - the tutorial has been uploaded to slideshare, and you can view it there. Enjoy.


Tomorrow is my last planned day in Singapore, and I'll write more about this very impressive country soon.

On Hong Kong and ambidexterity

Hello from Singapore.    I have arrived here from Hong Kong yesterday,  here is a picture of myself in the Victoria Park in Hong Kong.   Hong Kong is run as a "special autonomy region" of China, under the "one country, two systems" idea.    I am not sure exactly what are the borders of autonomy, but it sure have a different look and feel from China -- different currency, car are driving on the left side, all announcements in the public transportation are done in Chinese and English, even in the local newspaper I've read about a possible teachers strike, something we are used to in Israel, but I am sure you don't see in China.   Hong Kong also has its own flag 

The "one country - two systems"  method, reminded me of the previous director of IBM Haifa Research Lab, who used to talk about ambidexterity,  where he meant that different activities in the same organizations have to be managed differently;   while this idea was not implemented at that time, I think that it is a really good idea,  organizations often tend to impose the same management model on all its activities, and this can be counter-productive,  managing going on activities, and innovative activities that might lead to breakthroughs should be totally different, as they are different in expectations, way people should be measured, and often in the organizational structure.   I think that we should see more of "one organization - two systems", some of it is managed by the right hand (and left part of the brain), and some by the left hand (and right side of the brain).

 

I'll write more on Hong Kong and Singapore - later.  

Going east

Packing to go abroad again -- this time to the east, the destination is Singapore, where I am participating in VLDB 2010 and giving a tutorial on event processing - past, present and future. 
But first -- I am flying later today to Hong Kong, as a first stop in the east.    

Some thoughts on data mining and event processing - take one

Somebody with whom I've talked to last week said with some irony that in order to get attention today - no matter what you do, you have to say that it has something to do with clouds, and that it performs some kind of analytics. Well - it is a clear bright day today here without any cloud, so I'll delay the discussion about clouds to rainy day.   As for analytics, there are various types of analytics, today I'll write something about data mining and event processing.  There are two sides here: what event processing can do for data mining, and what data mining can do for event processing.  Let's focus the discussion now about the second issue.  The answer seems easy,  an event processing application is modeled by event processing network, that consists of event processing agents, which in most current implementations are implemented by rules/queries or some other constructs.  Today the application is being composed manually using some authoring tool -  however, there is a frequently asked question, can the computer somehow use magic to compose the application itself -- this is a natural candidate problem for data mining.    The achievements of data mining in event processing until today are somewhat modest, but there still might be a promise there.


So - let's go further and explore the potential.   We can look at three types of functions  that an event processing application may be assisted by data mining:

1.  Case in which we are looking for anomalies in general, the data mining can assist in identifying that we have anomaly now,  this is actually a different type of application in which there are no preset patterns.
2. Case of  detect of trends/thresholds oriented, where the thresholds can be adjusted by data mining
3. Case of pattern detection - where the patterns can be determined by data mining.

The first type is a classification issue - and this can be done by some types of learning of what is normal behavior. 
The second type -- learning thresholds also has some known methods in data mining.
The main problem is the third one -- learning patterns.   There are several difficulties there, the first one is the intent;  data mining typically discovers events that happen together, this by itself may not be of interest, since the aim of patterns are to detect situations that require reaction, thus there is some additional semantic knowledge here that is not captured by data mining without providing additional informations, furthermore the pattern may occur very rarely, such that it will not be captured within the existing data; another difficulty is the richness of pattern types and the various variations of patterns, so looking at the space of large possibilities.  
Successes in this area were typically limited to a certain type of pattern within a certain temporal window -- for example, there was some work that I familiar with to mine sequence of two events within a given temporal window, this again belongs to events that happen together, where a human has to go over all combination and decide whether they create an interesting situation.     


Bottom line -- no magic bullet,  but any breakthrough in this area will be helpful  

Congratulations to Richard Tibbetts for being named as TR35 2010 young innovator

Congratulations to one of the notable persons in the event processing community - Richard Tibbetts,  Streambase's CTO, for being named as TR35 2010 young innovator by the "Technology Review", a media company owned by MIT. 

This award is granted to young innovators under 35 years old, whose inventions and research are the most exciting, according to the judgement of the editors.   

There are also some other awards, most notably some the prestigious mathematics awards, that has age limitations - well the world belong to the young people these days. 

Anyway -- Richard is certainly both young and a notable innovator, and this shows another indication of the interest in event processing within various forums.  The award is indeed well deserved.