Tuesday, April 14, 2009

On Innovation, Twitter and Event Processing

Reading in the newspapers about some troubles in Thailand reminded me about my family trip to Thailand two years ago, country that is full of paradoxes, but one of the most attractive to tourists, tourism is a major industry in Thailand and the local people have very innovative ideas for tourist attractions, one of them is to let people touch tigers, an animal known as one which is not easily tamed, here is a proof.

Talking about innovation, two recent postings recently motioned Twitter in conjunction with event processing, the one by Richard Veryard talked about "innovation in a small bakery" describing a small bakery which sent Twitter notification whenever freshly baked bread came out of the oven, and wondered why we don't see more innovation like this from IT departments in organizations. The second is a discussion thread that started by David Luckham on his CEP forum asking whether CEP can be used to trace Twitter notes.

Some people have already answered David, I guess that one of the difficulties is by the fact that Twitter messages are written in a free text, however, trying to apply full scale information retrieval techniques with statistical reasoning will miss the point of simplicity, the main consumers of such applications are individual consumers that cannot apply complex software and reasoning, one cannot kill flies with a cannonball. So what can be done: the messages (like in the bakery example) should be fixed and include keywords that can be easily filtered. For example, the bakery may have three messages about three products: bread, rolls and cakes and can also send alert when each of them have been fully sold. The bakery can provide a way to subscribe to each or make more complex subscriptions, and here comes event processing patterns to be used for example: I want to go to the bakery only when there is both fresh bread and fresh roles are available (and not sold out yet), which is a conjunction pattern, or I want to restrict it if the conjunction of the two event happened within 15 minutes, since I like it very fresh... In this case the identification of the event type can be done by a simple filter that search for the keyword "bread" or "cake" together with the event source (bakery) and time-stamp. If more stores join this type of service, I can do a conjunction of Twitter messages from both stores buying fresh bread and fresh shrimps in the same trip (adding some condition on the distance between the stores, given by an additional service. Furthermore, I can also condition any subscription in my personal context (e.g. only when I am at home, only in the morning hours etc...), and this is initial ideas, so for sure, Twitter as a platform for event notifications can have many usages (which does not say that every Twitter message carries event information).

Richard's question about innovation and IT department is more complicated one, however, there is some truth in his observation, based on my own experience in an IT department of a large organization, IT departments may be more conservative than their users, and typically will be cautious about be anything that is giving a "programming capabilities" to the users (and is conceived as "losing the control"). Since many event processing applications' value to the business users is doing exactly this (giving more power to the business user to do their own programming) it is sometimes a challenge to get it through the IT department, but this is a longer discussion...


Andreas Bauer said...

Yes, I agree with you that the use of cep systems should be easy and the complexitiy should be hidden from the user. For this the complexity of IR techniques, etc. should be taken away from the user by providing RIAs on top of the CEP systems, that ease the use and creation of patterns.
This would also facilitate the acceptance of CEP in the consumer area. Sure, consumers aren't interested in how to process twitter data or text data in general, but I'm also sure that there are many questions out there that consumers want to get answered and that can be answered by cep, like the bakery example.
As more and more people are ubiquitously connected there is lots of real time information crackling down on them. And foremost it's not well structured but un- or semistructured information created by human beings. As you can't expect people to format their tweets, emails or blog posts, you have to apply IR or text mining techniques to extract the necessary information, that - then - can be fed into an event stream for furhter processing.
On this point we should think of a meta model for mapping text events or human initiated events. I think this is worth some research, isn't it?

Opher Etzion said...

Hello Anderas. Thanks for your comment, certainly having a combination of event processing and pre-processing done for various types of unstructured events (voice, video, text) is a very worthy topic for research - this can increase the use of event procesing in the consumer domain.