More on Twitter as an event source

I have written sometimes ago about creative use of Twitter as an event source; the recent days riots in Iran, where strict media censorship is imposed, indicate the role of means like Twitter. It turns out that Twitter has a major role in communicating to the outside world what is happening in Iran. The state department has asked Twitter to cancel scheduled maintenance to keep Twitter up and running all day. To all people who doubt the power of what can be expressed in 140 characters, it seems that Twitter is becoming a major event source in our universe, due to the ease of connecting it from everywhere. Of course, one can abuse any event source and send false information, but the mass quantity exposes the truth. I'll revisit text messages as event sources in later posts.

The Babylon tower and event formats

Still in the USA, after the OMG meeting in the Washington DC area, I got to the Boston area, and now I am in Burlington, MA, visiting the (former) Aptsoft guys. When driving abroad I am renting a car with GPS (see above to determine which one), and it typically gets me to where I want - however, it is still not totally reliable, in the last day it confused me twice, one yesterday night to find the hotel, it told me to turn left, and meant in the next turn, not in the current turn, but it did not say so, thus, I found myself back on the I95, and had to turn around at the next exit, and try again. Today it sent me to some shopping center instead of the Aptsoft site, and after I ignored it and found it - the local people told me that the GPS maps have the numbers of the street in the wrong directions (starting from the other end) - well, sometimes the technology is fine, and the weakest link is the data it uses.

Talking about data, one of the topics that were discussed in the OMG meeting about standards is the topic of -- semantic/structural standards for events. I have used the term "Babylon tower" in one of the earliest postings in this Blog and meant that we have Babylon tower of languages - like the original tower who separated the languages. However, there is another Babylon tower that relates to event format - syntax and semantics, and here the problem is even harder, since there are multiple formats in multiple domains, nobody even made an inventory survey. One of the presenters said (and it is true in some cases) that 80 percent of the cost of building EP application is to set up the events from the sources and transform them to a processable format by composing adapters (hand coded, or by using transformation engines). This is some of the domains that need more investigation, and perhaps we need a meta-data standard here.

On the classification of events

I am returning to the area of "classification of events", although it has been discussed in this blog several times regarding certain aspects, the reason is - some discussion in which I realized that the confusion is still alive and kicking.

Events can be classified according to their structure, semantics, or pragmatics. This classification is orthogonal.

Based on Structure:

• Simple or atomic event - an event that reports on a single occurrence.

The event structure may be flat (like relational database), or semi-structured (XML) - which also can express non-flat structure (e.g. hierarchical). In some cases events are represented as objects. Typically events contain "header" - information about this event (event class, time stamp, source...), and "payload" - the content of the event.

• Complex event - an event that is composed of multiple events

There is some difference between this and the term "composite event" from active databases which mixes structure and semantic aspect, but I don't see it useful to use this term any more.

Based on source

• Raw event: Event that has been reported to the event processing system by an external event producer.

Note that a raw event can be simple or complex. The term "raw" is relative, since it can be result of some computation process in the event producer that is transparent to the event processing system.

• Derived event: Event that is created by the event processing system as a result of some processing.

Again, the derived event can be simple/atomic - where the attributes are computed as functions of raw and derived events, or complex - in which the derived event is a concatenation of events.

Based on ontology:

• Real event: An event that designates something that occurs in reality
• Virtual event: Hypothetical, simulated or uncertain event.

Again, this is a separate dimension - can be in any structure and by any source.

Based on pragmatics

• Reactionable event (AKA situation): An event that is consumed by an external consumer at the end of the event processing network (either notify or orchestrate).

We used the term "situation" for a phenomenon in the consumer's domain of discourse that the consumer wishes to react to. This can be raw event (and in this case the event processing system is reduced to "simple event processing" - i.e. filtering and routing) or derived event, it can also be real or virtual, and in any structure.

BTW - ECA - "Event-Condition-Action" rules are rules in which the events are situations, and they are executed within the consumer domain, but I'll continue to discuss the conceptual model in the next posting.

Events from unstructured sources

This is an holiday in Israel ("succot"), and I have travelled with my family to the national park in Ramat-Gan (about 100 KM away from home, not that far, well - Israel is a small country, you can't go far anywhere..) to see the exhibition of "speaking chairs", many artists have painted and sculptured chairs, where all the basic chairs look like this, and the chairs are spread across the park (here is one sample in the picture)...

Today's topic is about getting events from unstructured sources. Event has been defined as something that happens, and one of the questions - how do we know it happen. Some of the events are pushed (e.g. stock quotes), but many of the events in the universe, should be obtained by analysis of various sources of input. Video streams is one source - e.g. understanding from a video stream that somebody climbs a security fence, a stolen car just entered the highway, and a person whose driving licence has been revoked is driving. Some other interesting events can be obtained from news streams, even from Emails, and other type of unstructured texts.

I believe that the event processing of the future will deal not only with the events that are easy to get, since somebody pushes them, but in many cases, with events that are not easy to get, and not easy to realize that they happened, since it will involve analyzing different media streams, we can see some early implementation of this area, and I believe it has a big potential, good topics for theses, also.... have fun.