In the EPIA book we referred to the spatial dimension of context, and also to spatial patterns of events, there is also a paper on this topic that I've written last year with Nir Zolotorevsky, There are also combination of the spatial and temporal perspectives, both in sense of composite contexts, and also spatiotemporal patterns
(such as: going north).
It seems that spatiotemporal event processing is becoming popular. Alex Alves from Oracle presented a paper about it in the industrial track of DEBS'11, and also recently wrote a short posting about it in his Blog;
Seems that Oracle is putting a support of spatiotemporal features in its product.
I also came across some description of spatiotemporal processing within Microsoft StreamInsight,
The current support in spatiotemporal capabilities is quite elementary, and various extensions are possible (a student project for the next semester?). It will be interesting to see more about applications that utilize the spatiotemporal capabilities, and their functional requirements.
This is typically a professional Blog, with some off-topic deviations; this posting is one of them.
Nowadays nobody that lives in Israel can ignore the phenomenon of the last couple of weeks: mass protest in term of people demonstrating by living in tents in various places, mainly central Tel-Aviv, and the demonstrations all over the country last night (see picture above from the Tel-Aviv demonstration). What is the demonstration about? -- it started as protest about the cost of housing both purchase price and rental price, and accumulated other reasons for protest - talking generally on "social justice". In Israel the political focus as well as the major voting pattern is determined according to the war-and-peace decisions. Thus there is some abuse of the terms "right" and "left" in the Israeli politics, where (in gross generalization) typically "left" is referred as those who believe that Israel should take active steps towards peace, and support the establishment of Palestinian state, and "right" is referred as those who don't believe that peace is possible in our generation, and are against the establishment of Palestinian state. Since many of the people are frustrated from the "peace process", there are more in Israel who voted for the "right" block in that sense, and currently we have the most extreme-right coalition ruling the country ever. However the politically right administration happens also to possess social and economic right ideology, and it seems that this is the break point between the government and the people.
There are two types of socio-economic systems: the American system in which there are relatively low taxes, and relatively low social services; and the system of some of the European countries - with high taxes and high social services. It seems that gradually Israel adopts the bad aspects of both -- taxes are high (used to maintain the enormous defense expenses as well as supporting sectors that are part of the current coalition), social services are gradually decreasing. On top of it, the "free market competition" ideology that the current Prime-Minister imported from the USA in its extreme form, does not seem to work here. The reason is that the country is too small to have real competition, thus in each branch of the economy (banking, communication, food...) there are a small number of players who realize that it will be good for all to divide the market and maximize the revenues from its market share, and not really compete. The Cartel oriented market and high sales taxes in various forms results in the fact that the prices of everything in Israel are now more expensive than almost anywhere else in the universe. Those who suffer most are the young people that try now to establish themselves, and discover that even with relatively high salaries, they get to financial difficulties, this is the population that started the protest, and this is the root cause of the current protest. The required solution -- total change in the priorities and returning to the socially-based country. Is it going to happen under the most right-wing government ever? Will this become a social revolution or just phase out? we'll have to wait and see.
A footnote: the entire protest started within Facebook, which still serves as a major vehicle for coordination.
This picture is taken from the first slide of the presentation given by Richard Tibbetts last week in Oreill'y OSCON Open Source Convention. The presentation is now available on the web.
In the presentation Richard brings five lessons for the implementation and use of visual programming from Streambase experience. Note that the visual programming in this case is intended for developers, building visual programming for domain experts is a different saga.
In my event processing course, students have a choice to do projects in various available platforms (including implementing everything hard-coded), I found that many of the students prefer this style of visual programming, but some prefer to work with text-oriented programming that looks more like the programming they are used to, claiming they can have better control that way, I guess from the same reason that there are people that still prefer to drive manual cars. Personally I think that visual programming is the way to move forward, but we still have a challenge to do visual programming for semi-technical people.
This is nice picture I borrowed from a BI blog, talking about seven ways to report BI results.
Last week I have written on the fact that nowadays we see various sources who classify event processing as a kind of analytics, Gartner, in a recent report, used the term "active analytics" in a report classifying event processing as analytics. Jeff Wooton from Sybase said in the Sybase Blog that I am pondering whether event processing is analytics, and stated his positive answer to that, After looking up the meaning of the word "pondering" (I still have some holes in my English vocabulary), I am not sure it is the right phrase to express my attitude. I have drawn an observation that people did not use to classify event processing as analytics in the past, and started to do it in the last year or so. Since I did not see a big shift in what event processing is doing at that time, then either there is a shift in how analytics is defined, or that the event processing marketing guys are riding on the analytics hype. I guess that there is a truth in both. The analytics guys are talking about event processing as part of analytics from their side.
Paul Vincent noticed that there is a recent Communication of ACM article entitled "An overview of Business Intelligence technology". Paul complains about the fact that the authors looks at event processing as equals to continuous queries, while in reality many of the models (and products) are base on other programming models such as rules (Paul has illustrated the distribution into programming models on a genealogy illustration of event processing languages). Paul is right in the sense that people from the database research community view all the universe as centered around their terminology, and are sometimes ignorant about other approaches. What makes it more interest is that one of the CACM article authors is Umesh Dayal (currently in HP Labs), Umesh (which is one of my early professional inspirations) is considered to be the father of "active databases", and the HiPaC project in CCA that he managed has coined the Event-Condition-Action architecture, and in a conversation I had with him a few years ago he thought that the ECA architecture had a lot of traction; Seems that Umesh did not keep track on the descendants of HiPac!
Anyway -- besides the complains, this is an indication that the BI guys adopted event processing to be part of BI from their side, which is somewhat extending the definition of what BI is. Interesting!
Yesterday I got an Email from LinkedIn with 77 pictures that show those of my contacts who moved from the beginning of 2011, while some of them just sent title within their organizations, most of them indeed moved to another organizations, some from senior positions in big companies to start-ups. Is it a trend now that 2011 is a year in which many people are moving?
ACM keeps sending me hard copy of the "Communication of ACM", in addition to send Email whenever the electronic copy is available. Yesterday I browsed through the June 2011 issue (it takes more than a month to get it delivered), and found out the paper entitled: "10 rules for scalable performance in simple operation's Datastores" by Mike Stonebraker and Rick Kattell. The 10 rules are summarized in the illustration below.
This is a mix of various types of advises: from - use high availability and automatic recovery, to don't try to build ACID consistency yourself, through don't be afraid to use high level languages, and even use open source. The domain of this paper are - as declared in the title - "simple operation's data stores", the question is what can we learn about scalability in event processing -- which is somewhat different -- neither focused around data stores, nor around simple operations. Also, scalability in event processing have several dimensions, not just scalability in the number of events, in fact in the DEBS 2011 tutorial we mapped all the scalability dimensions in event processing
I guess that shared-nothing architecture is always a good practice, the use of high level languages and the utilization of main memory are also good practices. Recovery is a matter of application's requirement, for some application recoverability is vital, for others - it is not really necessary. As for the use of open source, it is again depending on the context. In summary -- some of the rules are rather well known best practices, some are subjective, and some are context dependent.
BIG DATA is a hot phrase, one can also be labelled as "BIG DATA NERD" by purchasing the shirt shown above. The term BIG DATA refers to the explosion of data in the universe, especially methods to store and process huge amount of data.
Paul Vincent poses the question: CEP = real-time bid-data? Actually he doesn't answer this question on his blog posting, but the spirit of the postings makes the impression that the "=" relations is evaluated to "True".
The answer is that it is not really an equality, but more like an intersecting concepts.
Event processing of course does neither necessarily works on big data nor it necessarily has to satisfy real-time constraints.
On the other hand, real-time big data, is not necessarily event-driven and process events at all, as big data can be fairly static. As an example for real-time big data, I'll return to an invited talk in DEBS 2011 on Watson, in which there has been a live demonstration of the Jeopardy! game with the Watson computerized system, you can see in the picture that Paul Vincent is playing (very skilfully) the host.
Watson is a "big data" crunching application, it has very strict real-time constraints, so it certainly can qualify as "real-time big-data", alas, it has nothing to do with event processing, in fact the data exists prior to the game and no additional data is added during the game.
While the equality does not work, there is an obvious relation; in some cases there are substantial number of data inputs, which much of it can be reduced, or filtered out. Event processing can be used to process streaming data on the fly, and not follow the paradigm of store now process later. This is important especially one the processing is required in real-time.
More on event processing and big data -- later
Some other input related to student's work, is the posting of Sascha Retter, based on his diploma thesis (the term discloses the origin in Germany). The observation is about critical success factors of event processing systems - integration with workflow management system, and modeling tools. There is some truth in both observations, but I would like to take broader view.
There is a market for stand-alone event processing systems, but we have observed that the larger market is event processing embedded in other systems, workflow management systems, or in the more modern name "Business Process Management" (BPM) is one of these systems, but this is not the only one, there are both other systems in which event processing can be embedded inside, e.g. analytic frameworks, sensor networks and more. It may also be embedded within packaged applications - like trading platforms, supply chain management and much more.
Modeling tool are a vehicle for usability, and indeed usability and ease of use has been identified as a critical success factor. This is now a well recognized fact.
There are other critical success factors for various applications, like non-functional requirements (performance). I also believe that standards are critical success factor for the entire area.
The EPTS use case survey provides more insights into properties and preferences of event processing customers.
