Grok by Numenta - real-time pattern discovery

Thanks to Jeff Adkins,  I have looked at the website of a company called "Numenta" which has a seemingly interesting product called Grok.  As a science fiction fan, I like words from science fiction books that made their way into the language, and Heinlien's word from the immortal book "stranger in a strange land" is one of them.  

The Grok product by  Numenta,  illustrated by this figure -

is described as a tool that discovers in real-time patterns in events (data streams) and generate predictions and anomalies detection. The technology behind it is described to emulate the human brain and belongs to the neural nets family.  There is a white paper on the website explaining it.  

The site describes the types of created patterns as: temporal, spatial, and spatiotemporal.  However, their use of the term "spatial" is non conventional in the sense, that it does not have any necessary relationship to location, but is defined as "relationships between things that happen at the same time", which in the examples relate to relations between attributes of the same event (e.g. the relationship between age, gender and income to loan amount).  Calling this relation "spatial pattern" is kind of confusing to me.

Other than that -- seems interesting, I will be curious to get more information about real-life experience of this technology. 

Some blog postings for year end

We don't really celebrate Christmas here, but it is the end of the year, and we need to finish our vacation days, and since our colleagues from overseas are now in vacation,  it is a good time to do it, so it is a good time to do catch up on things that are not part of the regular work.   Looking at some recent Blogs,  I borrowed the above picture from Scot Fingerhut's Blog entitled:  " How Santa Uses CEP - Elf Productivity, Real-time Naughty And Nice Rating" .    BTW - I found another Christmas related postings in OMG facts claiming that Jesus was not really borne in December,  

Back to the professional Blogs, my IBM Haifa Research Lab colleague Yishai Feldman started a Blog entitled "The Dusty Deck"  with punched card on the top -  I belong to the generation who really punched cards, I guess that today punched card belongs to museums along with typewriter and sliding rule. 

Yishai is a local software engineering guru,  who deals with analysis of programs, so I'll follow his Blog with interest.

More in the Blogland,;

The Prova Blog reports on a new version of the Prova open source event processing system that was released this week. 

Colin Clark brings us predictions about 2011.   About event processing he predicts that it will become part of a larger horizontal offering.  I think it is already happening,  as I stated before there is still a niche for the "stand-alone" event processing,  the major market is in having event processing capabilities pervasive over computing offerings.

Last but not least, Rainer von Ammon summarized the workshop he organized recently in conjunction with ServiceWare 2010.   

OMG event processing symposium - replay is available now

For those who missed the original broadcast of the event processing symposium - a replay is now available. My presentation entitled "event processing - seven years from now" (Brenda Michelson asked me to talk about "event processing in 2015", but I thought that 2017 is safer...) is now on Slideshare. I am just listening to myself now, it is quite strange to hear your own voice and realize your English accent is somewhat strange. Enjoy!

Review of Mark Palmer's event processing predictions for 2010

As an skeptic person, I have never really believed in astrology, but reading it from time to time, and somehow there is some correlation between years in which I had major changed in life, to years in which the astrology has predicted it, but this is not a scientific evidence, of course.

Anyway, the current prediction about this year came from Mark Palmer, the dynamic CEO of Streambase, who made nine predictions. Of course, his predictions were made from the viewpoint of his current role, but I thought that there is some value to review it. I'll say a few comments on them, one by one.

Prediction #1: 100+ Event Processing Applications Per Firm Milestone will be Passed.

This is something that I would like to see, and if will happen it will show that event processing is really part of the main stream computing. Personally I don't know any organization that is even close to 100+ applications, but I guess that Mark has somebody in mind. It will be even more interesting, if the 100+ applications will not be just thousand islands, but will also be connected, use heterogeneous event processing platforms...

Prediction #2: The "Deluge of Real-Time Data" Will Not Drive Event Processing Adoption

A refreshing view when it comes from Streambase which in the past over-hyped the mythical "event per second",

I totally agree, the main value of event processing seems to be the agility and reduction of total cost of ownership.

Prediction #3: Cognitive Physics Will Be as Important as Computing Physics

This is a challenging one, I was not familiar with the term cognitive physics, but with the help of all mighty Google, I found something about it on the Web. Anyway, Mark means that the way we communicate with computerized systems should align better with the way we think, getting to higher level abstractions, and consequently again to agility to implement new stuff. I totally agree that higher level languages is one of the most important directions for event processing, as a matter of fact, this is one of my major interest areas.

Prediction #4: Quantitative Thinking will Trump Traditional Thinking

This again has to do with higher level way to express quantitative thinking. Event processing is just one of various techniques to express quantitative thinking, the rest are - business rules and various analytics. Decision platforms will be combined of all. Event processing is certainly a player here, but not the only one.

Prediction #5: Software Stacks will Continue to Miss the Mark

There are systems in which event processing can be stand-alone island, but in many cases the producers and consumers for the events are coming from applications that have already been developed using the software stacks, thus, there will be a benefit to have good integration of event processing with these software stacks. As mark said customers are interested in solution to their business needs and not in technology, in some of the event processing systems I've seen, the main investment was not in the event processing system itself, but in its connectivity with the various producers and consumers. I agree that it is not cost-effective to buy an entire stack only to use event processing, but the same organizations that will have 100+ applications of event processing (prediction #1) are also those who already have BPM/messaging/application servers and other type of middleware and they are looking for good integration story, so this is not a black-or-white situation.

Prediction #6: A New Event-Based Software Stack will Emerge

The main claim here is that alternative stacks will emerge as combination of innovative technologies that comes from small companies. I've heard this idea before, in order to make it happen, more interoperability standards are needed, since the different parts of the stack need to talk to one another. I guess these standards will come, but from my limited experience, standards is a slow world.

Prediction #7: Stream-Based Platforms will Continue to Lead a Siege Against the $15B Data-Base Market

I am not sure what is the meaning of stream-based platforms, since the first predictions were made about event processing, and the body of this prediction also talks about event processing. Anyway, I agree that event processing is positioned as complementary to the business intelligence and general DBMS market. Analysts also claim that BI tools will have event processing capabilities (another software stack!), but still have some way to go until becoming main-stream there.

Prediction #8: Open Source CEP Won't Impact the Market, But Open Source Will

This is probably true for 2010, but might not be true for the longer future. When the event processing area will mature more and have its own standards, the event processing instance of MySQL may emerge.

Prediction #9: Event processing will Yield a Great New Software Powerhouse

Sometimes the emergence of new market can grow a software vendor from start-up to a medium or large software vendor, this has happened before, it might happen again - if indeed event processing will become a multi-billion business, I guess that part of it will be done by the current super powers in the software world, but some can go to growing companies. This will probably not happen in 2010, but I wish Mark success in his aspirations. Thinking big is a good practice, and a necessary but not sufficient condition for success.

Live from DEBS 2009 - Keynote Address by John Bates

In my role as the industrial track chair of DEBS 2009, I have been asked to invite an industrial keynote speaker, and this year I chose John Bates, the General Manager of the Apama Division of Progress Software, and a person that made a transition from being a faculty member in Cambridge University, leading one of the first start-ups in this area, and now managing a division in a larger company. This was certainly a good choice; John outlined the assumptions he had as a researcher, and the confrontation between these assumptions and the reality that forced them to find a low hanging fruit, until the bigger market will be mature enough for the ideas for event processing, and they found trading in capital marketing as this low hanging fruit, and the early adopters of such technologies. However, the maturity in thinking about this area now takes it to more areas. As for John's prediction for what will happen in the event processing market in the next five years, he mentioned the following points:

• Event Processing will be used for tracking everything -- car, plane, bag, package, ship

• EDA -- will achieve wider adoption -- federating services in the enterprise, providing agile enterprise nervous system; The next wave of SOA.

• Event processing everywhere -- EP in the cloud.

• Event Processing will be part of bigger event-driven business process management market - BPM, rules, event processing are going to merge.

• Event processing will be embedded in many vertical applications.

More about further sessions in DEBS 2009 - later