Thoughts on Data Science, IT Operations Analytics, programming and other random topics

IT Operations - hot again

11 Nov 2013

A recent study highlighted a number of interesting trends in analytics. It confirmed what I already instinctively knew i.e. that a majority of the Big Data Analytics focus to date has been on those areas directly related to customer experience and management. You know, the kind of analytics which helps answer questions such as 'what products does my customer most like', 'what/who is influencing my customer', 'which customers are likely to leave me and go to a competitor any time soon'. Sometimes, this category of analytics is referred to as 'beer and diapers' analytics (see the reference here) and while the original story turns out to not be quite true, I like it as a convenient label for the kind of analytics where the focus is on data mining to understand customer behaviour - more often called 'Customer Sentiment Analysis'. What I found most interesting in the report was that after the Customer Sentiment Analysis, the next largest category were those related to improving the operations of the underlying infrastructures supporting the business.. This lines up nicely with my own area of specialization i.e. IT Operations Analytics (ITOA), and I'm pretty pumped to see this area rising in importance.

I've heard some say that the data involved in IT Operations is the original big data, and of course, depending on where you are coming from, your own sense of what was the original big data might differ. However, there's no doubt that there has always been a relatively large amount of IT operations data available in various forms (logs, events, metrics etc) and that deriving insight from it has been a critical function in maintaining those IT environments and the services riding on top of them. While there was quite a flurry of innovation in this area during the height of the last internet bubble, in my own view, it's been relatively quiet in recent years, and whatever improvements have emerged, have been generally incremental in nature. Progress such as A bit more scalability here, support for a few more data types there, and few fancier reports over here - progress, but hardly inspiring. Also noteworthy, and most relevant to this blog, despite all the progress to date, there was precious little serious 'analytics' deployed. The focus had been on collecting and managing the data, culminating in simple summary reports, aggregated over various dimensions and perspectives( services, geographies, technologies etc ) and while useful, or indeed, critical, we're not exactly talking about deep analytics here.

However, after the better part of a decade of relative stability, I am happy to report that this situation is rapidly changing and we are now at another inflection point in the history of IT management and there are fundamental and timely changes coming. Combinations of the huge increase in data driven by mobile, social, changes in Dev Ops practices, Cloud (both as envs being monitored, and as hosting environments) are all conspiring to turn the world of IT operations on its head. The original Big Data is now a lot bigger, and no end to the growth is in sight. More data and more dynamic environments are making a difficult operational management problem even more challenging - challenging to the point that we're just about run the course with the current crop of technologies.

In this space there are many emerging 'must-answer' questions analagous to those you'd find in the beer and diapers analytics. For example, instead of 'will this customer leave me any time soon', we have questions like 'will any of my services degrade to unaccepatable levels over the next N hours'. The difficulty is compounded even more because most answers have to be obtained in something much closer to 'real-time' and the luxury of offline 'batch' type analysis is not available. Instead, real-time analytics are the order of the day in this ITOA space.

It is to help answer these kinds of questions in larger, more dynamic environments, that the newest crop of IT Operation Analytics tools are being developed and deployed. Many of these tools are still in the early stages, building out from areas of previous expertise, developing first more advanced analytics in those familiar areas, this tool focused on Log Analytics, that one focused on Metric Analytics, this other one dealing with Event analytics, each with an intent to broaden scope to include data from related domains. However, we've made a start, and over time abilities to work with the wider sets of data, drawing deeper insights from the analytics applied across a wide of a wide range of the data will be key to providing answers to the critical questions. Of course, they must do this, all the while effectively keeping up with the requirements of scale and real-time responsiveness implied by the target environments. We're building upon decades of data collection and management techniques. We have the data, and we're only beginning to explore what we we can learn from and do with this data.

@ IBM, I've been working on developing some of the tools/products mentioned here Achieving Actionable Insights from IT Operations Big Data. There have been a number of exciting announcements in this area over the last month or so, in the areas of Predictive and Search analytics. Check them (and of course, the competitor products) out and see how much new capability is being created in this space.

ITOA has emerged from a decade long hibernation, and is again a hotbed of innovation and change.

comments powered by Disqus