Thoughts on Data Science, IT Operations Analytics, programming and other random topics

Combine perspectives to extract more value from your IT data

16 Dec 2014

(first published on IBM Service Management 360 site)

Over the years we have derived plenty of value from IT operational data. Unfortunately, we’ve also left a significant amount of potential value behind, either because it’s been impractical or we haven’t had the technology to get at it. Perhaps, even more interesting, is the potential value that exists within the data that is not yet known.

The great news is now, with emerging new tools and techniques for extracting and using big data, such as IBM InfoSphere BigInsights, which has a broad analytic platform focus, and IBM SmartCloud Analytics – Predictive Insights, with its vertical anomaly detection focus, we will be able to unlock those next levels of insight and, from there, uncover the value.

Organizations that rely heavily on IT for a competitive advantage can easily generate terabytes of IT operational data each day. There is a wide variety of data types such as events, logs, performance metrics, configuration changes, trouble tickets and so on, that are collected, stored in data warehouses and analyzed. We extract what we can and then eventually delete that data, often due to limited storage and processing capacity constraints. Who knows the value of the data we’re throwing away?

While it is true that the value of this data declines rapidly with age (do you really care about that month-old CPU utilization data?), it is far from worthless; there’s plenty of residual value there (think of this as a sort of “long-tail”). Using big data techniques, it becomes more practical to analyze both historical and more recent data, turning it into powerful intelligence we can use to improve day-to-day operations.

Big data experts often compare extracting value from data with that of refining oil. “Data is the new oil,” they say. Oil in the ground has limited value; its true value is not recognized until it is extracted and refined. This is similarly true of IT data, although there is an added complexity when it comes to refining it. IT data is far from uniform and it’s often locked up in various organizational domains. The analogy also works when you think of marginal oil fields, which are not yet economically viable. Typically the data, which exists in different forms, is collected by a myriad of tools and held (or guarded) in multiple silos within the organization.

Data from different kinds of IT operations provides its own unique perspective on system behavior. Performance metrics can provide key insights into how an application or system is responding, while log files are essential to understanding what errors are causing a slow down or service degradation. Similarly, alarms, events and trouble tickets provide critical data sets enabling organizations to manage, identify and remediate IT-related issues. Analyzing each of these perspectives in isolation provides value; this, after all, is what we’ve been doing for years.

However, one of the next big opportunities to get at the value locked in the data will come from combining these different perspectives to gain a deeper understanding of the IT environment as a whole. This combination can happen in both the spatial (looking across the environment) and the temporal (across time) realms.

Access to a truly holistic view of the IT environment is rare. Organizations are used to dealing with segmented views of their data, primarily using email to share perspectives. Merging these perspectives is generally a human-powered activity. Producing such combined holistic views in real time, on huge data sets, while providing convenient facilities to navigate between them, will be key to communicating insights. Using the available data to determine which aspects of a system’s behaviors truly affect a customer’s experience has huge benefits and can lead to significant improvements in that experience.

These new big data capabilities could not have come soon enough. The environments are significantly more dynamic and highly interdependent. A problem in one area can manifest elsewhere, much like a drop in a pond ripples across its environment. It is increasingly difficult to keep on top of things, and clearly old processes are being strained.

IT failures and slowdowns massively impact revenue, brand image and customer retention. IT operations data is an even more critical resource to assure service delivery. Much like oil, the potential from IT data is almost limitless. We collect it and store it. Let’s also refine it to a higher grade, work those marginal fields, and look at it from many perspectives. The insights gained will enable better service, proactive outage prevention and happy customers.

comments powered by Disqus