Data and Analytics Strategies Need More-Concrete Metrics of Success
Data and Analytics Strategies Need More-Concrete Metrics of Success
Data and Analytics Strategies Need More-Concrete Metrics of Success
Analyst(s): Frank Buytendijk, Ankush Jain, Thomas Oestreich, Alan D. Duncan, Michael Smith
"Better decision making" and "single version of the truth" no longer serve to
justify data and analytics investments. CDOs and other data and analytics
leaders need to create concrete, measurable metrics that link data and
analytics initiatives to information, business and stakeholder value.
FOUNDATIONAL DOCUMENT
This research is reviewed periodically for accuracy. Last reviewed on 31 December 2018.
Key Challenges
■ Only 15% of Gartner-reviewed data and analytics strategies contain concrete metrics of
success, despite the trend in business being to demand tangible measures of success from
data and analytics initiatives.
■ With an increase in business management attention to data as a true business asset, data and
analytics professionals will be called on to openly demonstrate and quantify the value that they
add.
Recommendations
CDOs and other data and analytics leaders:
■ Link data and analytics initiatives to three types of value: information value (improving the
information management process itself); business value (improving business processes with
data and analytics); and stakeholder value (what the data and analytics mean for stakeholders
such as customers, partners, shareholders and society at large).
■ Start data and analytics initiatives with a baseline measurement of the "as-is" state, and show a
clear path to financial and business objective contributions.
■ Use proven techniques such as A/B testing for continuous experiments on how to increase the
value of those initiatives, beyond the baseline measurement.
Table of Contents
List of Tables
List of Figures
Introduction
1
Of the data and analytics strategies that Gartner's data and analytics team reviewed in 2014 and
2015, 85% either did not contain any metrics to prove the strategy's success, had only soft
The digital world both enables and requires a more direct contribution of data and analytics to
business results.
■ Information becomes a monetizable asset in itself — a means to grow the business or a product
that can be sold itself ("Why and How to Measure the Value of Your Information Assets").
■ Sensor technology (Internet of Things [IoT]) and predictive analytics drive higher return on
investment based on real-time information on the use of products, often for the reason of
predictive asset maintenance — something that cannot be done without information
governance (see "The Identity of Things for the Internet of Things").
■ Product and service personalization also require a combination of product and customer master
data management (MDM).
■ Algorithms, trained by seas of data, take autonomous action — for example, in financial trading,
patient monitoring, manufacturing control, customer service and support, fraud detection,
advertising, and so on (see "Top 10 Things CIOs and CDOs Need to Know About Algorithmic
Business").
These business themes make it more realistic — and imperative — to connect data and analytics
strategies to concrete business results. Qualitative measures need to be replaced with more-
tangible measures of business outcomes (see "The Gartner Business Value Model: A Framework for
Measuring Business Performance").
Our review showed that the most common qualitative goals in data and analytics strategies are:
1. A description
2. Examples for various types of business outcomes. As laid out in "Measure Your Information
Yield to Maximize Return on Information and Analytics Investments," there is value in:
Analysis
Turn "Better Decisions" and "Right Information" to User Satisfaction, Forecast
Accuracy and Stakeholder Value
Description: Decision support is one of the most important use cases for data and analytics. In fact,
it has for a long time been the reason for why business intelligence initiatives exist. It has historically
been perceived as hard to create a tangible business case for. How can you test the business
outcome of a decision without the availability of a parallel universe in which you took the other
decision? How do you measure that you provide the right information to the right people at the right
time? "Supporting making better decisions" and "providing the right information at the right time to
the right people" are related goals. Making better decisions requires the right information — there is
a cause-and-effect relationship.
Information value: How can you know that you built the right reports or information streams? Most
tools have extensive built-in reporting capabilities on the use of the tool itself.
■ The right information — Measure this by looking at which reports are often used and which
ones not. This can also be offset by the times a user opens a standard report versus engages in
ad hoc reporting. This is a first approximate measure of the relevance of the information itself.
■ The right person — Build an understanding of this by tracking which reports are being
forwarded to which other people. Once these reports are shared with that person routinely, you
can again track whether these reports are regularly used as well.
■ The right moment — Monitor the time between the reports being published and the data being
used. Reports not being used at all can indicate that the information is either irrelevant or simply
too late. Reports consistently being used but with a consistent time lag may indicate that the
information is too early, or that there is a process problem that creates a delay in getting the
information to the right analyst as quickly as possible.
Other measurement instruments often used are user surveys and service-level agreements (SLAs).
You can simply ask the users whether they feel they get all the information they need, whether it is
timely, and whether they feel others need that information too. Regarding service-level
management, you need to track the metrics (traditionally, mostly on data accuracy, timeliness and
cost) in the SLA. Meeting these metrics means delivering information value.
■ On the operational level, better decision making translates into higher conversion rates, better
supply chain replenishment, lower fraud levels and so on.
■ On the tactical level, decision making is about planning. Measurable outcomes include forecast
accuracy or using new techniques such as prediction markets.
■ On the strategic level, data and analytics help decision making by reducing uncertainty — for
instance, by keeping your options open. Options can be valued.
Stakeholder value: There is a link between shareholder value and making better decisions through
better forecasting, reducing uncertainty and executing well on operational metrics. Shareholder
value is based on expected future results, and transparent, trustworthy and reliable management is
one of the most important indicators.
So if you break down "supporting making better decisions" into the operational, tactical and
strategic levels, it becomes possible to directly link better data and analytics to shareholder value.
■ Customer value (expressed in terms of their metrics of information value or business value), or in a
proxy such as NPS
Gartner advises clients to steer away from "single version of the truth" as a goal for data and
analytics. With the growth of external data sources and multistructured data, the trend clearly points
toward many different versions of the data. With external and multistructured data, the concept of
truth should be traded for the idea of trust (see "Big Data Governance From Truth to Trust").
Information/business/stakeholder value: Treat the single version of the truth not as a goal, but as
something that complicates reaching business outcomes. Whenever a business goal cannot be
attained or is not reached to its full potential because of issues with multiple versions of the truth,
that particular problem must be fixed.
Translate "Better Alignment" Into First Time Right and Cost of Doing Business
Description: When is an organization aligned? When all independent, dependent and
interdependent activities by various business functions come to a single result in the right way, in
the right order, deliver the anticipated result, and do not require unnecessary overhead from
additional coordination. For example:
■ A telecom company offering Internet and TV plus landline and mobile telephony in a single
combination offer.
■ A construction company remodeling a house and handling multiple disciplines (your author is
about to have that experience).
■ A public-sector agency going through the steps of providing all permits for developing a new
area.
■ 2
In the world of digital: Any business moment consisting of multiple parties jointly offering a
composite product or service to embrace an immediate opportunity (see "Digital Businesses
Will Compete and Seek Opportunity in the Span of a Moment").
MDM and metadata management are the most appropriate elements of information management to
improve that alignment. If the metrics show progress during A/B testing or after implementing
improvements in MDM and metadata, this is the contribution of the data and the analytics.
Information value: With multiple versions of the truth, value lies not in creating a single version but in
connecting them. Do this through MDM and metadata management. Think of:
■ The percentage of sources with aligned master data (weighted based on each system's size
and impact)
■ The percentage of system business terms covered by the information catalog
Business value: It is fairly easy to measure results with this way of looking at alignment — count the
percentage of cases, transactions, business moments and so on that fall within the boundaries of
the benchmark. In modern business environments, these metrics go way beyond the borders of
Typically, any high-performing help desk will be able to show these measures, as help desk
processes are structured in this way.
Stakeholder value: Alignment means more than being internally organized. It also means being
aligned with the interests of stakeholders.
Taking customers as a main stakeholder: Is it easy to do business with you? "Transaction cost
economics" is the discipline that helps manage and measure this. Think of:
■ The cost of doing business (the transaction cost — capital cost, management overhead and so
on)
■ The risk of doing business (switching cost because of lock-in, adjustments for risk, cost of
considering alternatives)
Table 2. Metrics for Alignment
Information value: As with business analytics tools that have statistics on the usage of reports, data
quality tools typically come with a range of reports on data quality. Think of the numbers of records
rejected or data quality warnings as part of a total dataset.
Business value: Data quality affects most current business trends. Consider the following scenarios:
■ Data quality issues in product and service personalization lead to failure of "first time right."
Wrong personalization leads to reworking and suboptimal service or product performance until
then.
■ Automated algorithms are often trained by data. Data quality issues lead to the algorithm not
fulfilling its business goals in part, or at all in very dysfunctional cases.
■ In the case of business moments, the metric is "lost opportunity."
■ In many industries, regulatory pressure continues to increase. Noncompliance can lead to fines.
In general, the effects of data quality on business processes can be estimated based on the Six
Sigma methodology (see "Measuring the Business Value of Data Quality"). With an average
business process maintaining a sigma value of 3.5, the average cost of quality is 20% of the overall
business process cost. A root cause analysis determines which defects are based on data quality
issues.
Bad data quality has a negative impact on any business performance indicator. Use the sensitivity
analysis in "Toolkit: Monetizing the Outcomes in the Business Value Model" for further
measurement.
Stakeholder value: Stakeholder value depends highly on trust. Has the data proven to be useful in
the past? Have others found the data useful? Can the data be triangulated with other data sources
that point in the same direction? If there is no trust in the data, there is no perceived value. This has
a measurable impact on data monetization initiatives, for instance, and this impact is
nonproportional — data quality issues in a small subset of data could lead to the value of the overall
dataset dropping to zero.
Another established way for stakeholders to express their trust is with a Net Promotor Score (NPS).
NPS describes customer loyalty based on a single question: How likely is it that you will
recommend a product or a company to others?
Information ■ Records rejected or flagged (such as data consistency issues on zip code, address, dial codes and
other attributes — %)
■ Cost of rework/correction
■ NPS
Information value: If reality changes, information about that reality must change as well. Measuring
agility starts with measuring how agile the information management process itself is. How quickly
and easily can you connect to new data sources? Or fix data quality issues? Or reach a new stable
situation of information supply, providing the right information to the right people at the right time —
after staff changes or reorganization, for example?
Business value: A process is only as agile as its least-agile part. If the information management
process doesn't change, the business process can't change either. Agility can be measured in terms
of how fast and easy it is to:
Stakeholder value: Even the most agile business will fail if it doesn't serve a stakeholder need, and if
the value of agility isn't recognized. The most concrete way of recognizing value for a customer is
by buying products and services. What percentage of revenue comes from products and services
■ Time to market
Stakeholder ■ Sales value of products and services with new data components
Case Study
Consider the example of an insurance company that launches a new product innovation. It hands
out fitness trackers to health insurance customers. Based on their number of steps per day,
customers can earn points and rewards. Moreover, through an app they can connect to peers,
exchanging tips on how to be more mobile or setting challenges.
5
Figure 1 is a cause-and-effect chart. It shows information value, business value and stakeholder
value, displaying how to create and link meaningful and concrete metrics.
Creating better alignment starts with accurate metadata/MDM. These support better underwriting
and improve shareholder value. They also support creating new, real-time business moments —
providing the right information at the right time to the right people (consumers). This leads to more
connections, higher loyalty and better health.
Improving agility is translated into time to market for new insights, such as new analytical outcomes
from the data. This positively affects decision making around underwriting and pricing, as well as
subsequent effects. It also leads to new features in the program and attracting new customers, as
well as subsequent effects. It supports all cause-and-effect chains in the initiative.
Fixing data quality leads to being able to support more relevant business moments and being on the
mark in more cases, as well as subsequent effects. It also supports the time to market for new
insights (less time in fixing data problems), higher usage of the insights (the data is trusted), and
better decision making about underwriting, as well as all subsequent effects, improving all cause-
and-effect chains across the initiative.
"How CIOs and CDOs Can Use Infonomics to Identify, Justify and Fund Initiatives"
"Toolkit: Selecting the Right Business Value for Your Organization's BI Initiatives"
"Start Your Data and Analytics Strategy With a Clear Value Proposition"
Evidence
Gartner takes hundreds of inquiries on information management every year. For this research, we
focused on the so-called "document reviews," where we review a data and analytics strategy
document. The data and analytics research community did a total of 106 strategy document reviews
between January 2014 and December 2015. These documents were called "strategy," "plan" or
"roadmap," and they focused on "data," "information" or "big data."
2A business moment is a transient opportunity in the digital world where people, things and
businesses connect for a brief moment.
3 Althoughit should also be said that in many cases the consistency of data is more important than
whether the quality is high or not, a consistent mistake in the data still provides the right direction.
5 Thiscause-and-effect chart is based on a concept called "strategy maps," which is part of the
balanced scorecard methodology.
For each business function, it is a straightforward process to come up with clear business
outcomes.
On the tactical level, many decisions are about planning. The most straightforward way of
measuring better planning is in the accuracy:
■ Linking predictive analytics based on actual patterns to a planning process helps to create an
adaptive planning process that is based on trends in reality, instead of an internal negotiation
process.
Even on the strategic level is it possible to create tangible measures? The answer is in uncertainty
reduction. The more uncertain a decision is, the more you need to keep your options open versus
taking a clear decision and executing on it (see "The Four Pillars of an Options-Based Information
Strategy" and "Rethinking Information Management for Bimodal IT"). Options-based strategies are
more expensive. The difference between keeping your options open and implementing a clear
decision can be calculated (refer to real options). This difference can then be compared with the
cost of additional information that reduces the uncertainty of the decision, and hence the need for
additional options. If the cost of the additional information is lower than the option cost, the
information has positive value.
Corporate Headquarters
56 Top Gallant Road
Stamford, CT 06902-7700
USA
+1 203 964 0096
Regional Headquarters
AUSTRALIA
BRAZIL
JAPAN
UNITED KINGDOM
© 2016 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates. This
publication may not be reproduced or distributed in any form without Gartner's prior written permission. It consists of the opinions of
Gartner's research organization, which should not be construed as statements of fact. While the information contained in this publication
has been obtained from sources believed to be reliable, Gartner disclaims all warranties as to the accuracy, completeness or adequacy of
such information. Although Gartner research may address legal and financial issues, Gartner does not provide legal or investment advice
and its research should not be construed or used as such. Your access and use of this publication are governed by Gartner Usage Policy.
Gartner prides itself on its reputation for independence and objectivity. Its research is produced independently by its research
organization without input or influence from any third party. For further information, see "Guiding Principles on Independence and
Objectivity."