Data and Analytics Strategies Need More-Concrete Metrics of Success

Download as pdf or txt
Download as pdf or txt
You are on page 1of 15
At a glance
Powered by AI
The key takeaways are that only 15% of strategies contain concrete metrics of success and data professionals will need to openly demonstrate and quantify the value they add with increased business attention on data as an asset.

Some challenges faced are that only 15% of strategies contain concrete metrics of success and data professionals will need to openly demonstrate and quantify the value they add with increased business attention on data as an asset.

Data and analytics professionals can demonstrate and quantify the value they add by linking initiatives to information, business and stakeholder value and starting initiatives with baseline measurements to show contributions to financial and business objectives.

Data and Analytics Strategies Need More-

Concrete Metrics of Success


FOUNDATIONAL Refreshed: 31 December 2018 | Published: 11 May 2016 ID: G00297356

Analyst(s): Frank Buytendijk, Ankush Jain, Thomas Oestreich, Alan D. Duncan, Michael Smith

"Better decision making" and "single version of the truth" no longer serve to
justify data and analytics investments. CDOs and other data and analytics
leaders need to create concrete, measurable metrics that link data and
analytics initiatives to information, business and stakeholder value.

FOUNDATIONAL DOCUMENT
This research is reviewed periodically for accuracy. Last reviewed on 31 December 2018.

Key Challenges
■ Only 15% of Gartner-reviewed data and analytics strategies contain concrete metrics of
success, despite the trend in business being to demand tangible measures of success from
data and analytics initiatives.
■ With an increase in business management attention to data as a true business asset, data and
analytics professionals will be called on to openly demonstrate and quantify the value that they
add.

Recommendations
CDOs and other data and analytics leaders:

■ Link data and analytics initiatives to three types of value: information value (improving the
information management process itself); business value (improving business processes with
data and analytics); and stakeholder value (what the data and analytics mean for stakeholders
such as customers, partners, shareholders and society at large).
■ Start data and analytics initiatives with a baseline measurement of the "as-is" state, and show a
clear path to financial and business objective contributions.
■ Use proven techniques such as A/B testing for continuous experiments on how to increase the
value of those initiatives, beyond the baseline measurement.

Table of Contents

Strategic Planning Assumption............................................................................................................... 2


Introduction............................................................................................................................................ 2
Analysis.................................................................................................................................................. 4
Turn "Better Decisions" and "Right Information" to User Satisfaction, Forecast Accuracy and
Stakeholder Value.............................................................................................................................4
Avoid "Single Version of the Truth".................................................................................................... 5
Translate "Better Alignment" Into First Time Right and Cost of Doing Business................................. 6
Express "Data Quality" in Terms of Six Sigma and Data Monetization................................................8
Measure "Agility" as Time to Market/Resolution and Cost of Change................................................9
Case Study.......................................................................................................................................... 10
Gartner Recommended Reading.......................................................................................................... 12

List of Tables

Table 1. Metrics for Better Decisions and Right Information.................................................................... 5


Table 2. Metrics for Alignment.................................................................................................................7
Table 3. Metrics for Data Quality............................................................................................................. 9
Table 4. Metrics for Agility..................................................................................................................... 10

List of Figures

Figure 1. Creating and Linking Meaningful Metrics................................................................................ 11

Strategic Planning Assumption


Through 2019, 90% of large organizations will have hired a chief data officer (CDO); of these, only
50% will be hailed a success.

Introduction
1
Of the data and analytics strategies that Gartner's data and analytics team reviewed in 2014 and
2015, 85% either did not contain any metrics to prove the strategy's success, had only soft

Page 2 of 15 Gartner, Inc. | G00297356


measures (less, more, higher, lower, better, faster and so on) or mentioned only very high-level
qualitative goals (see Note 1). This means that only 15% had concrete performance indicators for
the proposed strategy.

The digital world both enables and requires a more direct contribution of data and analytics to
business results.

■ Information becomes a monetizable asset in itself — a means to grow the business or a product
that can be sold itself ("Why and How to Measure the Value of Your Information Assets").
■ Sensor technology (Internet of Things [IoT]) and predictive analytics drive higher return on
investment based on real-time information on the use of products, often for the reason of
predictive asset maintenance — something that cannot be done without information
governance (see "The Identity of Things for the Internet of Things").
■ Product and service personalization also require a combination of product and customer master
data management (MDM).
■ Algorithms, trained by seas of data, take autonomous action — for example, in financial trading,
patient monitoring, manufacturing control, customer service and support, fraud detection,
advertising, and so on (see "Top 10 Things CIOs and CDOs Need to Know About Algorithmic
Business").

These business themes make it more realistic — and imperative — to connect data and analytics
strategies to concrete business results. Qualitative measures need to be replaced with more-
tangible measures of business outcomes (see "The Gartner Business Value Model: A Framework for
Measuring Business Performance").

Structure of This Research

Our review showed that the most common qualitative goals in data and analytics strategies are:

■ Supporting making better decisions


■ Providing the right information at the right time to the right people
■ Reaching a single version of the truth
■ Creating better alignment
■ Fixing data quality
■ Improving agility

For each of these qualitative goals, we will provide:

1. A description
2. Examples for various types of business outcomes. As laid out in "Measure Your Information
Yield to Maximize Return on Information and Analytics Investments," there is value in:

Gartner, Inc. | G00297356 Page 3 of 15


■ Improving the information management process itself (information value)
■ Improving the business process (business value)
■ Addressing the needs of customers, shareholders and other stakeholders (stakeholder
value)
3. Some examples of more tangible metrics you could create

Analysis
Turn "Better Decisions" and "Right Information" to User Satisfaction, Forecast
Accuracy and Stakeholder Value
Description: Decision support is one of the most important use cases for data and analytics. In fact,
it has for a long time been the reason for why business intelligence initiatives exist. It has historically
been perceived as hard to create a tangible business case for. How can you test the business
outcome of a decision without the availability of a parallel universe in which you took the other
decision? How do you measure that you provide the right information to the right people at the right
time? "Supporting making better decisions" and "providing the right information at the right time to
the right people" are related goals. Making better decisions requires the right information — there is
a cause-and-effect relationship.

Information value: How can you know that you built the right reports or information streams? Most
tools have extensive built-in reporting capabilities on the use of the tool itself.

■ The right information — Measure this by looking at which reports are often used and which
ones not. This can also be offset by the times a user opens a standard report versus engages in
ad hoc reporting. This is a first approximate measure of the relevance of the information itself.
■ The right person — Build an understanding of this by tracking which reports are being
forwarded to which other people. Once these reports are shared with that person routinely, you
can again track whether these reports are regularly used as well.
■ The right moment — Monitor the time between the reports being published and the data being
used. Reports not being used at all can indicate that the information is either irrelevant or simply
too late. Reports consistently being used but with a consistent time lag may indicate that the
information is too early, or that there is a process problem that creates a delay in getting the
information to the right analyst as quickly as possible.

Other measurement instruments often used are user surveys and service-level agreements (SLAs).
You can simply ask the users whether they feel they get all the information they need, whether it is
timely, and whether they feel others need that information too. Regarding service-level
management, you need to track the metrics (traditionally, mostly on data accuracy, timeliness and
cost) in the SLA. Meeting these metrics means delivering information value.

Page 4 of 15 Gartner, Inc. | G00297356


Business value: Not all types of decision making are the same; there are operational, tactical and
strategic decisions (see Note 2 for a more elaborate description).

■ On the operational level, better decision making translates into higher conversion rates, better
supply chain replenishment, lower fraud levels and so on.
■ On the tactical level, decision making is about planning. Measurable outcomes include forecast
accuracy or using new techniques such as prediction markets.
■ On the strategic level, data and analytics help decision making by reducing uncertainty — for
instance, by keeping your options open. Options can be valued.

Stakeholder value: There is a link between shareholder value and making better decisions through
better forecasting, reducing uncertainty and executing well on operational metrics. Shareholder
value is based on expected future results, and transparent, trustworthy and reliable management is
one of the most important indicators.

So if you break down "supporting making better decisions" into the operational, tactical and
strategic levels, it becomes possible to directly link better data and analytics to shareholder value.

Table 1. Metrics for Better Decisions and Right Information

Value Example Metrics

Information ■ Reports being used periodically (%)

■ Reports being used by multiple people (%)

■ User survey outcomes

Business ■ Uncertainty reduction vs. cost of options

■ Forecast accuracy (%)

■ Conversion rate (%)

Stakeholder ■ Shareholder value

■ Customer value (expressed in terms of their metrics of information value or business value), or in a
proxy such as NPS

■ Societal value (such as energy consumption, health, safety etc.)

NPS = Net Promotor Score

Source: Gartner (May 2016)

Avoid "Single Version of the Truth"


Description: This is the most misleading of the qualitative goals for data and analytics initiatives.
Because it intuitively feels like the right thing to do, it feels like an important goal to achieve and has

Gartner, Inc. | G00297356 Page 5 of 15


thus been one of the most common. Yet, at the same time, it has proven to be hard to connect
having a single version of the truth to a specific business outcome.

Gartner advises clients to steer away from "single version of the truth" as a goal for data and
analytics. With the growth of external data sources and multistructured data, the trend clearly points
toward many different versions of the data. With external and multistructured data, the concept of
truth should be traded for the idea of trust (see "Big Data Governance From Truth to Trust").

Information/business/stakeholder value: Treat the single version of the truth not as a goal, but as
something that complicates reaching business outcomes. Whenever a business goal cannot be
attained or is not reached to its full potential because of issues with multiple versions of the truth,
that particular problem must be fixed.

Translate "Better Alignment" Into First Time Right and Cost of Doing Business
Description: When is an organization aligned? When all independent, dependent and
interdependent activities by various business functions come to a single result in the right way, in
the right order, deliver the anticipated result, and do not require unnecessary overhead from
additional coordination. For example:

■ A telecom company offering Internet and TV plus landline and mobile telephony in a single
combination offer.
■ A construction company remodeling a house and handling multiple disciplines (your author is
about to have that experience).
■ A public-sector agency going through the steps of providing all permits for developing a new
area.
■ 2
In the world of digital: Any business moment consisting of multiple parties jointly offering a
composite product or service to embrace an immediate opportunity (see "Digital Businesses
Will Compete and Seek Opportunity in the Span of a Moment").

MDM and metadata management are the most appropriate elements of information management to
improve that alignment. If the metrics show progress during A/B testing or after implementing
improvements in MDM and metadata, this is the contribution of the data and the analytics.

Information value: With multiple versions of the truth, value lies not in creating a single version but in
connecting them. Do this through MDM and metadata management. Think of:

■ The percentage of sources with aligned master data (weighted based on each system's size
and impact)
■ The percentage of system business terms covered by the information catalog

Business value: It is fairly easy to measure results with this way of looking at alignment — count the
percentage of cases, transactions, business moments and so on that fall within the boundaries of
the benchmark. In modern business environments, these metrics go way beyond the borders of

Page 6 of 15 Gartner, Inc. | G00297356


your own organization to include the complete value network with all partners. Concrete measures
include:

■ Percentage of "first time right" transactions


■ Percentage of transactions within established timeframes
■ Time, resources and money spent on rework
■ Time, resources and money spent on overheads and coordination
■ Percentage of transactions not leading to any complaints or support requests for the first 12
months

Typically, any high-performing help desk will be able to show these measures, as help desk
processes are structured in this way.

Stakeholder value: Alignment means more than being internally organized. It also means being
aligned with the interests of stakeholders.

Taking customers as a main stakeholder: Is it easy to do business with you? "Transaction cost
economics" is the discipline that helps manage and measure this. Think of:

■ The cost of doing business (the transaction cost — capital cost, management overhead and so
on)
■ The risk of doing business (switching cost because of lock-in, adjustments for risk, cost of
considering alternatives)
Table 2. Metrics for Alignment

Value Example Metrics

Information ■ MDM in place (%)

■ Business terms in information catalog (%)

Business ■ First time right (process, action, decision — %)

■ Interactions within established timeframes (%)

■ Time, money and resources spent on rework and overhead

■ Transactions not leading to complaints or support (%)

Stakeholder ■ Cost of doing business with you

■ Risk of doing business with you

Source: Gartner (May 2016)

Gartner, Inc. | G00297356 Page 7 of 15


Express "Data Quality" in Terms of Six Sigma and Data Monetization
Description: Linking better data quality to business outcomes should be a straightforward process,
as data quality is directly linked to process integrity. Data quality is at the basis of the effectiveness
of all other goals; if not in order, it diminishes the contribution of data and analytics to any business
3
goal. Poor data quality is a primary reason for 40% of all business initiatives failing to achieve their
4
targeted benefits.

Information value: As with business analytics tools that have statistics on the usage of reports, data
quality tools typically come with a range of reports on data quality. Think of the numbers of records
rejected or data quality warnings as part of a total dataset.

Business value: Data quality affects most current business trends. Consider the following scenarios:

■ Data quality issues in product and service personalization lead to failure of "first time right."
Wrong personalization leads to reworking and suboptimal service or product performance until
then.
■ Automated algorithms are often trained by data. Data quality issues lead to the algorithm not
fulfilling its business goals in part, or at all in very dysfunctional cases.
■ In the case of business moments, the metric is "lost opportunity."
■ In many industries, regulatory pressure continues to increase. Noncompliance can lead to fines.

In general, the effects of data quality on business processes can be estimated based on the Six
Sigma methodology (see "Measuring the Business Value of Data Quality"). With an average
business process maintaining a sigma value of 3.5, the average cost of quality is 20% of the overall
business process cost. A root cause analysis determines which defects are based on data quality
issues.

Bad data quality has a negative impact on any business performance indicator. Use the sensitivity
analysis in "Toolkit: Monetizing the Outcomes in the Business Value Model" for further
measurement.

Stakeholder value: Stakeholder value depends highly on trust. Has the data proven to be useful in
the past? Have others found the data useful? Can the data be triangulated with other data sources
that point in the same direction? If there is no trust in the data, there is no perceived value. This has
a measurable impact on data monetization initiatives, for instance, and this impact is
nonproportional — data quality issues in a small subset of data could lead to the value of the overall
dataset dropping to zero.

Another established way for stakeholders to express their trust is with a Net Promotor Score (NPS).
NPS describes customer loyalty based on a single question: How likely is it that you will
recommend a product or a company to others?

Page 8 of 15 Gartner, Inc. | G00297356


Table 3. Metrics for Data Quality

Value Example Metrics

Information ■ Records rejected or flagged (such as data consistency issues on zip code, address, dial codes and
other attributes — %)

■ Cost of reprocessing data

Business ■ First time right (%)

■ Cost of rework/correction

■ Six Sigma cost and improvement

■ Lost business moment opportunity

■ Calculated negative impact on established business metrics

Stakeholder ■ Sales value of information

■ NPS

Source: Gartner (May 2016)

Measure "Agility" as Time to Market/Resolution and Cost of Change


Description: Achieving a higher level of agility is different. Where the other goals are linked to
tangible business outcomes, agility tells you how to change the way you get to all those other goals.

Information value: If reality changes, information about that reality must change as well. Measuring
agility starts with measuring how agile the information management process itself is. How quickly
and easily can you connect to new data sources? Or fix data quality issues? Or reach a new stable
situation of information supply, providing the right information to the right people at the right time —
after staff changes or reorganization, for example?

Business value: A process is only as agile as its least-agile part. If the information management
process doesn't change, the business process can't change either. Agility can be measured in terms
of how fast and easy it is to:

■ Incorporate new sources of information in operational decision making


■ Adapt plans when the market changes
■ Model various decision alternatives in strategic decision making
■ Support new business moment configurations in real time

Stakeholder value: Even the most agile business will fail if it doesn't serve a stakeholder need, and if
the value of agility isn't recognized. The most concrete way of recognizing value for a customer is
by buying products and services. What percentage of revenue comes from products and services

Gartner, Inc. | G00297356 Page 9 of 15


that were influenced by new data components? And what percentage of customers upgrade to this
new product or service? The metrics here will be very close to traditional metrics in business
development.

Table 4. Metrics for Agility

Value Example Metrics

Information ■ Cost, time and effort of connecting to new data sources

■ Time to resolve issues

Business ■ Cost of modelling a decision alternative

■ Time to market

Stakeholder ■ Sales value of products and services with new data components

■ Users upgraded to new product or service (%)

Source: Gartner (May 2016)

Case Study
Consider the example of an insurance company that launches a new product innovation. It hands
out fitness trackers to health insurance customers. Based on their number of steps per day,
customers can earn points and rewards. Moreover, through an app they can connect to peers,
exchanging tips on how to be more mobile or setting challenges.
5
Figure 1 is a cause-and-effect chart. It shows information value, business value and stakeholder
value, displaying how to create and link meaningful and concrete metrics.

Page 10 of 15 Gartner, Inc. | G00297356


Figure 1. Creating and Linking Meaningful Metrics

Source: Gartner (May 2016)

Gartner, Inc. | G00297356 Page 11 of 15


Making better decisions and providing the right information at the right time to the right people
translate to a metric called "usage insights (%)," which improves the accuracy of the underwriting,
leading to shareholder value. It also leads to being able to create segmented and attractive pricing.
This will attract new customers, leading to more connections in the network, more value and higher
customer loyalty. More customers brings greater economies of scale, leading to more-attractive
pricing again. Being part of the program for a prolonged time should have a positive effect on
health, leading to overall lower cost for the healthcare system.

Creating better alignment starts with accurate metadata/MDM. These support better underwriting
and improve shareholder value. They also support creating new, real-time business moments —
providing the right information at the right time to the right people (consumers). This leads to more
connections, higher loyalty and better health.

Improving agility is translated into time to market for new insights, such as new analytical outcomes
from the data. This positively affects decision making around underwriting and pricing, as well as
subsequent effects. It also leads to new features in the program and attracting new customers, as
well as subsequent effects. It supports all cause-and-effect chains in the initiative.

Fixing data quality leads to being able to support more relevant business moments and being on the
mark in more cases, as well as subsequent effects. It also supports the time to market for new
insights (less time in fixing data problems), higher usage of the insights (the data is trusted), and
better decision making about underwriting, as well as all subsequent effects, improving all cause-
and-effect chains across the initiative.

Gartner Recommended Reading


Some documents may not be available as part of your current Gartner subscription.

"Seven Steps to Monetizing Your Information Assets"

"How CIOs and CDOs Can Use Infonomics to Identify, Justify and Fund Initiatives"

"How to Overcome Critical Roadblocks to Succeed as Chief Data Office"

"Toolkit: Selecting the Right Business Value for Your Organization's BI Initiatives"

"Start Your Data and Analytics Strategy With a Clear Value Proposition"

Evidence
Gartner takes hundreds of inquiries on information management every year. For this research, we
focused on the so-called "document reviews," where we review a data and analytics strategy
document. The data and analytics research community did a total of 106 strategy document reviews
between January 2014 and December 2015. These documents were called "strategy," "plan" or
"roadmap," and they focused on "data," "information" or "big data."

Page 12 of 15 Gartner, Inc. | G00297356


1 Thesedocuments can go under different names. They were also labeled plans or strategies, and
have been about information, data, analytics or big data.

2A business moment is a transient opportunity in the digital world where people, things and
businesses connect for a brief moment.

3 Althoughit should also be said that in many cases the consistency of data is more important than
whether the quality is high or not, a consistent mistake in the data still provides the right direction.

4 See "Measuring the Business Value of Data Quality."

5 Thiscause-and-effect chart is based on a concept called "strategy maps," which is part of the
balanced scorecard methodology.

Note 1 Why Business Metrics Weren't Required Before


For a long time, such concrete measures were not required, as information management was
infrastructure cost. Descriptive and diagnostic management reporting are usually seen as the cost
of doing business. Enterprise information management as a strategic initiative contributes to all
business goals but not to specific ones, making it hard to create a specific set of linked metrics.
Hence the popularity of qualitative goals around improving business in general.

Note 2 Operational, Tactical and Strategic Decision Making


Over time, decision support has become more operational in nature. In fact, in some cases,
automated algorithms take over (see "How to Get Started With Prescriptive Analytics"). In the
operational space, it is easier to measure concrete business results such as:

■ Website conversion rate, because of improved recommendations


■ Supply chain replenishment or any type of resource allocation, based on better understand of
buying patterns
■ Fraud levels, based on understanding of fraud rings
■ Investment priorities, based on an economic value-add analysis

For each business function, it is a straightforward process to come up with clear business
outcomes.

On the tactical level, many decisions are about planning. The most straightforward way of
measuring better planning is in the accuracy:

■ Linking predictive analytics based on actual patterns to a planning process helps to create an
adaptive planning process that is based on trends in reality, instead of an internal negotiation
process.

Gartner, Inc. | G00297356 Page 13 of 15


■ Adding additional data sources to the planning process — such as the weather, the price of oil,
visitor estimates of upcoming festivities, or any other external factor that affects the plan — also
helps with forecasting accuracy.
■ Prediction markets are a form of crowdsourcing that are used in political elections, but can also
be used for business planning and forecasting.

Even on the strategic level is it possible to create tangible measures? The answer is in uncertainty
reduction. The more uncertain a decision is, the more you need to keep your options open versus
taking a clear decision and executing on it (see "The Four Pillars of an Options-Based Information
Strategy" and "Rethinking Information Management for Bimodal IT"). Options-based strategies are
more expensive. The difference between keeping your options open and implementing a clear
decision can be calculated (refer to real options). This difference can then be compared with the
cost of additional information that reduces the uncertainty of the decision, and hence the need for
additional options. If the cost of the additional information is lower than the option cost, the
information has positive value.

More on This Topic


This is part of two in-depth collections of research. See the collections:

■ Chief Data Officers' Handbook


■ The Future of Data and Analytics Is Now

Page 14 of 15 Gartner, Inc. | G00297356


GARTNER HEADQUARTERS

Corporate Headquarters
56 Top Gallant Road
Stamford, CT 06902-7700
USA
+1 203 964 0096

Regional Headquarters
AUSTRALIA
BRAZIL
JAPAN
UNITED KINGDOM

For a complete list of worldwide locations,


visit http://www.gartner.com/technology/about.jsp

© 2016 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates. This
publication may not be reproduced or distributed in any form without Gartner's prior written permission. It consists of the opinions of
Gartner's research organization, which should not be construed as statements of fact. While the information contained in this publication
has been obtained from sources believed to be reliable, Gartner disclaims all warranties as to the accuracy, completeness or adequacy of
such information. Although Gartner research may address legal and financial issues, Gartner does not provide legal or investment advice
and its research should not be construed or used as such. Your access and use of this publication are governed by Gartner Usage Policy.
Gartner prides itself on its reputation for independence and objectivity. Its research is produced independently by its research
organization without input or influence from any third party. For further information, see "Guiding Principles on Independence and
Objectivity."

Gartner, Inc. | G00297356 Page 15 of 15

You might also like