SMARTER THAN THE AVERAGE GUIDE

HBR Guide +
Tools for
Data Analytics
Basics for Managers

Tools Handbook
Contents

Introduction
PAGE 2

Getting Started
PAGE 3

TOOL 1: VIDEO Big Data and Analytics

PAGE 4
TOOL 2: TIP SHEET Select Your Analytics Software
PAGE 5
TOOL 3: WORKSHEET Defining Metrics That Matter
PAGE 6

Designing and Running Business Experiments

PAGE 7

TOOLS 4 & 5: ARTICLE AND SLIDE DECK The Discipline of Business Experimentation
PAGE 8
TOOL 6: WORKSHEET Running a Business Experiment
PAGE 9
TOOL 7: ARTICLE A Refresher on A/B Testing
PAGE 10
TOOL 8: ARTICLE Boost Your Marketing ROI with Experimental Design
PAGE 11

Analyzing Data
PAGE 12

TOOL 9: QUESTION SHEET Should You Trust Your Analysis?

PAGE 13
TOOLS 10 & 11: VIDEO AND QUICK REFERENCE Regression Analysis
PAGE 14
TOOL 12: QUESTION SHEET Avoid Biases in Data-Driven Decisions
PAGE 15

Communicating Results
PAGE 16

TOOL 13: ARTICLE Visualizations That Really Work

PAGE 17
TOOL 14: TIP SHEET Tell a Story with Your Data
PAGE 18

About the Contributors

PAGE 19

Product #10196
1 HBR Guide + Tools for Data Analytics Basics for Managers Copyright © 2017 Harvard Business School Publishing. All rights reserved.
Introduction
Welcome to the HBR Guide to Data Analytics While these tools are designed to be used
Basics for Managers Ebook + Tools. This together, they can also be used individually,
toolkit provides practical guidance to help according to your needs. If you are new to data,
you understand and use data in your decision it may be helpful to proceed through these tools
making. The materials included here are chronologically. But if you already have a data set
designed to help you take the key ideas about to work with or some initial findings, you may
collecting, analyzing, and communicating data choose to skip to tools that are more applicable to
described in the guide and implement them your situation.
in your daily work. By working through this
Learning how to work with data is a must-have
toolkit, you’ll learn how to run effective business
skill for any manager. With the help of this
experiments, ask the right questions of your data
toolkit, you’ll discover how to get what you really
and data experts, and share your findings in a
need out of data—and use that information to
way that influences and persuades.
make better decisions.
The first section, Getting Started, begins by
explaining two key concepts in the data space:
analytics and big data. It then provides practical
advice on how to select the appropriate software
to store and analyze your data, as well as how
to define clear metrics. Whether you use data
regularly or are setting up a new analytics practice
in your unit, start with this section to gain a basic
understanding of data and your goals.

The next three sections will help you run tests

to collect information, crunch the numbers, and
share your findings. Designing and Running
Business Experiments describes two types of
business experiments and includes tools to help
you and your team plan your own. Analyzing
Data provides questions to ask of your data and
data scientists so that you can understand where
findings come from, assess data quality, and
prevent bias from affecting your decision making.
Finally, Communicating Results explains how
to visualize key takeaways and craft a compelling
and persuasive data story.

2 HBR Guide + Tools for Data Analytics Basics for Managers

Getting Started

Incorporating analytics into your everyday work requires careful READ MORE
thought and preparation. Before diving into experimentation and HBR Guide to
Data Analytics Basics
data collection, you need to decide what you want to measure for Managers
and how to store and analyze that information. In this section, Chapter 1
you’ll learn what big data and analytics are and how they can Are You Data-Driven?
Chapter 2
help your business. Then, you’ll learn how to find the best Keep Up with Your Quants
software to use for your data needs and identify key performance Chapter 7
Know the Difference Between
indicators (KPIs) for what you want to measure. Your Data and Your Metrics

3 HBR Guide + Tools for Data Analytics Basics for Managers

GETTING STARTED

TOOL 1: VIDEO
Big Data and Analytics
To make better decisions using data, you must first understand how data
can serve your company’s overall strategy. But concepts like “big data” WATCH THIS VIDEO ONLINE
OR OPEN THE FILE INCLUDED
and “analytics” are often misunderstood. IN THIS TOOLKIT >

The video “Big Data and Analytics” defines these two terms and
explains how companies can use them for competitive advantage. Watch
this video to gain a basic understanding of these terms before diving into
the rest of the toolkit.

TIP

• This video can also be useful to help your team understand analytics. Circulate
it, post it to an internal wiki, or play it during a team meeting when data is on
the agenda.

4 HBR Guide + Tools for Data Analytics Basics for Managers

SELECT YOUR
GETTING STARTED
ANALYTICS SOFTWARE

TOOL 2: TIP SHEET

Select Your Analytics Software
Analytics software allows you to collect and review data from a range
of sources. These programs typically provide statistical analysis,
visualization options, and reporting features.

Your company may already have a business analytics tool—such as

Oracle or SAP—in place for enterprise-level intelligence. But what
if you need an analytics tool for your specific department, team, or
project? To choose the most useful—and usable—software, you’ll
need to identify specific features that will help you achieve your data
goals. Whether you’re looking for a web analytics tool, an HR analytics
package, or a tool to create data visualizations, the Select Your
Analytics Software tip sheet by tech researcher Alexandra Samuel will
help you structure your decision.

TIPS

• It’s better to find a widely used tool that does 90% of what you need than a
niche tool that meets every single one of your requirements. Niche tools can
become outdated more quickly—or even disappear altogether. A widely used
tool is more likely to have ongoing support.
• Adjust your expectations to the complexity of what you need: If you need a
powerful and flexible tool that can work with many kinds of data, be prepared
to invest some time in learning how to use it.
• Don’t overthink system compatibility. Even if your analytics software runs on
only one platform, such as Windows, it most likely produces reports that are
viewable on other platforms. If you have colleagues who use other operating
systems who need to view those reports but won’t use the software directly,
you should be fine.

5 HBR Guide + Tools for Data Analytics Basics for Managers

GETTING STARTED

TOOL 3: WORKSHEET
Defining Metrics That Matter
As we collect data, all too often we measure what is easy or what we
think our stakeholders want rather than the metrics that actually provide
a clear indication of success or improvement. But choosing the right
metrics is crucial to everything else you do with data: collection, analysis,
reporting, and action. You must identify key performance indicators
(KPIs) that directly relate to your core business objectives and to the
actions you are taking to achieve those goals.

The worksheet Defining Metrics That Matter, written by Alexandra

Samuel, provides a step-by-step approach to identifying metrics by
creating a problem statement. This statement requires you to answer
three key questions: What do you want to improve? What will you do to
improve? and What will you measure to assess results? The decisions you
make at this stage can determine the success you’ll have throughout the
rest of your project.

TIPS

• Read through the entire tool before you tackle the individual steps so that you
see how they fit together.
• Attach a problem statement to every metric you’re tracking, and never measure
anything without a clear sense of the goal it relates to. Your metrics are only
useful if they are related to specific objectives.
• If you have multiple goals, you’ll need several problem statements. Each of
these problem statements might include two or three metrics.
• Confer with other colleagues and teams to find out more information about
which kinds of metrics resonate with those you report to.

6 HBR Guide + Tools for Data Analytics Basics for Managers

Designing and Running Business
Experiments

While data can help you understand the past, it can also help READ MORE
you make predictions about the future. Business experiments HBR Guide to
Data Analytics Basics
allow you to test new ideas to assess their viability—without for Managers
investing time or money in a full rollout. The materials in this Chapter 6
section will teach you how to craft experiments that achieve fair, How to Design a Business
Experiment
unbiased results and help you understand the basics behind Chapter 7
common tests. Know the Difference Between
Your Data and Your Metrics
Chapter 13
When to Act On a Correlation,
and When Not To

7 HBR Guide + Tools for Data Analytics Basics for Managers

DESIGNING AND RUNNING BUSINESS EXPERIMENTS

TOOLS 4 & 5: ARTICLE AND SLIDE DECK

The Discipline of Business Experimentation
A business experiment can be as simple as testing the response rate
for different email subject lines or seeing how customers react to a
small change on your company website. They can also allow you to test
innovative ideas before investing in full implementation. But to ensure
that your test yields the results you want—and is worth the expense
and effort—you need to carefully think about your goals in advance.
In their article, “The Discipline of Business Experimentation,”
Stefan Thomke, professor at Harvard Business School, and Jim Manzi,
founder of Applied Predictive Technologies, provide five questions to ask
yourself when considering running an experiment. Read the article to
understand these questions as you begin designing an experiment.

The accompanying slide deck will help you introduce the concept
of business experimentation to your team. Present this deck when
kicking off a planning discussion for an idea you’d like to test.
Customize the slides to meet your specific situation and include
examples from your business.

TIPS

• As you read the article, consider the broader corporate culture and its effect on
your ability to stage a successful experiment. Talk to others in your company
who have launched experiments to learn from their experience.
• Relate your discussion points to your group’s shared priorities or the company’s
strategy, and consider potential objections. What changes will your team
members balk at? What will they be skeptical of? Write down any questions or
issues you think people will raise, and prepare your responses in advance.
• Make the slide deck an interactive experience for your audience by preparing a
use case. For example, think of the last time your team launched an innovative
product or service that didn’t pan out. Then, use Thomke and Manzi’s rubric
to analyze what went wrong and how experimentation can help you avoid this
outcome in the future.

8 HBR Guide + Tools for Data Analytics Basics for Managers

SELECT YOUR
DESIGNING AND
ANALYTICS
RUNNING SOFTWARE
BUSINESS EXPERIMENTS

TOOL 6: WORKSHEET
Running a Business Experiment
Projects involving experimentation usually require collaboration
between team members, and there’s a lot to keep track of. Carefully
consider each step when designing a rigorous test, from clarifying your
purpose and what you hope to learn to defining your sample size and
assessing your results.

The worksheet Running a Business Experiment, drawn from Thomke

and Manzi’s article, provides questions to answer as you outline your
test. In planning sessions with your team, answer each question in
detail. Once you’ve completed the exercise, circulate a final version of the
document or post it to a shared workspace, project website, or wiki so
that it can serve as a reference point if disagreements or surprises arise.

TIPS

• For complex projects, filling out this worksheet may be a months-long exercise,
consisting of discussions, planning, and consensus building. For smaller
projects, you and your team may be able to complete it within a half-hour
meeting.
• For some of the more challenging questions on the worksheet, set up
brainstorming sessions with your team, or ask a colleague with experience
running business experiments to help you.
• As you design your test, keep in mind the stakeholders whose cooperation
you need to execute your vision. Your goals must align with theirs for the
experiment to go smoothly.

9 HBR Guide + Tools for Data Analytics Basics for Managers

SELECT YOUR
DESIGNING AND
ANALYTICS
RUNNING SOFTWARE
BUSINESS EXPERIMENTS

TOOL 7: ARTICLE
A Refresher on A/B Testing
An A/B test is a basic type of experiment that compares two versions
of something against each other to see which produces a better result.
While this technique has been around for almost a century, it has
recently grown in importance as the cost of running these types of
experiments has steadily decreased. Managers often turn to these tests
because they are easy to implement and usually don’t require a statistical
background to design.

In the article “A Refresher on A/B Testing,” Harvard Business Review

contributing editor Amy Gallo describes the basics behind A/B testing—
how to set up an experiment, how to interpret the results, and the
mistakes to avoid. Read through the article to gain an understanding of
this kind of experiment so that you can use it in your own work.

TIPS

• Though these types of tests may be simple to run (with the advent of A/B
testing software), all the same care and principles for running effective
experiments, as discussed in Thomke and Manzi’s article, should apply.
• If you have multiple variables to test, a more complex experiment may be a
better option than running a sequence of A/B tests, since it will allow you to see
not only how individuals react to certain stimuli but also how variables interact
with one another.

10 HBR Guide + Tools for Data Analytics Basics for Managers

DESIGNING AND RUNNING BUSINESS EXPERIMENTS

TOOL 8: ARTICLE
Boost Your Marketing ROI with Experimental Design
While A/B testing allows you to test one variable at a time, in some
instances you may need to study many variables at once. This situation
requires a more complex experiment.

In their article, “Boost Your Marketing ROI with Experimental

Design,” management consultants Eric Almquist and Gordon Wyner use
a variety of examples to demonstrate how clever experimental design in
marketing campaigns can project the impact of a wide variety of variables
by testing only a few. By combining this approach with statistical analysis,
it’s possible to deduce which variables produce change—to identify where
there is causation, not just correlation.

While the authors focus on marketing, these techniques can also be used
in operational research, human resources, product development, or any
other business unit. Read this article to learn how experimental design
works, then consider where opportunities exist in your own business.

TIP

• Identify people in your company or professional networks who have experience

with experimental design. Meet with them to discuss their approach and what
they’ve learned about the process before you run your own test.

11 HBR Guide + Tools for Data Analytics Basics for Managers

Analyzing Data

Once you’ve collected data—whether that’s through a rigorous READ MORE

experiment or other means—it’s time to examine it and draw HBR Guide to
Data Analytics Basics
conclusions. You and your team may be running these analyses for Managers
yourself, or you may rely on highly specialized data experts to Chapter 2
crunch the numbers for you. In either case, it’s important to have Keep Up with Your Quants
Chapter 4
a basic understanding of what questions to ask of the data to Do You Need All That Data?
ensure it can be trusted and common ways to draw conclusions. Chapter 9
Can Your Data Be Trusted?
The tools in this section will help you assess the quality of your
Chapter 12
data, understand a basic statistical approach, and ensure you’re Understanding Regression
Analysis
making fair, unbiased decisions.
Chapter 13
When to Act On a Correlation,
and When Not To
Chapter 16
Pitfalls of Data-Driven
Decisions

12 HBR Guide + Tools for Data Analytics Basics for Managers

ANALYZING DATA

TOOL 9: QUESTION SHEET

Should You Trust Your Analysis?
As a manager, knowing the key findings of an analysis is not enough
to inform your decision making. Even if you rely on a specially trained
team of data scientists to study the data set, there are key questions you
should ask to understand whether the data you have is of high quality,
how conclusions were reached, and whether those findings are valid
and useful.

The tool Should You Trust Your Analysis?, based on the work of Ron
Ashkenas, Thomas H. Davenport, and Thomas C. Redman, provides a
set of questions to ask of your data. Use it as an agenda when you meet
with your data scientists to discuss new findings, or work through it by
yourself to check your own analysis.

TIPS

• Talk directly to the individuals who are performing your analysis, and don’t be
afraid to ask follow-up questions. If you don’t understand a reply, ask them to
clarify their meaning.
• If you’re using these questions to check your own work, jot down your answers
or type them out in a separate document. This will keep you honest and
prevent you from skipping the difficult questions.

13 HBR Guide + Tools for Data Analytics Basics for Managers

ANALYZING DATA

TOOLS 10 & 11: VIDEO AND QUICK REFERENCE

Regression Analysis
Regression analysis is a powerful statistical method that lets you
identify correlations between different factors in your business. You WATCH THIS VIDEO ONLINE
OR OPEN THE FILE INCLUDED
would use this technique to answer questions like: “Do seasonal IN THIS TOOLKIT >
temperature changes affect sales volume?” or “How do staffing levels
affect the number of customer service calls?” Note that these questions
involve continuous variables (like time and money, which can take
any value within a given range) rather than discrete variables (like
whether a product is sold internationally, which is a binary property
of the product).

With a bit of practice, regression analysis is relatively straightforward

to perform, and it is frequently used in business settings. The chapter
“Understanding Regression Analysis” in the guide provides a detailed
explanation of this analytical process. The video “Regression Analysis”
and quick reference sheet in this toolkit provide brief overviews of
this method, so you can use them as resources when data experts are
describing their methodology or as reminders when you’re about to
run a regression analysis yourself. These two tools provide similar
information, so choose the format that works best for your learning style
to understand this basic statistical concept.

TIPS

• Post the quick reference in your cubicle or save it on your desktop so that you
can refer to it easily.
• Encourage your team to learn about regression analysis by posting these
resources on a bulletin board in your office or on an internal wiki.

14 HBR Guide + Tools for Data Analytics Basics for Managers

ANALYZING DATA

TOOL 12: QUESTION SHEET

Avoid Biases in Data-Driven Decisions
When we process a lot of information at once, our brains fall back on
deeply ingrained strategies for finding patterns and creating meaning.
Unfortunately, these shortcuts are often wrong and lead to flawed
decisions. In their guide chapter, “Pitfalls of Data-Driven Decisions,”
professors Megan MacGarvie and Kristina McElheran define three main
cognitive biases that decision makers often fall into while interpreting
data: the confirmation, overconfidence, and overfitting traps.

The question sheet Avoid Biases in Data-Driven Decisions, adapted

from MacGarvie and McElheran’s work, will help you identify and correct
these cognitive biases. When you’re preparing to make a decision based
on data, review these questions with your team to ensure that you’re
drawing unbiased conclusions and making well-informed decisions.

TIPS

• You will need to work with those who performed your analysis to complete
this tool, but bring others into the conversation as well. Nonquantitative team
members and colleagues can double-check your assumptions and predictions,
and outside analysts can critique your model as a whole.
• Approach these discussions with an open mind. Think of yourself as a scientist,
studying your own thought process. If you set a clinical, egoless tone, others
will follow your lead.

15 HBR Guide + Tools for Data Analytics Basics for Managers

Communicating Results

Data doesn’t speak for itself. The findings you discover through READ MORE
your analysis matter only if you can use that information to HBR Guide to
Data Analytics Basics
persuade others and drive a decision forward by showing other for Managers
people what the numbers mean and why they matter. This Chapter 18
section will help you put together visuals and stories that grab Data Is Worthless If You Don’t
Communicate It
your audience’s attention—and change their minds. Chapter 20
When Data Visualization
Works—and When It Doesn’t
Chapter 21
How to Make Visuals That Pop
and Persuade

16 HBR Guide + Tools for Data Analytics Basics for Managers

COMMUNICATING RESULTS

TOOL 13: ARTICLE

Visualizations That Really Work
As decision making increasingly relies on data, the ability to visualize
that data in a way that explains and persuades is becoming a must-have
skill for managers. In fact, a growing number of affordable tools in this
area allow individuals to create charts and figures with the click of a
button. But simply converting a spreadsheet into a chart isn’t enough
to illustrate an idea. To make effective data visualizations for diverse
audiences, you need a clear understanding about what you hope to
accomplish with a given graphic.

In the article “Visualizations That Really Work,” Scott Berinato,

Harvard Business Review senior editor and author of Good Charts:
The HBR Guide to Making Smarter, More Persuasive Data
Visualizations, shares two key questions to ask yourself for making
visuals: Is the information conceptual or data-driven? and Am I
declaring something or exploring something? Read through this piece
before you illustrate your data so that you can carefully assess the nature
and purpose of your graphic, and then choose one of four types of
visualization that best meets your needs.

TIP

• Limit the amount of information you put in your visualization so that it is

focused on one key takeaway. If you’re struggling to streamline your graphic,
eliminate features one at a time—colors, labels, even variables—until you have
a clean, simple, and clear chart.

17 HBR Guide + Tools for Data Analytics Basics for Managers

COMMUNICATING RESULTS

TOOL 14: TIP SHEET

Tell a Story with Your Data
Data becomes truly powerful when you use it to tell a story: to deliver a
message through a narrative that combines explanation with emotion
and includes some element of conflict or drama.

The tip sheet Tell a Story with Your Data, created by tech researcher
Alexandra Samuel, will help you craft a compelling, relevant story from
your data. Use this tool when you’re starting your data analysis and again
when you’re ready to turn a collection of insights into a presentation,
report, video, or other form of communication. Samuel’s advice will
help you articulate a clear goal, define your audience, and craft a strong,
persuasive data story.

TIPS

• Write your key message on a sticky note and put it on your computer or desk
as you draft your findings. If the message is literally in front of your eyes as you
create a narrative, you’re less likely to lose sight of your main argument.
• While one format (slide deck, blog post, infographic, and so forth) will work
best for your data story, consider what supplemental pieces may also be useful.
If you’re writing a detailed white paper or e-book, for example, an animated
video might serve as a promotional tool. Or if you’re presenting a slide deck, a
single chart on a handout may remind your audience of your findings long after
your presentation has ended.

18 HBR Guide + Tools for Data Analytics Basics for Managers

About the Contributors

ERIC ALMQUIST AMY GALLO

Eric Almquist is a partner with Bain & Company’s Customer Amy Gallo is a contributing editor at Harvard Business
Strategy & Marketing practice and the global head of Review and the author of the HBR Guide to Dealing
consumer insights for Bain. with Conflict. She writes and speaks about workplace
dynamics. Follow her on Twitter @amyegallo.
RON ASHKENAS
Ron Ashkenas is a Partner Emeritus at Schaffer MEGAN MacGARVIE
Consulting. He is a coauthor of The GE Work-Out and The Megan MacGarvie is an associate professor in the markets,
Boundaryless Organization and the author of Simply public policy, and law group at Boston University’s Questrom
Effective: How to Cut Through Complexity in Your School of Business, where she teaches data-driven decision
Organization and Get Things Done. He’s also written making and business analytics. She is also a research
seven magazine articles for Harvard Business Review and associate of the National Bureau of Economic Research.
is coauthoring The Harvard Business Review Leadership
Handbook, which will be published in 2018. JIM MANZI
Jim Manzi is the founder and partner of Foundry.ai, a
SCOTT BERINATO technology studio that develops and launches artificial
Scott Berinato is a senior editor at Harvard Business intelligence software.
Review and the author of Good Charts: The HBR Guide to
Making Smarter, More Persuasive Data Visualizations. KRISTINA McELHERAN
Kristina McElheran is an assistant professor of strategic
THOMAS H. DAVENPORT management at the University of Toronto and a digital
Thomas H. Davenport is the President’s Distinguished fellow at the MIT Initiative on the Digital Economy. Her
Professor in Management and Information Technology ongoing work on data-driven decision making with Erik
at Babson College and a cofounder of the International Brynjolfsson has been featured on hbr.org and in the
Institute for Analytics. He also contributes to the MIT American Economic Review.
Initiative on the Digital Economy as a fellow and is a senior
adviser to Deloitte Analytics. Author of more than a dozen THOMAS C. REDMAN
management books, his latest is Only Humans Need Thomas C. Redman, “the Data Doc,” helps companies,
Apply: Winners and Losers in the Age of Smart Machines. including many of the Fortune 100, improve data quality.
He is the author of Getting in Front on Data: Who
Does What and Data Driven: Profiting from Your Most
Important Business Asset. His articles have appeared in
many publications, including Harvard Business Review, the
Wall Street Journal, and MIT Sloan Management Review.

19 HBR Guide + Tools for Data Analytics Basics for Managers

ABOUT THE CONTRIBUTORS

ALEXANDRA SAMUEL
Alexandra Samuel is a speaker, researcher, and writer who
works with the world’s leading companies to understand
their online customers and craft data-driven reports
like “Sharing Is the New Buying.” The author of Work
Smarter with Social Media, she holds a doctorate in
political science from Harvard University. Follow her on
Twitter @awsamuel.

STEFAN THOMKE
Stefan Thomke is the William Barclay Harding Professor of
Business Administration at Harvard Business School.

GORDON WYNER
Gordon Wyner is the research director at the
Marketing Science Institute, based in Cambridge, Mass.

20 HBR Guide + Tools for Data Analytics Basics for Managers

SMARTER THAN THE AVERAGE GUIDE
The HBR Guides series provides smart answers to your most pressing work challenges.
We’ve taken our most popular topics and added downloadable tools, videos, case studies,
templates, and much more to help you implement what you’ve learned and succeed.

DOC PDF DOC

XLS
PPT XLS PDF

HBR Guide to HBR Guide to HBR Guide to

Building Your Business Case Finance Basics for Leading Teams
Ebook + Tools Managers Ebook + Tools Ebook + Tools
Product #16980E-KND-ENG Product #11185H-KND-ENG Product #10022E-KND-ENG

PDF DOC
DOC XLS
PDF

HBR Guide to HBR Guide to HBR Guide to

Persuasive Presentations Negotiating Ebook + Project Management
Ebook + Video Tools + Video Ebook + Tools
Product #11150V-KND-ENG Product #14201H-KND-ENG Product #10061E-KND-ENG

AVAILABLE EXCLUSIVELY ON HBR.ORG

HBR Guide +
Tools for
Data Analytics Basics
for Managers
WANT TO TELL US HOW WE DID?
We hope we were able to meet your needs with this
toolkit. We’re constantly striving to improve our
products. We’d love to get your feedback with a brief
customer survey. Please visit us, and tell us how we
can better serve you in the future!

Copyright © 2017 Harvard Business School Publishing Corporation. All rights reserved.
Product #10196
www.hbr.org