1

SECTION 00 In the age of big data, it feels like everything has to be data-
driven, a prefix attributed to nearly every function in our
professional and personal lives, and often without any action
to back it up. Buzzword aside, data is central to a product
manager’s (PM’s) job.

Introduction
Gartner predicts that by 2021, 75% of software
providers will rely on insights from embedded software
analytics to inform product decisions and measure
customer health.

But having data that you can make sense of, have confidence
in, connect to the rest of your systems, and access easily and
quickly is easier said than done. While this is always going
to be a challenge, product managers need to recognize the
opportunity in front of them: to leverage data to bolster their
decisions, inform new ideas, and ultimately, create a better
experience for customers.

To help shed light on how product practitioners of all levels

are thinking about this, we interviewed ten experts to hear
how they view a product manager’s relationship with data
(whether you prefer to call it data-driven, data-informed, or
none of the above).

Let’s dive in.

 3

Contents SECTION 01 What does it mean to be data-driven?

SECTION 02 There’s no data-driven product

management without the PM

Making decisions
Talking to customers
Product managers must use data responsibly

SECTION 03 How product managers use data

Discovery
Roadmapping
Feature launches
Onboarding
Experimentation and testing
Retention
Data doesn’t start and end with the PM

SECTION 04 Metrics that matter

The north star metric

Balancing quantitative and qualitative

SECTION 05 How to be a better data-driven

product manager
 4

SECTION 01 In Pendo’s 2020 State of Product Leadership report, product

managers said they see their decision-making process as
more reliant on data than instinct. But like many things in
product, everyone (or every company) has their own methods
and overall philosophy about data.

What does it mean When we asked product pros what data-driven product

to be data-driven? management means to them, the answers varied, but had

some clear overlaps.

“
Product management which follows the
philosophy of making customer and market-
informed decisions about your product
direction, as opposed to gut thinking or listening
to just one or two internal points of view.

Greg Bayer /// SVP of Product

SECTION 01 | WHAT DOES IT MEAN TO BE DATA-DRIVEN? 5

“ “
It’s essential to always learn and get a better To me, data-driven product management
understanding of our users and what is is about leveraging data through multiple
important to them when considering different venues to make informed decisions as you
changes or improvements to the product. To work through your product hypothesis and
me, the best way to monitor that on a day-to- throughout the entire product life cycle.
day basis is through analytics.
Rekha Venkatakrishnan /// Senior Manager,
Sam Benson /// Product Operations Specialist Group Product Management
SECTION 01 | WHAT DOES IT MEAN TO BE DATA-DRIVEN? 6

“ “ “
I think being data-driven I like to think about it as The overall philosophy I
means using data at the being data-informed, which think about is harmonizing a
right time, with a heavy to me means enabling balance of signals gathered
emphasis on the richness people with data they from data to better inform
of qualitative data. It’s less can be confident in, that’s your intuition. I think
looking at metrics and timely, and allows them there’s always an element
dashboards every single to make decisions in of interpretation and insight
day, and more that data response to customer gathering that needs to
feeds into everything you needs—giving them enough happen, even with data.
do as a product manager. data to be useful, but not
so much that it slows them Manosai Eerabathini ///
down. Product Manager
Bella Renney /// Head of Product

Travis Turney /// Senior Data Strategist

SECTION 01 | WHAT DOES IT MEAN TO BE DATA-DRIVEN? 7

“ “
Data helps provide objectiveness, so I think It is about making informed decisions based on
being data-driven is about looking at the data, and making data from various sources
data first, while still acknowledging that actionable—all geared toward providing
there’s always a place for gut instinct and the customer with a great experience. Most
experiences to drive decision making. importantly, it helps connect the dots across
the organization by capturing upstream and
Beatrice Fabris /// Manager, UX, downstream information and channeling it for
Content & Development customer success.

Viraj Phanse /// Senior Product Manager

Amazon Web Services

BACK TO TABLE OF CONTENTS

 8

From the responses, we can pull out a handful of key themes: using data to make decisions in service of your customers, leveraging data
throughout the product life cycle, collecting data from multiple channels, and balancing hard data with your own instincts. We’ll dig into
each of these in more detail throughout the rest of the e-book, but let’s start with the elephant in the room: the human side of data analysis.

Using data at Entire product Qualitative and

the right time life cycle quantitative

Balancing signals Actionable data Data-informed

Using the right Addressing Connecting the

amount of data customer needs organization

Intuition Understanding users

 9

SECTION 02 In his latest book, 21 Lessons for the 21st Century, historian
Yuval Noah Harari writes that “in a world deluged by irrelevant
information, clarity is power.” For product managers, that
irrelevant information (and hopefully, plenty of relevant
information) is data, coming from a multitude of sources. It

There is no data- is then up to the PM to make sense of it.

driven product
management
without the PM

“
For all of the value and objectivity that comes
with the data, there are enough variables or
situations where the human component is
always going to be part of the experience.
Having that objectivity but also bringing in your
own insights, experiences, and gut instinct is
the better way to go about it.

Beatrice Fabris /// Manager, UX, Content & Development

SECTION 02 | THERE IS NO DATA-DRIVEN PRODUCT MANAGEMENT WITHOUT THE PM 10

As we heard in the previous section, “data-driven” doesn’t 1. Making decisions

necessarily mean blindly following the data. In fact, the person
tasked with analyzing the data in front of them is equally as No matter how informative or eye-opening the data may be,
important (if not more so) as the data itself. And there are two there is always going to be a need for some sort of human
keys areas where the human element of data analysis can’t judgement. According to Tatyana Mamut, head of product at
be swapped for an algorithm or automated system. Nextdoor, the job of a product leader is to interpret the data
effectively, understand its meaning, and then move forward
depending on whether the situation requires a decision based
on empirical observation or values (or both).

Travis Turney, senior data strategist at Rapid7, echoed this

sentiment: “You’re never going to have a program where you
put in some inputs and it spits out exactly what the solution
is. You can’t have data analysis without a human in the loop.”

“ Sure, there are some questions in product management that

can be answered with a single data point—which feature has
The data is what it is, but the data never tells more usage? What’s our average NPS score? What percentage
you what a decision should be. of X type of users did we retain after three months?—but
there are many more that cannot be.

Tatyana Mamut /// Head of Product

If you’re trying to figure out why users are dropping off at a
certain point in the workflow, or how usage has changed for
a particular feature, or what version of an in-app guide will
drive the most action, there needs to be a human decision-
maker involved.
SECTION 02 | THERE IS NO DATA-DRIVEN PRODUCT MANAGEMENT WITHOUT THE PM 11

2. Talking to customers
Everything a product manager does is in service of the
customer, and the best way to understand what your customers
need or want from your product is to talk to them directly.
Greg Bayer, SVP of product at Nielsen, says that spending
time with customers periodically to get feedback on specific
features, help with roadmap planning, or even learn how they
use complementary technologies “informs a macro point of
view for a product manager that is critical for the full vision.”
“
We use product analytics in our ‘learn’ phase
Even if you think you know what customers want based to find gaps in our product and understand
on how they interact with (i.e. from product usage data) or user workflows. We then build on that with
feel about (i.e. from NPS or CSAT surveys) your product, the human element and interview users to
nothing can replace the insights you gain from face-to-face (or really empathize with their experience and
virtual) conversations. Sam Benson said that for the product
identify how to improve it.
operations team at Firefly Learning, it’s extremely important
to pair quantitative data with qualitative research from user
interviews to understand the “why” at every stage of the Sam Benson /// Product Operations Specialist
discovery process.
SECTION 02 | THERE IS NO DATA-DRIVEN PRODUCT MANAGEMENT WITHOUT THE PM 12

Product managers must use data

responsibly
The more data product managers use, (ideally) the more
value it brings. But something Beatrice Fabris, manager, UX,
content, and development at Mimecast, realized quickly is

“
that you can often spin the data any way you want. Although
quantitative data is thought to be objective, it can still be
misused depending on how the human behind the analysis
chooses to present it. That’s something I view as a bit of a concern
because people often assume that once you
Product managers have a responsibility to use data in a way have the data, it’s foolproof. But all the person
that doesn’t skew or hide the full picture. There will always
needs to do is show one part of the frame
be a need for product managers to use their own intuition,
experiences, and opinions to make the right decisions. But
rather than another part, and it’s an entirely
data plays a big part in informing these decisions at every different story.
stage of the product development lifecycle.
Beatrice Fabris /// Manager, UX, Content & Development

BACK TO TABLE OF CONTENTS

 13

SECTION 03

How product While data is always useful for a product manager, it’s useful
in different ways. Rekha Venkatakrishnan, a senior manager of
managers use data group product management, sees data as an ongoing part of
the product development life cycle, but the key is to ensure you
know the data you want to use (and why) to get to your goals.

Beatrice Fabris from Mimecast echoed this sentiment: “I wouldn’t

actually say there’s one area where data is the most valuable.
Data can be used throughout the whole process. Making sure
we use the data in the right way is where we get the value from.”

She went on to explain that on the UX team, they approach

data differently depending on the phase a product is in. Before
a new feature release, she looks at where users spend a lot of
time in the product to inform how they release complementary
in-app guides. For a product that’s already out in the market,
they use data to identify areas that need to be improved, and
to benchmark in-app guide performance.
SECTION 03 | HOW PRODUCT MANAGERS USE DATA 14

All that being said, it’s worth highlighting areas of the product Feature launches
development and customer life cycles where data can play
an important role: For new feature launches, Greg Bayer of Nielsen says his
team always tracks adoption and usage first and foremost,
Discovery including: how many customers are using the new feature,
and is it leading to more overall engagement in the product?
In the discovery process, product managers can leverage data He emphasized the importance of linking success criteria with
to identify any gaps in the product to inform what changes your customers because at the end of the day, you want them
or updates to pursue. On the flip side, it’s also useful to look to be successful because they used your product.
at your most used features to see if there are any ways to
improve that experience for your users. If you know a particular You should also analyze a new feature’s success to understand
feature is associated with key customer outcomes, you should any usage patterns and what customers might still need from
always be looking for new ways to make it better. the product. Most importantly, make sure you have tracking
in place before going into a new feature launch. Travis Turney
Roadmapping of Rapid7 believes the metric itself doesn’t matter as much
as having some sort of measurement program so you can
Product usage data is valuable for directly informing the learn over time what “good” looks like.
product roadmap (i.e. which features are customers using the
most, and therefore require more attention?), but Viraj Phanse, Onboarding
senior product manager at Amazon Web Services, brought up
another scenario. He said that sometimes, a product manager When building an onboarding strategy, product data can help
will have an idea for the roadmap but no data available to you determine which features to include in your onboarding
back it up. In this case, PMs still need to make the call one flow (aka those that are most likely to lead to success in the
way or the other and take smart risks. Adopting a customer- product) as well as help you measure onboarding effectiveness.
centric approach can help as well: focus on your customers’
problems throughout the roadmap journey, and constantly
ask yourself what customer problem you’re solving.
SECTION 03 | HOW PRODUCT MANAGERS USE DATA 15

Travis Turney of Rapid7 also called out onboarding as an Retention

important time to learn about your customers through the
data you collect. Since customers are more likely to feel Retention is always important, but it has become even
comfortable providing information about themselves when more critical for businesses to stay ahead of retention by
they first start using your product, Travis says it is your best understanding how customers are using the application. Sam
opportunity to capture data (e.g. their role in the company) Benson explained how at Firefly Learning, customer success
that can then allow you to provide a more personalized managers (CSMs) need to have an idea of their customers’
product experience going forward. user journeys in order to have productive conversations,
especially when it comes time for renewals. This way, CSMs
Experimentation and testing can offer the right recommendations tailored to each customer
account’s usage.
The experimentation process hinges on data as you test,
analyze, and learn what works and what doesn’t. Manosai You can learn the most from
Eerabathini of Google sees data as crucial to experimentation,
especially for products that are well established and where
customers who don’t stick around
testing and implementing small tweaks to the user experience
can make a large impact. In addition to leveraging data to proactively address
retention, Bella Renney, head of product at Tray.io,
Andy Browning, UX writer at Mimecast, explained this from believes the most valuable information comes from
the perspective of the UX team: “We found that when we do talking to the customers who don’t renew their
user testing, we have our own ideas and instincts going in, subscription or continue using your product:
but then we actually see the users engaging with the product
and get the data, and we might be completely wrong. And “I’m much more interested in the customers who came
you can’t really argue with it—you know where you need to to the tool, used it, and then decided that it wasn’t
make improvements.” valuable. Why did they churn? I think that’s the most
important piece of data you can get as a product
manager to make prioritization decisions.”
SECTION 03 | HOW PRODUCT MANAGERS USE DATA 16

Data doesn’t start and end with the PM

When used effectively, product data exists as a shared

language across the company. Nearly every team has a
reason to care about how the product is performing or how
customers are using it, especially at organizations that are

“
product led. So, a big part of a product manager’s job is
to empower the rest of the organization with the data and
insights they’ve collected.
I make sure all stakeholders have access to
Here are some tips for sharing product data effectively: broader reporting data if they want, but I always
• Ensure all stakeholders have access to product data
find it most useful to hold a monthly show and
tell whereby the product team talks about
• When sharing product data, deliver the information that’s recent releases and their value to clients.
relevant to each type of stakeholder (e.g. different teams)

• If you have a north star metric, utilize that as your mechanism Greg Bayer /// SVP of Product
for sharing product data

• Include product data in what’s shared at company-wide

all-hands meetings or town halls

• Provide context as to why the data is important to each

stakeholder or team specifically

• Use Slack (or another communication channel) to share

product data and foster collaborative conversations

BACK TO TABLE OF CONTENTS

 17

A product ops use case CASE STUDY

When product ops is an established function in an organization, To offer examples, Sam said she analyzes user workflows
it is often tasked with owning product data—including collecting and customer data, then reports trending issues to cross-
it, helping analyze it, and making it easily accessible for other functional teams throughout the company. The product ops
teams. Sam Benson of Firefly Learning does just that: org also makes sure every team across the business is nudged
into utilizing the tools, data, and insights the product team
“I work with all of our teams at Firefly to help provides. Sam also helps funnel product ideas and feedback
from customers back to the product management team.
relieve their workload and enable them to
dig deep into user data via product analytics With all of the different ways product managers use data, it
systems like Pendo, to make sure we’re providing begets the question: which metrics should they use?
a platform that our users love.”
 18

SECTION 04 Using data to build a great product is only half the story.
Product managers need to track things like product usage,
user sentiment, and customer feedback in order to continuously
provide the best product experience and ensure customers’
needs are met. Even Sweetgreen, a fast-casual salad chain,

Metrics that matter

identifies as a technology company, and is shifting to tracking
metrics like active users in their app and lifetime customer
value rather than just store sales or foot traffic.

Humans naturally gravitate toward information that is organized

and when possible, simplified. And one of the most difficult
exercises can be determining which product metrics to measure
and then, which metrics matter the most. When we asked our
interviewees what the most important metrics are for PMs to
measure, there was one overwhelming consensus: it all depends.

Here are some of the factors that impact which metrics product
managers should track:

• Whether your company is B2B, B2C, or a hybrid

• Your industry (including Saas vs. non-Saas, enterprise

vs. consumer)

• The stage your product is in

• Your organization’s goals (e.g. new customer acquisition

or retention)

Since there is no one-size-fits-all for product metrics, here are

some different ways to approach choosing what to focus on:
SECTION 04 | METRICS THAT MATTER 19

“ “
I think at the end of the day there needs to be You’re going to have a trade off between
a way to connect the dots back to what the quantity of data and completeness—do you
overall business objectives are. If you can’t do need a lot of data or do you need fewer data
that, then it probably means you’re not focusing points that are more complete, like written
on the right thing. responses from a survey? This is when you
might need to employ multiple methods
Manosai Eerabathini /// Product Manager for data collection, for example running a
sentiment survey in-app and through email.

Travis Turney /// Senior Data Strategist

SECTION 04 | METRICS THAT MATTER 20

“ “
As someone who has worked on B2B and B2C I’ve worked primarily in B2B where I look for
products, it’s all about customers first: acquire, success around feature adoption and activity—
engage, and retain. how often are clients logging in, or how many
individuals are using the product per client. If
Rekha Venkatakrishnan /// Senior Manager, there is sales data available, I also look at how
Group Product Management different KPIs (e.g. ad revenue) are trending
based on feature adoption.

Greg Bayer /// SVP of Product

 21

Two example metrics frameworks CASE STUDY

1 Viraj Phanse of Amazon Web Services shared a metrics

• •
2 Tatyana Mamut of Nextdoor shared the metrics framework
framework whereby you categorize metrics by the internal her team developed, which is broken down into four
team or external stakeholder they are relevant to. Here categories as well:
are sample metrics for each of his four defined categories:
• Product value: these are lagging indicators of the product’s
• Management: revenue, profitability, Net Promoter Score success, for example engagement metrics (e.g. DAU/
(NPS), retention, customer lifetime value (CLV) MAU) and revenue.

• Customer: NPS, customer satisfaction (CSAT) score, • Product love: these help the product team understand if
customer effort score (CES) they are not only building a product that customers use,
but one that they love. Ideally, they are leading indicators
• Marketing: CLV, conversions, growth in daily and monthly like week 4 WAU and NPS.
active users
• Product purpose: most companies have a purpose
• Engineering: number of issues, velocity, click-through statement, but don’t necessarily measure their effectiveness
rate, NPS at delivering on that purpose. At Nextdoor, the team does
this through a Neighborhood Vitality metric, which they
track by looking at the number of High Quality Connections
that are made (online and offline) and the amount of
reported content on their platform.

• Employee passion: since you can’t create a great product

without passionate employees, Nextdoor measures
employee passion (via frequent surveys and constant
feedback) and elevates it to the top level of their
measurement framework.
SECTION 04 | METRICS THAT MATTER 22

The north star metric

The idea of a north star metric is a common practice for Her advice for deciding on a north star metric or compilation
product teams looking to narrow in on one data point. Since score of your own? “Take a long, hard look at what success is
it can be challenging to balance all of the data coming in from for your users and build your score around that specifically.”
multiple places, a north star metric helps teams focus. The
only downside being that this could cause product managers
to lose sight of other product signals that matter, too.

Manosai Eerabathini of Google wasn’t always a believer in

the north star metric, but has come to recognize the value of
collapsing your focus and being able to point to one overall
metric to track progress. His approach to the north star metric
is to build a hierarchy of supporting metrics that ladder up
to your north star. “
At Tray.io, the team created a custom north star metric called
If a PM doesn’t have conviction around their
a user score, which aggregates multiple data points (e.g. ultimate goal, then it’s going to be difficult
daily active users, clicks, and other funnel metrics) into one to parse out what you’re trying to optimize
high-level view. Bella Renney says she looks at this metric for. When you have a bunch of different
daily, usually by examining the number of users who are in
signals you’re either going to run the risk of
three different thresholds of scores. This aggregated score
randomizing yourself, or you will be whip-lashed
also allows her team to drill into underlying data points to
understand why a particular customer’s score has increased around with what the data is telling you.
or decreased.
Manosai Eerabathini /// Product Manager
SECTION 04 | METRICS THAT MATTER 23

Balancing quantitative and qualitative

Another theme that came up in nearly every conversation is From there, it’s a matter of digging into the quantitative side of
that PMs need to balance quantitative and qualitative data. things by looking at NPS survey responses or cohort analysis
According to Greg Bayer of Nielsen, “a good product manager for the particular feature where the customer is having trouble.
needs to blend both together equally.”
Next, we’ll hear what our experts believe product managers
Here’s what that can look like: need in order to use data effectively.

At Mimecast, Beatrice Fabris says they use their product

analytics tool to validate whether the problem areas the team
hears most often in customer calls match where users are
spending the most time in the product. The data complements
qualitative information coming from support calls in order to
form a more complete picture.

Travis Turney of Rapid7 said the power of quantitative and

qualitative data was especially crucial when the team worked
“
to optimize their onboarding flow. By combining information When you have quantitative data like product
from support tickets and product usage, they were better usage or survey responses based on a ten-
able to identify areas of the product where they could invest point scale, it’s important to combine that with
more and provide a better experience for new customers.
qualitative data like open-ended text-based
Viraj Phanse of Amazon Web Services offered an example
surveys or customer feedback. I think that’s
related to understanding customers’ problems—he believes where you find the greatest insights.
product managers need to actually be on calls with customers
in order to understand the problems they’re facing. Travis Turney /// Senior Data Strategist

BACK TO TABLE OF CONTENTS

 24

SECTION 05 To help fellow product leaders continue improving, we asked

our interviewees to name the one skill or tool that a data-
driven product manager needs to do their job. Here’s what
they had to say (PMs, take notes).

How to be a better
data-driven product
manager

“
There are two skills that tend to separate the
great product leaders from the rest. One is
intuition for the customer, built out of spending
countless hours with customers and using
the product as the customers would use it.
The second is courage to make leap-of-faith
decisions, even when the decision is unpopular
or the data is ambiguous or non-existent.

Tatyana Mamut /// Head of Product

SECTION 05 | HOW TO BE A BETTER DATA-DRIVEN PRODUCT MANAGER 25

“ “ “
Use Excel wisely. Be able I’d say it’s a product Invest your time in
to create pivot tables, trend manager’s ability to understanding the data—
graphs, and tell a story with interpret the data given to experiment and play with
a large, raw data set. them, and how they then it, and finally, learn from it.
present it to others. There is no time boxing to
Greg Bayer /// SVP of Product this and it is a continuous
Sam Benson /// skill to hone.
Product Operations Specialist

Rekha Venkatakrishnan ///

Senior Manager, Group Product
Management
SECTION 05 | HOW TO BE A BETTER DATA-DRIVEN PRODUCT MANAGER 26

“ “
You can be the best PM in the world and track Statistics and data visualization are two skills
everything so diligently and make sure that all that all product managers should acquire.
of your JIRA tickets are in order. But at the end They help PMs convert data into meaningful
of the day, if you’re not able to communicate insights, run user experiments, test product
with the designer who is passionate about hypotheses, and present data to stakeholders
their own meticulousness, or the vice president in a meaningful way.
who has no time and needs to understand
everything in two seconds, then it makes your Viraj Phanse /// Senior Product Manager
own job as a PM much harder. Amazon Web Services

Beatrice Fabris /// Manager, UX, Content & Development

SECTION 05 | HOW TO BE A BETTER DATA-DRIVEN PRODUCT MANAGER 27

“ “ “
Establishing a north star— From the UX perspective, Having data literacy is
some high level metric or I think you really need to key. People who really
goal that you can look up at, deeply understand your understand data and have
especially when you’re deep users because once you get that fluency are able to
in the weeds and wondering any data, it’s about how you choose goals that are
if you’re still on course. I interpret it to create a better going to help them predict
find that really grounding experience for the end user. success better than the
and I would advocate for all folks who don’t have that
product managers to do that. Andy Browning /// UX Writer knowledge.

Manosai Eerabathini /// Travis Turney ///

Product Manager Senior Data Strategist

BACK TO TABLE OF CONTENTS

 28

www.pendo.io