From specific advice to general

wisdom about Product

Management.
Product School provides certifications in Product
Management across 20 campuses worldwide. In
addition to on-site campuses, we also offer the same
training, live online. Both on-site and online cohorts
are taught by real-world product managers who
work at top technology companies such as Google,
Facebook, PayPal, Airbnb, LinkedIn and Netflix.

www.productschool.com

If you want to learn more about Product

Management head to our blog. 



productschool.com/blog

Editor: Nathan Thomas and Ellen Merryweather. 


Proofreader: Gabriela Araujo.

Contributors: Roy Cobby. 



Excerpt from Article in Medium by Jack Moore. 



All rights reserved. This book may not be
reproduced or used in any manner whatsoever
without our express permission except for the use
of brief quotations in a book review or blogs. Please,
give us credit if you decide to quote our book. 



© 2018 Product School.
TABLE OF CONTENT

How To Achieve Product-Market Fit 6

User Research to Inspire and Empower 11

The Only Metric that Matters 15

Leading Through Influence 20

Why FinTech Matters 26

Building a Culture of Experimentation 29

How to Crack the PM Interview 32

The Evolution of the PM Career 36

Lessons from Managing Products at Facebook 39

How to Unleash Your Inner Leader 42

Delight Customers in Margin-Enhancing Ways 46

Google’s Approach: How to Grow to 100 Million Users 51

Prioritization for Impossible Product Decisions 60

A Product-Centric Overview of Machine Learning 64

The Product and Design Partnership 72

Why Customers Come Back: Product-Driven Growth 77

AI for Fun and Profit 82

SHIP IT: SILICON VALLEY PRODUCT like Airbnb, Reddit, and Square,
‘Ship it: Silicon Valley Product
MANAGERS REVEAL ALL
 Managers Reveal All’ shows you how

 to succeed as a Product Manager by
Dear (Future) Product Manager, putting the user first, and getting
your team on your side.

You’re up against a million deadlines.

Your engineers are fretting about the The chapters you’re about to read
functionality. Your designers are in provide specific advice on conducting
mutiny about specifications. Your user research and applying metrics
business manager is screaming about that actually matter, as well as more
the budget, and your sales team just general wisdom about the Product
wants to talk pricing. What do you Manager career path, communication
do? skills, and life lessons from PMs at the
top of their game.

The biggest challenge and most

important responsibility of a Product No matter where you are on your PM
Manager is to stay focused on the journey, you’re bound to find nuggets
user. You are the advocate for the of wisdom here that allow you to
customer. You put their interest first, upgrade your thinking, and achieve
and ensure that the end result more for your users and your career.
actually achieves what you set out to
accomplish in the first place. Enjoy the voyage!



Even the best Product Managers
cannot do this through instinct alone.
You’re not psychic. You need to gather 

data. You need to systematically 

understand your users. And you need 

to learn how to effectively communicate Carlos González de Villaumbrosia, 

this understanding to your team, so CEO of Product School 

they are inspired to share your vision,
and stay on track.

Presenting practical advice and

thought leadership from PMs at
companies 

 

How To Achieve Everything else builds from here: The

Product-Market Fit
value proposition is a statement of
how you meet the target customers
unmet needs, the features deliver that
Dan Olsen —  
 value, and the UX is how the
Author of The Lean Product Playbook
 customer experiences receiving that
value. If everything is aligned, then

 you have Product-Market Fit.
What does it actually mean to
have Product-Market fit?
The Lean Product Process
The Product-Market Fit Pyramid
To achieve Product-Market Fit via the
• UX Lean Product Process, you begin at
• Feature Set the bottom of the pyramid and work
your way up.
• Value Proposition
• Determine your target customer.
• Underserved Needs
• Identify underserved customer
• Target Customer needs.

• Define your value proposition.

As you can see, the topmost three
layers are concerned with the • Specify your MVP feature set.
product. The base of the pyramid is
• Create your MVP prototype.
concerned with the market.
• Test your MVP with customers.
The target customer is at the bottom
because it is the most important In this chapter, we’re going to
element. If you mistake the customer explore steps one through three,
that you’re going after, the entire which is the foundation of Dan
pyramid can collapse and you might Olsen’s pyramid.
have to start all over again. You only
want to target customers with
underserved needs - otherwise, they
will have no use for your product.

6
One: Determine Your Target Identify Underserved Customer
Customer Needs

Often companies have an idea of their When talking about needs, we can
target customer which sounds good drill down into Problem Space versus
at first but, upon digging deeper, is Solution Space.
actually very vague. “Millennials”
sounds good at first glance, but when • Problem Space: A customer problem,
you stop and think about it you need or benefit that the product
realize this is a hugely diverse group should address. This is a product
with diverse needs and problems. You requirement.
need to go deeper. For example: • Solution Space: A specific
“Millennials who aspire to cook at implementation or design that is
home.” intended to address that requirement.

Customers can share high-level needs Too often, PMs barrel right into
but have different detailed needs. For solution space without taking the
example, a soccer mom and a young time to think about the problem
speed demon may share the high- space. This is why a NASA contractor
level need of: “Transportation within spent $1 million back in the 60s on
100 miles of my home”, but their R&D to invent a “Space Pen” that
detailed needs are going to be very could write in zero gravity, while the
different! Soviet Union merely gave their
astronauts $0.50 pencils! Both solve
The soccer mom probably needs space the problem, but with significantly
to transport her kids, their friends, different levels of cost!
and their athletic gear. She probably
values safety, and maybe fuel The NASA “Space Pen” exists purely
economy. in the solution space. Dan Olsen calls
this “solution pollution” - NASA’s
The speed demon probably values goal was to “create a pen that writes
speed, style, image. This is why there in space.” If the goal was focused on
are so many different types of cars on the problem - “the need to write in
the road! space” the idea of a pen would not
have polluted their focus, and they
would have saved $1 million!

7
Given that the problem space is likely • How satisfied is the user while using
to contain many needs, it’s important this product?
to prioritize based on customer value.
But how do we do these? Dan Olsen Plotting these two metrics on a
proposes a formula: simple graph, we come up with three
important types of features/benefits.
Importance vs. Satisfaction
Must-have - A met need does not
• Importance: Take a customer’s always lead directly to satisfaction,
stated needs - for example, to save but it can help users avoid the
time, save money, and ask them unhappiness of having that need
“How important is this to you on a unmet. For example, seat belts are a
scale of 1 to 10.” This is an easy way must-have for a car, but using them
to determine the importance of a doesn’t confer satisfaction to the
particular need. customer! Once each seat has a belt,
there is no room for this need to be
• Satisfaction: Ask the customer how
further met. That’s the end of the
satisfied they are with the current
road. Must-have features can be the
solutions available to them.
core service your product delivers,
but what differentiates it from the
A competitive market is a need with a competition.
high importance that is currently being
well met by other companies. Try to
Performance - More is better, less is
look for a need that is of high
worse. A microchip that is 10% faster
importance, but is currently being
than a competitors’ is 10% better. Not
poorly met. This is where opportunities
having it causes problems, having it
are found.
provides value.

Defining Your Value Proposition Delighter - Not having it doesn’t

cause problems, having it causes
happiness and satisfaction. Needs and
Which benefits are you going to
features migrate over time. GPS
deliver, and how are you going to be
navigation was a delighter. Now it’s
better? Here, Dan Olsen recommends
expected, so it has moved to the
applying the Kano Model:
performance or event must-have
• How fully does the product meet category. Yesterday’s delighters
the customer need?

8
becomes today’s performance features Example: Uber’s Value Proposition
and tomorrow’s must-haves.

How do we use these three categories

to come up with our value proposition?

Your Value Proposition in a Nutshell:

• W h i c h u s e r b e n e fi t s a r e yo u
providing?

• How are you better than the

competition?

To answer these questions, Dan

suggests creating a grid where at least
one benefit is listed in each of the
m u s t - h a v e , p e r fo r m a n c e , a n d 

delighter categories. Next, add a
column for each of your competitors, Summary:
and one for your product. Score
• Peel the onion to:
yourself and your competitors on
each of these benefits. You won’t be - Determine your target customer.
high on all of them, but this exercise
will show you where you have the - Identify their customer needs.
edge - your unique differentiators: - Define the problem space.
where are you going to outperform,
where are you going to out-delight? • Identify underserved needs by using
importance and satisfaction.

• Define your value proposition using

the Kano model.

If you want to see Dan Olsen’s

ProductCon SF full talk, watch it
here. 


9
 

User Research to Doing great user research can

Inspire and Empower

empower two groups of people:

• Your team - It helps your designers

Ketan Neyak—  

and salespeople do more targeted
Fmr Product Manager, Dropbox 

work.



A PM who doesn’t speak to • Your users - It empowers them to
users is missing out on one of their key better use your product to fill their
responsibilities, and so understanding needs.
the importance of such research is
foundational to building a great Just asking people what they want
product. isn’t enough. People don’t always
know what they want. Instead, ask a
Ketan’s philosophy can be summed slightly different question: What jobs
up as: are your customers “hiring” your
product to do? Customers don’t want
products - they want something that
“Empower People by does some of their day to day “jobs”
for them.
Inspiring Empathy and
How To Get Your Product “Hired” -
Uncovering Truths.” Find Out:

• What are the jobs that occur in your

Let’s break this down. customers’ lives?

• Under what circumstances do these

Empowering People jobs occur?

• How can your product be “hired” to

Empowering people to overcome do these jobs for your customer?
needs requires an understanding of
what your users’ needs really are.

11
You can look at “jobs to be done” (jtbd) For example, a user may have a great
as a particular kind of nuanced User experience BUT if they’re unable to
Story. Here’s a tasty example: explain that experience to their
bosses, your product won’t succeed.
Milkshake Metaphor: Two types of You need to constantly ask the
people buy milkshakes. Adults who fundamental questions about the
buy them for their long commute to why. You are striving for a sense of
t h e s t o r y t h a t yo u r c u s t o m e r
work, and adults who buy them after
school for their kids. The two users experiences as they use your product.
need the milkshake to do different
“jobs.” The first group wants a large Uncovering Truths
milkshake with a small straw that
takes longer to drink and “entertains”
them during the commute. The Uncovering truths mean having a
second group needs a small milkshake methodology of validating the many
with a big straw that will quickly assumptions that go into designing
satisfy their kids without making great products.
them sick.
Facts and data that can allow you to
make better, smarter decisions.
Inspiring Empathy
Quantitative analysis can be powerful.
Facebook found that when you have
Inspiring empathy requires systematically seven or more friends, you are likely to
communicating user problems in a way stick around. With Twitter, the
that can be easily understood. It shows threshold seems to be 30 followers.
your users that you understand them,
and builds a connection. Users can tell you a lot but often that
themselves don’t accurately understand
Think about the delight that users how they are using a product. If you
experience when they use your product. don’t gather data, you can build a
You want your customers to be product based on assumptions. For
delighted, but the only way to do this is e x a m pl e , wh e n Fa c e b o ok fi r s t
by learning something which gives you released the newsfeed feature in
a deep, empathetic understanding of 2006, there was a huge backlash. 

how your users feel. 


12
Users said they didn't like it.
Numbers showed otherwise. The
newsfeed has been an undeniable
success. In this case, data succeeded
where users’ own self-knowledge
had failed. If you want to see Ketan
Nayak’s ProductCon SF full talk,
watch it here.

13
 

The Only Metric that Some of the key concepts regarding

Matters
this overall topic of measuring
success include understanding key
users, churn, virality, and your user
Josh Elman — 
 l i fe t i m e . The se c o nc ep ts c o me
VP of Product at Robinhood together to form the vital framework
that allows you to discern what
impact your product truly has on the
The best way to build products
world.
that solve real problems and have an
impact on users is to understand if and
how people are really using them. If Analyzing the Product Launch
you really want to understand your
consumers, you must first define the
metrics you’ll use to measure their One of the most important aspects of
overall satisfaction. developing a product is the launch.
All products, companies, and even
new features start with zero users. As
One of the most important ideas that
with any venture, you should have
product people should be thinking
already designed something that
about is metrics. The best way to
attracts users and delivers value, if
make sure your product performs to
not, what are you doing?
its full potential is by measuring your
progress at every step.
The first set of data you must retrieve
is asking your early adopters, even
“Success is measured the ones who stopped using the
product, about their journey. These
according to how many experiences can be positive, negative,
confused, or any feeling that gives
times your users you an idea of who is using your
product and how. Once you have a
perform a key behavior collection of stories, you can translate

within an expected that into a data set and look for

statistical correlations. You can then
time cycle.”
 turn this data back into encompassing
stories that help you take action

 within each of the different user

 journeys, depending on their needs.

15
Key Questions to ask During the This metric of real usage allows
Launch Phase LinkedIn to see that their consumers
value the ability to be available on the
• Is my product creating a story that platform, keeping them from wasting
actually matters? efforts on improving daily usage and
• How well are we satisfying user allowing them to focus on their
needs? onboarding.

• What feeling/emotion do users

• Yelp: This a product that is not an
experience when they use your
everyday need; you use it to find
product? Are they happy? Relaxed?
local business and services around
Connected?
you and you maybe use it once or
• Are they receiving what they want twice a week. If you aren’t using it
from your product? this often, you’re most likely
fulfilling the need through another
provider, such as Google. 

Once you have an idea of how your


product satisfies, you can figure out
So, a weekly metric can be suitable
how often it satisfies, or, how often
for understanding how well you are
consumers are really using your
performing. When it comes to
product. However, no two companies
reviewers, what is the cycle of usage
can use the same metrics.
like compared to the users looking
to find businesses? Do they post all
Examples of How Metrics Vary in the reviews at once? This is the type
of data you have to understand
Different Companies when defining and analyzing your
metrics.
• LinkedIn: Their users can be thought
of as two groups. Recruiters, using • Facebook: Hands down, Facebook is
LinkedIn more than three times a the definition of a daily use, even
week to find people, and, career- hourly use, a product for staying
focused people who just want to be connected to what you care about.
found, using it about three times a However, they have somewhat
month. In this case, LinkedIn’s skewed the bar for other businesses
success shouldn't be measured by who believe they need to have
habitual use, rather, by its onboarding. 
 metrics similar to Facebook’s to

 reach success. 


16
 
 

As you’ve seen in the other examples, This is a difficult task to accomplish.
this is not the case. Depending on the That cycle of a host and renter
product or service, daily usage having a good experience is much
doesn’t always accurately measure different to measure compared to
performance and is not needed to be a simply logging onto Facebook once
highly successful business. an hour.

• D i s c o r d : This communication The Three Types of User

platform for PC gamers can also be
divided into two groups, daily
gamers, and weekend gamers. When There are typically three types of
analyzing the usage, Discord has to user: core, casual and cold.
understand that if consumers aren’t
gaming, they won’t be using the Cold users rarely use the product
product, so, weekend gamers should while the casual users are habitual,
not negatively affect the usage metric but they aren’t in love with the
during the weekdays and vice versa. 
 product. The core is where we need to

 focus; these are the users who went
Breaking down the consumers like through a journey and really love and
this allows Discord to have the goal support your product.
of “whenever they are gaming, they
are on Discord” and not waste
The core user is the most important
efforts elsewhere.
because they have specific experiences
that have led them to loyalty. These
• Airbnb: This service can barely be core users, the ones really using your
called monthly usage, more like product, can provide invaluable
trimonthly. So instead of putting a insights. If you can discover and
time frame on usage, their goal is measure the users journey and the
that whenever you’re taking a trip emotions they associate with it, it will
and need to find a place to stay, you be much easier to “build more things
should be using Airbnb. How do you that will cause more people to become
convince your user to use your my core user.”
service 3-5 times a year? You have
to make sure you have a strong
Once you have a group of core users,
brand presence, so when they do
consider ways that these users can
think travel, they think Airbnb. 

“clone” themselves. This is how you

17
achieve virality; by making core users
into recruiters.

To Succeed, Understand Your

Users

How are your users actually using

your product? Consider metrics that
realistically show how frequently
users are interacting with your
product, within the context of the
needs that your product serves. Not
all products require daily use in order
to succeed. 



What matters is that when your users
encounter the need that your product
serves, your product is their first,
natural and habitual choice. Discover
and measure the user journey and the
associated emotions to reward
loyalty, and create a group of “core
users” who will be the foundation of
your success. 



If you want to see Elman’s
ProductCon SF full talk, watch it
here.

18
 

Leading Through This being such a dynamic and

Influence
diverse community, the product team
cannot simply apply the changes they
deem fit. And the PM can’t take the 



Tyler Odean — 

team’s confidence for granted. They
Director of Product at Reddit
both need to be on board for the
roadmap to success.
Some leadership advice is very
explicit: it emphasizes what you must
As a result, as Odean argues, influence
do. For instance, to take up a
is fundamental. Influence is not
particular approach towards teams,
understood in the popular, superficial
or employ different tools in your
sense, as having a lot of traction on
planning.
social media or within a certain group.
Influence means that you can generate
But effective leadership pathways are trust and persuade others that your
not about doing, but about changing product vision is the correct one.
your whole outlook. Obviously,
trying and experimenting with new
Now, does this mean that your
formulas matters a good deal. It’s just
preferred option always needs to be
even more effective if you launch
the chosen one?
these solutions from an entirely new
framework.
Let’s find out.
This is exactly what Tyler Odean,
Director of Product at Reddit, has Two Systems to Generate
found most productive. In the past,
he led teams at Google for almost a
Consensus
decade. His current PM team is
constantly improving the experiences Product Management, like any other
of thousands of Reddit contributors discipline, helps us deal with
across the world. Ranking, relevance, complexity from basic assumptions.
search, discovery… These are really Here, Odean stresses, we must refer
important for the site to work. back to how human beings form and
communicate beliefs, according to
psychology. The baseline assumption
is that we aim to make our preferred

20
messages sound more truthful. Let’s take this example from Odean:
Obviously, if we want to lead
a nyb o dy t o wa r d s a p a r t i c u l a r The team has a proposal. The
direction, we want that direction to presentation is flawless. The plan
look like the right one! looks feasible, and the potential gains
are really solid and logical. Yet, you’re
There are two general “Systems” still doubtful. This is because they
where humans deal with persuasion. really appealed to you through
System one is natural, intuitive and System 2. But they failed to engage
subconscious. System two is with you on a basic, instinctive level:
deliberate, cognitive and literal. System 1. Here, things are binary: yes
or no. Interestingly enough, you will
System 1 is the most productive. express your disagreements with
System 2: rationally, in a coherent
Regardless if we are at the emitting or
receiving end of a particular opinion, way. And your team will face you
under System 1 we make choices with similar, contrary arguments.
quickly and effortlessly. It’s the kind
of decision-making that takes place And this could go on forever, with no
below your consciousness. consensus in sight.

System 2, on the other hand, requires Avoid the Bias Trap

more focus. It resembles the image
most of us have of our thinking process
- that voice in our conscience. It’s more To set yourself free from this kind of
pitfalls, you need to address System 1.
literal and descriptive. However, it
doesn’t translate as well as System 1.
Why? Well, according to Odean, the As we said at the start, changing your
problem is it requires thoroughness on leadership implies changing your
the side of the speaker; and attention whole approach in addition to applying
on the side of the listener. Here’s where new methods and tools. Odean is very
we rationalize opinions, we try to adamant in acknowledging the biases
justify them with a combination of data that shape our perspectives.
and arguments. However, this is also
where most mistrust and disagreement As seen above, even if you rationally
takes place. know something to be true, this is not
enough to believe it. It’s basically
impossible to trick your brain.

21
At the same time, you can prepare the need to tickle System 1 as well. As
yourself for some of these preconceptions. Odean says “If it feels effortless, it feels
Let’s check them out: true”.

1. Availability
“If it feels effortless, it
Picture this situation. You’re at your feels true.”
favorite lunch joint. There are two
queues: one for sandwiches, another
one for salad. Your doctor recently 2. Anchoring
told you that you have to cut down on
sandwiches. At the same time, there Here’s another example for you. And
are way more people queueing for a it can be a very personal one. When
salad. In fact, you only have a few they make you a salary offer during
minutes for lunch today. an interview, that first number can
determine your whole outlook on the
What do you pick? Even though you job. Even if you know amounts are
rationally know that you should go open for negotiation, and you feel in a
for a salad today, the fact that the strong position, that first exchange
sandwich is much easier to reach will will likely determine your attitude
influence your decision. It will take a during the selection process.
lot of willpower to patiently queue for
the salad. In other words, “the first thing
considered always carries more
Believing an idea might work is based on weight than it should.”
its availability. Can you communicate it
easily? Do you have the tools to start As Odean argues, most “deals”
tomorrow? Is its impact easy to operate on System 1, where we don’t
measure? have much time to think or make
complicated operations. Initial values
Both your team and your users will get will work as a benchmark for future
behind your features and products if ones. So be very careful in your initial
they feel like they’re simple and projections, even if you manage to
direct. A good marketing campaign justify costs with System 2 arguments
will appeal to System 2 thinking, but a and a thousand spreadsheets!
great roadmap will take into account

22
 “The first thing Leveraging Cognitive Biases: 7
Steps
considered always
carries more weight Odean has a set of essential tips he
uses to avoid these biases. Check
than it should.” them out:

1. Keep it Simple - The easier your

3. Representation message is to remember, the easier
it is to believe. The shorter it is, the
This final word of advice is all about less likely you are to use System 2
human efficiency. As seen above, to evaluate it.
math is an abstract, complicated
language. System 1 thought is 2. Never Surprise Anyone - People
basically immune to this approach. will be skeptical of surprises. They
do not like being persuaded so
As a result, we transform operations quick because it feels as if they are
into pictures. Picture this. You’re being manipulated.
offered two boxes. One has three new
iPhones. The second one has four, but 3. Make it Really Easy to Agree -
one of them is broken. If you let Allow the people who will agree to
yourself be guided by your first agree easily, while the action is put
thoughts, you’ll go for the first box. on the people who disagree.
Should we ship 50 units? Unless
We tend to average everything. A anyone objects, we are shipping 50
product plan is like baking. If there’s units.
an ingredient in our recipe that
doesn’t fit that well, this will ruin the 4. Control the Comparison - Audiences
whole cake. Pick your best shots and are always comparing, so provide
don’t waste time with elements that the most beneficial comparisons
don’t add to your story. and delete the most destructive
ones.

“Focus on your 5. Argue as Little as Possible - Every

highlight reel.” time you disagree with your
audience, you are portraying

23
 yourself as dumb and allowing them At the other extreme, if you learn how
to dismiss you as uninfluential. the human brain works and the
connections between System 1 and 2,
6. Argue Forwards, Not Backwards - you’ll be on the highway to product
It’s much easier to argue that new success. The point of Odean’s
information will change things explanation is that you need to appeal
than arguing that past assessments to people’s cognitive biases before you
start selling them an idea.
of things were wrong.

7. Play the Long Game - The people Things like averaging values, or
you are trying to persuade will championing simplicity, are permanent
most likely matter much more characteristics of the human thought.
than the decision you are trying to Facing your internal and external
stakeholders with this in mind will
persuade them of. Be prepared to
be persuaded if your audience is greatly facilitate your product plans.
more experienced, this will show
cooperation and help establish a For more examples of cognitive biases
relation for future interactions. and tips for how understanding them
can improve your persuasiveness,
check out Tyler Odean’s recommended
Summing Up: Don’t Ditch your reading Thinking Fast and Slow.

Tools, Change your Thinking! 



If you want to watch Tyler Odean’s
One of the most common problems ProductCon SF full talk, watch it here.
with leadership in product management
guides is their tendency to overshoot.
They often list several approaches and
tools you can use to improve now.

This is all good, but it doesn’t address

the main problem: influence. If you
aren’t able to convince others, make
them embrace your solutions or even
follow you in tough times; then your
product plans are on shaky grounds.

24

 were served well, the bottom 80%
were left out.
Why FinTech Matters
Now, banks are not physically
distributed. The barriers are now about
How FinTech is Breaking Barriers
digital experience. The costs have
and Changing Finance plummeted, so access can be expanded.

Four billion people now own Banks provide many services which are
smartphones with affordable data. The broken into departments and treated
majority of the world is now connected separately - loans, mortgages,
to the digital ecosystem. Barriers to transfers, savings. There was nothing
financial technology no longer exist for that addressed an individual's financial
much of the world. The marginal cost of life as a whole. With technology coming
distribution has gone down to zero. in at scale, banking is becoming
Startups no longer need to begin with unbundled. Individual apps can each
massive investments in data storage - deliver an excellent user experience for
everything is already in the cloud, at their one specific part of this service. AliPay,
fingertips. PayTM and many others provide the
transfer service once exclusive to
banks.
In the United States, much has changed
since the late 1990s. In economic terms,
banks are a financial intermediary. The Getting access to credit used to be a
core function of a bank is managing risk major challenge for most people. The
of capital, with a lot of regulatory Catch-22 was that only people who
oversight. Banks depend on physical already have credit were permitted to
coverage - your local branch, your access more. Others were left out.
personal banker who knows you. Now, technology allows lenders to lend
responsibly to people who don’t have a
credit history. People have a digital
Because of this structure, people from
footprint, which provides the credibility
different financial classes were excluded
once only apparent through credit
from the system - only people wealthy 

rating.




enough to be valued by the bank Investment has also become more
received a good service. The top 20% accessible. It used to be available only
via expensive brokers. Now, with apps

26
like Robinhood, anyone can invest with trust you with their money, that it is
little or no fees. As banks are getting regularly available and can be tracked
unbundled, individual apps that once with certainty. Understand how people
will use their money at the level of why
Amrit Pal  —  
 - what will their earnings be applied to
Product Manager at Square - school fees, medical bills, travel?

served only one function are now 3. Serve Adjacencies

bundling more functions in, and
becoming their own financial
Serve adjacent needs around the
ecosystem.
problem of money. FinTech is an
infrastructure itself. You can’t handle
How to Build FinTech Products money without financial infrastructure. 

Once you have created this infrastructure
and have served a core market, you can
1. Develop Cultural Empathy apply your existing resources to also
serve adjacent needs.
How people treat money is highly
personal and a deep part of their
Where Are We Going From Here?
psyche. So how do you understand
what money means to someone?
We are leveling the playing field. The
• The best way to do this is to actually tools that were available to only the
spend time with customers. Talk to very privileged 20 years ago will shortly
them and understand the specifics of be available to everyone. Today’s new
their lives and the immediate players will become tomorrow’s
situations they are dealing with. Data incumbents, and new, established
cannot tell you everything. ecosystems will emerge. Highly
tailored experiences will be available
• Understand the emotion behind for everyone, so an individual can
money. Are they aggressive, or arrange their financial life the way they
conservative? want to. Today, financial data rests
within a financial institution. In the
2. Provide Utility with Trust future, this will hopefully be within an
individual’s control. 

Want more examples or tips? Then
Trust is going to be a huge factor no
check out the full talk here.
matter what you do. People need to

27
Building a Culture of Netflix set out to build a product that
Experimentation optimizes for customer lifetime value
(CLV), but they realized that they can’t
wait for 29 months to see whether
Byron Jones  — 
 someone takes more or less time to
Director of PM at Optimizely churn as a result of an experiment.
Ultimately, they turned to data science
Byron Jones is a former NASA to predict user CLV based on short-
Engineer who has brought his term changes in behavior.
knowledge and skills into the Product
Management realm. His NASA
background and five years as a PM at The Six Habits of Experimentation
Optimizely makes him well equipped
to provide advice on Building a Culture Building these “Six Habits of
of Experimentation. 
 Experimentation” will help your

 company gain usable data to make
Jones posits that fostering experimentation better decisions and improve behavior
can lead to major growth if you have an quickly.
experimentation mindset and form
strong habits early on. 

1. The first habit Jones speaks on is the


necessity of leaders asking about
Why Experimentation Matters experiment results. Jones says, "culture
starts from the top," so PMs must be
available and attentive to their team
How do you know that a product, or a
members. This shows that ideas are
product change, is successful or not?
valued, giving them the confidence to
Different people, different places,
work harder knowing they will be
different experiences, it’s impossible to
heard.
control for all of the externalities that
make it difficult to explain the story
your data is trying to tell you. 2. Second, Jones says that teams can
Experimentation is the only way to "protect" their ideas by framing them
systematically determine the effect that as experiments. This changes the
your product is going to have. conversation from risk and investment
to testing and data, allowing teams to

29
have more opportunities to capitalize simplifying experiment launches and
on promising ideas. their requirements, and automating
experiment tracking. With these
3. The third habit is that leaders should actions, you will see an increase in
always celebrate wins and losses; while experiments and an increase in
wins increase profit or have other successful decisions.
positive effects, losses allow for teams
to re-evaluate, re-test, and find a By implementing these six habits, you
solution. Jones refers to this as the will be able to build your own culture
Experimentation Singularity, where of experimentation at your venture,
experimentation becomes self- allowing for major growth and
reinforcing. innovation from your teams.

4. For the fourth habit, Jones says that If you want to listen to Byron Jones’ full
teams must hire for an experimentation ProductCon SF talk, watch it here.
mindset. He says this can be achieved by
requiring interviewees to present a data
set, and then, look for the applicant who
can best use context to tell a story about
the truth of the data.

5. The fifth habit relates to choosing the

right metrics for your experiments.
Experimentation allows teams to
remove guesswork and objectively
make decisions, so, you have to make
sure your metrics are suitable for your
optimization goals.

6. The sixth habit is about building

experiment velocity; t h e m o r e
experiments that are conducted, the
more ideas that can see the light of day.
A few ways to achieve a higher
experiment velocity is through investing
in education and training, holding
experiment review meetings,

30
 


How to Crack the PM

Interview Searching for the Perfect PM

Gayle McDowell  —
 McDowell started off by highlighting

 Founder/CEO at CareerCup the fact that “the perfect PM is a
myth.” She said that it is impossible to
Everybody knows Product be highly skilled in every discipline a PM
Management is one of the most requires, so, you should understand
attractive positions in tech. This is why your strengths and weaknesses and be
there is heavy competition from people able to portray them in a complementary
coming from diverse disciplines: way. 

engineering, design, marketing, even 

sales! Once you know where you lack
experience, you can prepare your pitch
with stories that demonstrate how you
One of the positives of heavy competition
will be successful in tackling the
is that, with a little effort, you can rise
problems you haven’t yet experienced.
above it.

In order to formulate your pitch, you

Gayle McDowell, author of several
must understand the requirements of
books, including Cracking the PM
the position and how you fit in. If
Interview, is the Founder and CEO of
you’re coming from an engineering
CareerCup, a website/forum, publishing
background, McDowell said you may
company, and consulting firm oriented
be stereotyped as great with numbers
around landing tech jobs.
but lacking the ability to make
business decisions. 

She has a lot of experience with 

interview processes. Let’s start with Knowing that the position will require
the basics. a strong business mindset, along with
other skills, allows you to prepare

 examples that show how you will be

 effective in these disciplines.



32
Reinforce your Resume Without users’ behavior, their age and where
they will use the product. A bookshelf
Repeating it for all the books in a classroom is
much different than one for just a few
in your house. Get creative and
When developing your pitch, McDowell
reinforce your message.
said the most important thing to
remember is that the interviewer will be
looking for how your stories actually The Favorite Product Question: You
represent the work you will do. This is should never come to a PM interview
why it is vital to provide stories that without having a favorite mobile app,
show how you were adaptable and FMCG, company product, etc. For all
successful. Remember, the interviewer of your favorite products, you should
only has your resume, so you have to prepare answers to questions such as: 

present yourself in a way that enforces
your resume without repeating it.
• What needs does it solve?

Hobbies can be a great way to contrast • Why do people love it?

your “stereotype.” She takes this even
• Why do you like it over alternatives?
further by saying that a hobby can be
used as your motivation or a driver, • How would you improve it?
showing the interviewer that you are
passionate and want to learn about
Most importantly, you need to show
something that may not be your insight. You need to think of the
career. An example of this is a data product as a PM and be able to
analyst who loves to paint; this demonstrate that to your interviewer.
portrays that, even though they are a Bring competitors, users, and
numbers person, they still have a
improvements into your answer to
creative design side.
show that you understand how the
product is successful and that you
Typical Interview Questions have insights on how to improve it.

A quick tip for preparing answers:

The Product Design Question: The
Find a list of strengths and weaknesses
interviewer will ask you to design a
and pick five from each side that you
product and it is difficult to do so
believe you have. For each of these ten
without asking as many questions as
characteristics, write out a story that
possible. You need to know about your
represents how you were successful or

33
improved. Make sure your stories have
a strong message by structuring them
with the situation, action, and result.

Want more examples or tips from

Gayle? Then check out the full talk
here.

34


The Evolution of the
Rung One: Entry Level
PM Career
As with any new position, you will
Helen Sims —
 most likely begin at the entry level.
Product Manager at Airbnb Sims stated that the entry level PM
position is all about “doing work with
Helen Sims, PM at Airbnb, has help” in order to gain an understanding
over six years of experience in the of your organization and your function
product industry and is eager to reveal within it.
her journey. Understanding Sims
evolution of the PM career will provide Entry level PM work entails making
a more complete idea of what iterations to existing products by
becoming a PM entails and how to learning how the other disciplines
become proficient at the role. function and beginning to interpret
and present the results. Remember,
experiment results are never what you
Your 50-Year Career Trajectory expect and human emotions are
complex, so, be flexible and adapt your
Before jumping into the PM career communications accordingly.
evolution, Sims first asks you to picture
your career in terms of a 50-year Lastly, a quick tip for that first
trajectory. This allows you to hold presentation: Never leave your messenger
yourself accountable by solidifying how apps running while presenting!
you would like to invest your time and
what results you would like to see.
Without these goals, you may become Rung Two: Doing Work (Without
lost in your career and not know what
Help!)
you want or how to achieve it.

With this broader framework in mind, The next step in the PM evolution is
let’s explore each “rung” on the PM “Doing work with help.” You’re no
ladder, and dive into the type of work longer in the entry position, you’re
and thinking what will be required now a full blown Product Manager.
from you at each level. Congrats! Sims said that now, it isn’t

36
about receiving help, rather, doing said that you must increase your team’s
work. performance through effective leadership.

At this point, you will be building Effective leadership means you need to
products from end to end and make sure you set the direction of an
interpreting massive amounts of data, entire company initiative using your
requiring you to proactively communicate own product philosophy. If your team
with your team. Lastly, Sims stated the doesn’t hold a unified philosophy, you
importance of instilling a vision in your may run into many inconsistencies
team and making sure you deflect between disciplines. Sims provided us
praise and absorb blame. with an example using her own
product philosophy:

Rung Three: PM Manager

“Customers first,
Now you’re a PM Manager… This
involves “helping others do work.” Sims
business second, ego
states that you’ll be focused on attracting last.”
and retaining quality talent, improving
your team’s overall effectiveness. Also,
you will have to develop frameworks that If you want to listen to Sims’
simplify processes such as solving ProductCon SF full talk, watch it here.
problems and giving advice or feedback.

Sims says that “people don’t leave bad

jobs, they leave bad managers,” so,
make sure you are attentive to your
team members issues and desires.

Rung Three: PM Manager Manager

The final step of the PM career

evolution is becoming a Manager of
Managers. Besides the main goal of
solving problems (lots of them!), Sims

37
Lessons from Hold to your beliefs and believe them
Managing Products at with passion, while still having a
flexible perspective that is essential
Facebook when managing an innovative and
diverse team. Product Managers need
to lead through conviction, but also
David Breger —
 be willing to change course when
Product Manager at Facebook presented with irrefutable data. It’s a
delicate - but an essential - balancing
act.
David Breger explained the
nine valuable lessons he's learned
from his time at Facebook as a Lesson Three: Be Adjustable
Product Manager in hopes that new
PMs can better handle common As a PM, you come into a team with
issues. engineers, designers and others. You
can come in with your pre-prepared
Lesson One: Prepare for Success. style. Or you can take the temperature
of the room, and adjust to the existing
style without imposing your way. It
PMs tend to have a data/engineering
can be easier to adapt yourself, rather
background, meaning they are often
than expecting others to adapt to you.
analytical and skeptical. They are
trained to prepare for failure, and are
therefore often unready for success Lesson Four: Skate Where the Product
when it eventually comes. Design is Going
products that will work if they
succeed - if your product quickly Companies who prepared for success
acquired as many users as you hope on mobile platforms before they were
for, will it cope? Apply the same paramount increased their market
attitude to your career - are you share and lead. Anticipate what is
ready for the next move? going to be big, and get there before
everyone else. This is a key way to
Lesson Two: Have Strong Opinions, differentiate yourself in the competitive
Loosely Held PM field - become one of the first

39
experts in something new, before it projects because they will look at it
becomes obvious. from a fresh angle. The Product world is
changing so quickly that backgrounds
Lesson Five: If You Want to Understand are less relevant than talent and
Your Users, Have a Diverse Team intelligence.

Breger stated that having a diverse Lesson Eight: Scale Your Product
team is crucial to understanding your Team Thoughtfully, But Make Sure
They’re Ready for Success
users and developing the products they
want. By surrounding yourself with
team members who have a diverse Understand the stage of the product
background, you can create an and feature set that you have. Assign
environment where employees can the right people to the right areas
complement each other. This based on a clear strategy and roadmap.
introduces new perspectives, so you A fundamental aspect to forming a
can benefit from insights that you scalable team is making sure they're
wouldn’t be able to come up with on set up for success; it is easy for great
your own. ideas to be lost when teams aren't
given the resources or attention they
Lesson Six: Surround Yourself with need to succeed.
People Who Compliment You
Lesson Nine: Know What You’re
Looking for In Your Career, and Do
Surround yourself with people whose
strengths compliment your weaknesses, Everything to Make it Happen
and vice-versa. This requires you to be
honest about your own limitations, and PMs must have career goals and the
seek out people who have the talents motivation to achieve them. Without
and abilities that you lack. this growth mindset, you will become
stagnant and so will your team.
Lesson Seven: Put Smart People on Always have career goals and be on
Hard Projects, Don’t Worry About the lookout for opportunities to achieve
Background. them; your personal motivation will be
contagious to your employees,
facilitating a more productive and
Don’t worry about team members prosperous team. If you want to
backgrounds when assigning tasks, watch Breger’s ProductCon SF full
rather, put smart people on hard talk, watch it here.

40
How to Unleash Your
Lesson Learned at Adobe: Listen
Inner Leader to Your Gut
Kenneth Berger —  

Fmr Product Manager, Slack Kenneth spent over 8 years at Adobe.
Starting as an enthusiastic newcomer
Everyone thinks leadership is determined to shake things up, he
something they are missing. People soon found himself as part of the “old
try and fill this perceived “hole” with guard.” He was comfortable with the
conferences, books, and articles on hours and the paycheck. So he stayed,
Medium. But Kenneth Berger explains despite the gut feeling that he wasn’t
that leadership does not come from a in the right place. Then, after 8 years,
book. It is not a missing skill. What Kenneth was laid off.
prevents most people from embracing
their leadership style are the limiting The key lesson here is to listen to your
beliefs that they don’t even know gut and be curious about what those
they have. inner feelings have to say. It can be a
little scary, because your gut is
probably telling you to break out of
What Beliefs Are Limiting Your your comfort zone. But the payoff is
Potential as a Leader? huge - it means you can work in a
way that is consistent with your
values. It means you’re fulfilled.
This is something that many people
aren’t comfortable thinking about,
but this is what is really holding us Lesson Learned at YesGraph: It’s
back.
Not Your Job to be Right
From working at a number of leading
Silicon Valley companies and having YesGraph is a startup co-founded by
both positive and negative experiences, Kenneth. His co-founder had an
Kenneth learned a number of key engineering background, Kenneth
lessons about leadership that we will had a design background. The
explore below. founders quickly fell into heated
arguments. It seemed critically

42
important to be right. The thing is, month. He asked what he could do to
the job of a founder isn’t to be right. make it right. He was reinstated, but
It’s to make a company successful. at a cost.
When the founders put being right
above being effective, the startup Suddenly, the work was imbued with
suffered. fear. It was clear that something had
not been right. After three months,
Your job is not about winning the Kenneth’s position was eliminated
argument in the meeting. It’s about and he was moved to another role.
the bigger stuff. When you let go of Kenneth fought for his old role, and
the good feeling that comes from again held onto the position. But the
fighting to be right and focus instead u n d e r ly i n g i s s u e s we r e n e ve r
on the bigger mission, that’s when resolved, and after another 9 months
you step into the role of being a Kenneth left Slack permanently.
leader.
The lesson here is to always discuss
Lesson Learned at Slack: Name the underlying causes of tension and
negativity. If they are not discussed,
the Elephant in the Room they only get bigger. Once you name
the “elephant,” it’s not an elephant
anymore. It’s just a regular business
When Kenneth left YesGraph, he was
problem. Good leaders name the hard
determined to avoid drama and focus
truths.
instead on putting his head down and
working. He believed in Slack from the
start. They were making an impact. Lesson Learned Coaching: You
But Kenneth didn’t fully understand
his job at the company. He thought it Choose Your Own Story
was conventional PM work: talking to
customers, determining features, etc. After leaving Slack after only a year,
Kenneth felt like a failure. But he
What Kenneth hadn’t realized was didn’t stay in this negative frame of
that Slack was the CEO’s “baby.” mind. He chose instead to tell his own
Kenneth wasn’t hired to write specs. story. He started writing and
He was hired to take care of this speaking. Slack wasn’t a failure, it
“baby.” And that’s a very different was an amazing achievement that
job description. Kenneth was almost came with many lessons. Kenneth
fired from Slack after about one

43
chose to define his own perspective, When you choose a perspective
and embrace new possibilities. where more is possible, you’re being
a visionary. 

Kenneth learned that the value he 

gained from his Product work often If you want to listen to Kenneth’ s
came down to offering people support ProductCon SF full talk, watch it
through one on one conversations. here.
When he realized the impact a simple
hour-long conversation could have
on someone’s life and career, he
knew that was the path for him. So he
became a coach. This was the
perspective that felt right for him.

Never accept it when someone says

“that’s just the way it is.” You can
challenge this. Choose a perspective
where more is possible.

Leadership Comes from Within

Leadership isn’t something that you

do. It’s something that you are. It’s
about being:

When you listen to your gut, you’re

being introspective and curious.

When you let go of being right, you’re

being humble.

When you name the elephant in the

room, you’re being vulnerable and
courageous.

44
NEW
 Netflix’s Big Problem
Delight Customers in
Margin-Enhancing In 2005 Netflix had a problem.

Ways
Everyone wanted the new release
DVDs as soon as possible. But when
50,000 people demand a DVD, and
Gibson Biddle —  
 the company could only afford to buy
Fmr VP of Product, Netflix 25,000, lots of people had to wait.
Across the board this was the main
Customers are always beautifully, area for improvement reported by
wonderfully dissatisfied, even when consumers and focus groups.
they report being happy and business is
good. Great things happen if you chase
So they created the Perfect New
their discomfort and address it.
Release Test, where 10,000 customers
would add a new release DVD to the
Gibson’s main mission as VP of Product top of their queue and the very next
was to delight customers and experiment day it would arrive in the mail.
madly to invent a better future. In
Gibson’s own words, early Netflix sucked
Netflix then asked ‘are these 10,000
but all early startups suck in some way.
people delighted?’ and their metric
for delight was retention. On average
What they had to come up with was a 4.5% of customers would cancel
balancing act between delight and every month in 2005. In the ‘Perfect
margin. They also factored in ‘hard-to- New Release Experience’ that number
copy.’ What could they do that was dropped to 4.45%.
unique? 20 years later, Netflix has a
hard-to-copy device ecosystem where
Doing the math, if they rolled the
thousands of tv-based devices
experience out to all of their
“automagically” stream Netflix. They
customers, they would “save” 5,000
also have their original content, massive
customers, which was worth about $1
amounts of data to aid personalization,
Million. The lifetime value was $100,
and brand trust.
which they doubled for word-of-
mouth value — each customer would
But it took a long time to get there. presumably tell a friend about Netflix.

46
However, the cost of rolling out the Netflix’s huge economy of scale —
feature would cost $5 Million. this year they will spend $19 billion
on content as they can spread this
Gibson and his team were surprised. cost against 150 million members.

•N e x t i s N e t fl i x ’ s u n i q u e
They gave their customers the one personalization technology. Netflix
thing everyone said they wanted, and knows the member tastes of 150
they didn’t value it as much as they million members around the world,
hoped. If the team increased the word giving them lots of of data to predict
of mouth factor to 8x instead of 2x, which TV shows and movies they
the math would work out (something should invest in.
which, Gibson reports, Amazon does
all the time) but unfortunately they • They have a huge device ecosystem
n e t w o r k e ff e c t - e v e r y T V-
just didn’t have the money. They
connected device on the planet is
didn’t roll out the feature, but this
pre-wired to enable you to stream
case illustrates the balancing act
Netflix.
between delight and margin.
• The last hard-to-copy advantage:
the Netflix brand. Millions of people
Stay Unique - Make Your Products around the globe trust Netflix with
Hard-to-Copy their credit card each and every
month.
Hard-to-Copy is very important for
customer retention, as it creates an
environment where you don't have to “What’s Your Social Strategy?”
worry about competition as much,
and instead focus on delighting your Finding the intersection of delighting
customers. customers in hard-to-copy, margin-
A thought experiment: If you were enhancing ways is hard.
given $500 million to create a startup
to compete with Netflix, what would In Silicon Valley in 2008, as Facebook
be hard-to-copy about what Netflix and social networks became more
does? and more hyped, “What’s your social
strategy?” was the main question on
every Venture Capitalist’s lips.
• Thefirst is their original content,
which is difficult to copy given

47
So Netflix came up with “Friends”, Netflix’s personalization delights
their answer to social networking for members by making it easier to find
movie lovers and their best guess great movies and TV shows, and
about how to improve customer personalization technology makes it
retention. easier for the company to “right size”
their investment in original content.
The idea was that you wouldn’t leave
Netflix because you didn’t want to Netflix knew that 100 million members
leave your friends, creating a hard- would enjoy “Stranger Things” —
to-copy Network effect. You could making the company comfortable
get recommendations from your investing huge sums — while correctly
friends and see what they were predicting that niche content like
watching. “Bojack Horseman” would find a
smaller audience, necessitating a
What Netflix didn’t anticipate is that smaller investment.
our friends’ tastes in movies were
terrible, and no one wanted their Lots of projects also helped Netflix
friends to know their watching along the way to a more personalized
habits. experience. For instance, Netflix
switched from using a five-star rating
system, to using thumbs up and down
Social strategy worked for many
companies, but it didn’t work for based on A/B test results. “Thumbs”
Netflix. got them twice as much taste data from
members as their star ratings system.

The Path to Personalization The road to personalization was full of

surprises. The Product team tested
Ne t f l i x c r e a t e d a p e r s o n a l i z e d whether demographics like age and
experience by encouraging members to gender improved their predictions. To
rate billions of movies and then their surprise, it was not. It was far
marrying that data with the information more helpful to know the ratings of
they had about the content through three movies or TV shows than to
matching algorithms. Netflix’s know a member’s demographics.
personalization technology created a
huge hard-to-copy advantage. When streaming really took off and the
company went worldwide, they tested
whether knowing someone’s nationality

48
and location would help. Again, it did
not. This was yet another surprise along
the road to creating a more personalized
experience for each Netflix member.

Debate. Decide. Do

Finally, we finish with Gibson’s main

secrets for success:

1. Courage. It took Netflix 2 hours to

decide to invest $100M in House of
Cards, and only 30 minutes to
cancel it when the star, Kevin
Spacey, was accused of sexual
harassment. Both of those
decisions took courage, and both
paid off. Sometimes you just have
to take the leap.

2. Patience. Netflix is one of the

biggest entertainment streaming
sites around, but it took 20 years
and lots of hurdles to get there.

3. Humility. Aside from Friends,

Netflix experienced another blip in
2011 called Qwikster, now hailed
a s t h e b i g g e s t fl o p i n t h e
company’s history. The key to
product success is admitting to
your mistakes and moving on — it
takes time and humility to invent
the future.

49
NEW
 The app launched and quickly became
Google’s Approach: the highest rated Google app on Play

How to Grow to 100

Store. At the time of writing the app
is quickly reaching the 100M user

Million Users
mark, with a rating of 4.6/5 from
over 1M ratings.

Joris Van Mens —  
 In this chapter, we’ll walk through

Product Manager, Google the three main stages of product;
ideation, iteration, and growth.
Joris and his team at Google
focus primarily on India, as it’s the
second largest market for smartphone Stage 1. Ideation: The Road to
users worldwide, with 375M and a Launch
YoY growth of +75M. The US comes in
at a close third, with 252M users, but
with only +25M YoY growth, it’s a Ideation can be summed up as ‘lots of
mature market. talking, lots of trying.’

Joris leads a group of products built A Product Manager’s key inputs at

for emerging markets users, this stage starts with research.
including the Files by Google app. It Talking to users and trying to
has three main aims, to help users understand them is the very first step
clean their phones, browse local files, of any product. What it involves is
and share files offline easily. interviewing and observing your
target demographic, reading research
papers, and digging into market
In India these features are particularly
analysis.
useful. Statistics found that one in
three users ran out of space on their
phones every day. Not only does this A. Research
mean that cleaning comes in useful,
but that they have a lot of files to sort One of the things Joris’ team found at
through and organize. The ability to the research stage, was that people’s
s h a r e fi l e s o ffl i n e m e a n s t h a t phones were filled with ‘Good
information can be shared without Morning’ images. Most people they
using data, as the availability of WiFi spoke to were in large WhatsApp
is lower compared to the US. groups, and would receive dozens of

51
these images daily, taking up a large
‘Good Morning’ Images
portion of their storage. The team
on WhatsApp from India
could never have guessed that this
was a recurring problem for their
target market - it’s only thanks to
user interviews that they had this
information.



If you’re part of a larger corporation, Device sharing between family
taking a close look at the company’s members is fairly common, which
strategy to make sure your product is can cause problems if any of the users
in line with the overall company goals have sensitive information. As soon as
is key. The tech boundary will also be Google knew this was an issue, they
something for a Product Manager to worked on a solution to empower
consider, as well as prototype their users. Now, they are able to
feedback. maintain a certain level of privacy on
their smartphones.

At Google, OKRs are set every year

and made very clear to every team, B. The Tech Boundary
which helps to keep everyone
aligned. Exploring how you can help Another part of the ideation phase is
the company reach its goals will also understanding the tech boundary,
help you out when the finance teams and knowing what is just possible.
are inspecting your performance! For Google this meant running
complex machinery models on low-
While conducting interviews in India, end devices. At the time Machine
Joris also found that privacy was a Learning had been around for a
major concern for users. while, but mostly in the cloud and

52
very computationally expensive, Prototypes should be scrappy as the
requiring immediate connectivity whole point is to save yourself a lot of
which couldn’t be guaranteed in the time by not actually building the
Indian market. product.

Running great models on-device is Without prototypes you could spend

becoming more and more possible, all the time and money building and
opening up new solutions to problems. launching a product, only to have it
As the tech is so new, it’s a rich not do so well with users in the end.
opportunity for a company like A prototype should just get the idea
Google, as no one else has been able to across, and give you a tool to test with
provide these solutions before. users.

Success at the tech boundary requires A common tool for getting prototypes
working with a great Tech Lead, made and concepts drawn, is a design
particularly if you’re not a very sprint.
technical Product Manager. They’ll
help you to identify these new
opportunities.

C. Bugs and Prototypes

At this stage you have to ideate with

your team, taking all of the inputs
you’ve collected and brainstorming
new ideas. Of the best ideas (and
there’s plenty of ways of figuring
those out) you draw concepts or build
prototypes, depending on your A design sprint serves a shortcut,
resources. where instead of building a full
prototype, you test the idea with
users, gather feedback, and then go
This gives you something to test with
away and iterate on the idea.
users. It can be very enlightening to
see a user’s reaction, either positive
or negative. In Joris’ team, they always build
prototypes to test with users, with a
two to three-week turnaround. At

53
Google X, he’s heard, they try to test feedback are reviews in the app store
up to twenty ideas in a week, so it as well as in-app feedback.
depends on your circumstances and Companies like Google can also
how easy your prototypes are to expect their product to be talked
build. Anything is fine! about in forums such as Reddit, and
in the media through press coverage.
Once you have built and verified a There’s no shortage of opinions on
the internet once a product has been
series of prototypes, you end up
starting to build something that looks launched!
like a real product, which you can
then call your Alpha or Beta. It will A. Feedback People
have a lot of bugs, and a lot of features
will be missing, but it’s something Here you can see two examples of app
that you can get into the hands of
reviews:
users and get feedback on, which is
important for building the right
products.

After a while when your Alpha starts

to really solve users problems, you
can finalize it, call it your MVP, and
launch it into the market.

Stage 2. Iteration: The ‘Easy’ Part

The story of Files Go takes a different

turn here, as it was leaked. A few
months away from being ready, with
a few features missing and a couple of 

bugs, it got out onto the market. One positive and one negative.
Google owned up to it, and got a lot of What’s interesting to note is that
users and feedback very quickly. although the team initially planned
the product to be primarily a file
cleaning app, some users were
The key inputs after a launch (or in
thinking of it as a file manager, and
this case a leak) are user feedback and
usage data. The main forms of user

54
then criticizing it for not being a full ✓ The blue line represents users who
local file manager. used the app to clean up their
phones.
Feedback can be biased, to what can ✓ The red line represents how many
be called ‘feedback people.’ These are people used the app as a file
the kinds of users who enjoy going manager, which surprised them as
online and telling people what they it wasn’t the team’s intention.
think about software. This is a specific
demographic of people, who are ✓ The green line represents offline
tech-savvy early adopters, and tend sharing, which they hoped would
t o b e p r e d o m i n a n t ly m e n . S o be used more, but because that
although a lot of user feedback via feature requires both parties to
reviews is useful, it’s good to keep in have access to the tool, they didn’t
mind that it doesn’t all come from see as much pickup in the early
your average user. stages.

B. Usage Data Surprisingly, 60% of users didn’t free

up space with the app! Following this,
the team were able to go through the
From usage data you can understand
feedback and the data, taking away
feature usage and retention drivers,
action points for the future.
gather error reports, and learn some
surprising stats.

55
Stage 3. Growth: Analyze, After laying out what the possibilities
were, Joris looked at the potential and
Estimate, and Test! the cost of each channel. He then
thought through how they’d actually
be able to use these channels, asking
Growth is all about analyzing,
“if we use ads, where would we show
estimating, and testing. It’s a very
ads, on what type of network” etc.
quantitative endeavour, and the key
to the success of Joris’ team was a
very quantitative Product Marketing Ads has almost unlimited potential,
Manager who he was able to work but of course that would require
with closely. Product Marketing has unlimited budget! Feet on the street
to be data-focused to succeed. causes a lot of excitement within the
team, but it’s difficult to do at scale
and at good cost. Cross promotions
At the start, they laid out all the
didn’t have the right scale as they
options they had for growth, starting
needed. They tested paid and unpaid
with a list of channels.
referrals, both of which yielded a
• Organic: relying on people talking decent ROI, but didn’t have the scale
about it, or the press writing about they needed either.
it.
They soon found that partnerships
• Partnerships with another company was the most viable option, as
• Ads smartphone companies also benefited
from the user’s limited space problem
• Referrals: rewarding people for being fixed. When users quickly ran
referring the app to someone else, out of space on their new devices,
very popular in consumer tech they were quick to leave negative
reviews.
• Cross-promotions with another
product
Making deals (some paid and some
• Feet on the street: people with fliers free) with some major smartphone
or in a kiosk helping people
manufacturers, particularly of low-
download the app
end devices, gained a lot of traction
and yielded excellent results for the
app.

56
You can see from this graph of weekly • Week 4 Retained Users (proxy for
new-user acquisition how what long-term retention is like)
partnerships (represented in red)
boosted growth. Without marketing, • Google Play rating (gauges how
positive people feel towards the
they would have been left with only
app)
organic acquisition (represented in
blue):
These metrics can be applied in
different ways. Joris’ team looks at
them globally, by market, by partner,
by growth channel, etc. These metrics
can be split up in different ways to see
what works and what doesn’t.

2. Dashboards

You need a lot of dashboards to dig

into the data. Google have excellent
Marketing Analysts who build
dashboards for the teams to be able to
easily read and understand their data.



3. Ad hoc analyses
The 4 Pillars of Strong Analytics
Very often you’ll have a specific
1. Key Metrics question on your mind about growth,
and somebody does an analysis on the
There are 4 key metrics which Joris data to help you answer that
and his team used to track their question.
progress:
4. Weekly growth meetings
• 28 D ay Ac t ive U s e r s ( p r e t t y
standard at Google)
These meetings can include marketing,
• Week 1 Retained Users (useful for business development, product
seeing if users just open the app management, engineering, design, and
once, or if they come back) finance. Everybody comes together to

57
go through the metrics and decide if/
what action needs to be taken.

As an example, these are some of the

topics which come up in his weekly
growth meetings:

• Key metrics
• Acquisitions
• Pre-install deep-dive
• Feature usage
• Retention by country
• Onboarding by app version
• Referrals in India

From 0 to 100 Million

The key to growth is to analyze and

implement. Growth should always be
data-driven and highly quantitative.
Make this chapter your guide to
growing from 0 to your first million
users, using Google’s tried and tested
secrets.

58
NEW
 1. Know the context
Prioritization for
Impossible Product Before you can begin making a big
decision, you need to look around
Decisions you and know where you are. You
need to know what your competitors
are doing and what’s happening in
Mariano Capezzani —  

your industry. Ask yourself, will you
Head of Group PM, HSBC
be a sheep going with the rest of the
pack, or are you on a different
The Best Job in the World mission? Will you be focusing on a
particular trend in the market,
One of the most important leading it, or responding to it?
things you’ll ever do as a Product
Manager is to make decisions based You’ll also need to look at your own
on competing demand, and limited business, namely what the goals and
resources. ambitions are. In the end, whatever
you build will be tied back to the KPIs
It’s the art of making impossible of the company.
choices, which is what Product
Managers call...life. And of course if you’re in a regulated
environment like banking, you need
As a Product Manager you hold a to be aware of any current and
great level of responsibility. Often, upcoming regulations.
you’re the one who makes the
decisions that no-one else will. 2. Understand the need

To help make these decisions, with so As Mariano says, if you are a parent
many outside influences and an you know not to feed your children
interlocking web of things to nothing but candy even if they
consider, Mariano came up with his scream for it.
own prioritization system.
When you feel the heat of customer
The 4-Step Excellent Prioritization expectation, you probably know not
to commit to delivering features at
Technique any cost, despite knowing that

60
they’re not possible. As a Product Are you aware of the intricate
Manager, you need to make sure to network of dependencies and their
deliver real value over the perception interlock that are needed to deliver
of value. something? Sometimes you are not
the one building the pieces you need.
Of course, customers are not the only Sometimes the teams that you
ones asking things of you. Your depend on have other priorities, and
you need to know whether or not
internal partners and team members
will have their own demands. For they are inline with your priorities.
example, people on your design team
will always advocate for spending Sometimes you don’t have all the
time and resources on the best answers, but that’s not uncommon in
design, most elegant UX, etc. Product Management. It’s a risk
tolerance exercise. When taking those
Your engineers will want to spend a inevitable risks, it’s important to
lot of time making sure your product listen to your Risk Managers to
is resilient. People working in understand what the blast radius is if
operations will tell you that it’s time it goes wrong. Because sometimes it
to move onto other platforms. You’ll will go wrong.
need to listen to everyone, painfully
closely, and discern the best path to This becomes a complex relationship
take based on real value over between value, satisfaction,
perception. complexity, reach, cost, and effort.
Fortunately there are frameworks you
3. Consider the execution can use, for example the very basic
Value Over Effort matrix, RICE Scoring
Method or KANO model.
Ultimately, there’s cost and effort in
building anything. The outcomes you
want to deliver to your customers 4. Arrange the sequence
come down to the expenditure of
resources. At this point the most Here’s where it all comes together. In
important question you can ask the last step of Mariano’s 4-step
yourself is “do I have what I need to technique, he goes back to
deliver?” If you’re a chef in the recommending that you follow the
kitchen, you need to gather your story. The narrative you created to
ingredients before you can begin. articulate your product vision will

61
now help to articulate what sequence play it right you’ll come out with a
of events should be happening. beautiful and consistent cadence.

Customers (and your business) will If you’re in a hurry (as most of us are)
always want more from your products, you can apply a quick acid test. Ask
so plan for continuous increments of yourself a series of questions before
value. If you have a cadence of beginning on a new feature:
improvements on the jobs customers
need to do or improvements on the • Isthis feature required for my
product to operate?
existing jobs, you’re probably already
winning. • Does it contribute to the company's
strategy, goals, and vision?
At some point, you may need to pause
and think about the bigger pieces. • Does it benefit a significant,
addressable market?
You need to replatform and enable
your business to grow by spending • Does this maximize value over cost?
cycles and calories on building new
tools. • Once built, can it be released,
measured, and supported?

The choice is between continuing an • If it doesn’t work, can I handle the

initiative or stopping to pivot to consequences?
something else entirely. It’s a tough
decision often faced by Product
Finally, Mariano compares Product
Managers, and ultimately it’s a
Management to juggling live
decision only they can make.
chainsaws. If you make the wrong
choice, it can be painful. So use
In this last step, you’ll also have to models, get ready, and get prepared.
ensure that all teams involved are
working together harmoniously, and
work efficiently towards each other.

If there’s some enabler or dependency

that you know won’t be available,
don’t build elements towards it until
those are at least understood. It’s a
complex game of interlock, but if you

62
NEW
 According to Arthur Samuel, a
A Product-Centric pioneer of AI research; Machine

Overview of Machine
Learning is:

Learning “The field of study that

Rubén Lozano —  
 gives computers the
Product Manager at Google
ability to learn without
Machine Learning (ML) and
Artificial Intelligence (AI) are often being explicitly
lumped together as the same thing.
Rubén Lozano knows otherwise, by
programmed.”
explaining the key differences
between them and how ML is related There are many problems that can be,
to statistics. and still are, solved with classic
programming. You take your data and
rules, and use classic programming to
What is Machine Learning get your answers.
Anyway?
For example, simple customization
Machine Learning is a subset of can be done by using classic
Artificial Intelligence - when a programming to show one set of
machine is trying to mimic what a search results to Person A, and
human is doing - and uses a lot of another set to Person B. ML comes
algorithms and statistics to do so. into play when you simply have too
Deep Learning is a concept that often many rules and the answers you want
comes along with ML, and can be are more complicated. The key to ML
thought of as multiple layers of is having the data and the answers,
Machine Learning. but wanting to fi g u r e o u t t h e
rules.


64
How Should Product Managers Think About ML?

ML is clearly complex. So to make it easier and add focus, Rubén breaks down
what aspects of the process a PM should concern themselves with.

First, as a PM you should ask yourself; “what is the problem I need to solve?”
Then make sure that there’s enough data, without which ML may not be a viable
option. After that, you can start to think about the right algorithm to use to
predict the answer they want. At this point the Data Scientists will join the
conversation, working on the model and eventually the output.

When it comes to understanding the vocabulary, you can use statistics. Many of
the concepts directly translate, and if you have a basic understanding of
statistics, you’re already on your way to mastering ML. It’s important for a PM
considering ML as a possible tool to get comfortable with the language
surrounding it.

65
Th e m a i n d i ff e r e n c e b e t w e e n Here are some examples of problems
S u p e r v i s e d a n d Un s u p e r v i s e d that can be solved with ML:
learning, is whether you have labelled
data or not. With Unsupervised, you
• Ranking: Helping users find the
most relevant thing
are using ML to try to find the
similarities between data clusters ✦ Example: Ranking algorithm
which you don’t fully understand. within Amazon search
• Recommendation: Giving users
what they may be most interested
When Should We Use ML?
in
✦ Example: Recommendations
ML is a cycle which should start with
from Netflix
data. If you have no data, you need to
find a different solution. Not only • Classification:Figuring out what
should you have enough data, it kind of thing something is
should also adhere to a set of certain ✦ Example: Product classification
conditions: for Amazon catalog

•Regression: Predicting a numerical

value of a thing

• Example: Predicting sales for

specific Amazon products

•Clustering: Putting similar things

together

• Example: Related news

Whether or not you can use ML is also from Google search
based on what kind of problem you’re
• Anomaly: Finding uncommon
trying to solve. ML can be implemented things
when your problem:
• Example: Fruit freshness
• Handles very complex logic
• Scales-up fast
When is ML Not Needed?
• Adapts in real-time
• Requires specialized personalization There’s no point in using ML just for
• Has existing examples of actual the sake of doing something cool, and
answers a good PM knows when to step back

66
and admit that ML isn’t necessary. It •What products should be
should not be implemented if the exclusively sold to Hispanics in the
problem: US? —Tempting, as you might have
all the data and customer profiles,
• Can be solved by simple rules but it’s discrimination and makes
• Does not adapt to new data a lot of assumptions about people)

• Requires full interpretability • Which sellers have the greatest

revenue potential? — Could be
• Requires 100% accuracy argued to be discrimination as well
- but you cannot help every single
It should also not be implemented if seller and you need to find a way to
the data: use your limited resources to have
the maximum impact. So no.
• Is unavailable/insufficient
• Is not readily accessible to you • Where should Amazon build HQ2?
— No, it’s not a repeatable problem,
• Has privacy concerns or is unsecure and you don’t already have the
answers. You could use classic
• Is irrelevant, stale, biased, or programming, but ML isn’t
otherwise low quality
necessary.

So… To ML or Not To ML? • Which search queries should we

scope for the Amazon Fresh store?
— Yes, you’ll need a combination of
Here is a quick exercise, asking ML and classic programming.
whether ML should be applied to
answer the following questions:
Let’s Do ML!
• What apparel items should be
protected by copyright laws? — No,
Once a PM feels ready to take on ML,
b e c a u s e t h i s re q u i re s 1 0 0 %
it’s time to take a look at the ML
accuracy.
Lifecycle, starting with what you
• Which resumes should we prioritize need to do it.
to interview for our candidate
pipeline? — Has great qualities for People > Processes > Tools & Systems
an ML problem, but the data is
biased.

67
Get the right people: • Preprocessing
• Formatting
There are many different roles in
different organizations, but it’s
• Cleaning
important for PMs to differentiate • Sampling
between Science and Engineering.
• Feature engineering
The people who are working with the
• Feature: individual measurable
property or characteristic of
data and doing all the maths like the phenomenon being
choosing the right model (ML
observed
Scientist, Research Scientist, Data
Scientist etc) will not be doing the • Goals: Use domain and data
Engineering. knowledge to develop
r e l e va n t fe a t u r e s f r o m
existing raw features of the
People with titles like Data Engineer,
data to increase the
Software Engineer or Dev Manager
predictive power of ML
will be doing things like collecting,
cleaning, ranking, and processing the • Test and tune models
data.
• Productionize: Integrating ML with
existing software, and keeping it
Understand the process: running successfully over time

• Formulate the problem • Deployment environment

✦ What is the problem to • Data storage
solve?
• Security and privacy
✦ What is the measurable
goal?
• Monitoring 

and maintenance
✦ W h a t d o yo u wa n t t o
predict? Sometimes you can have everything
• Select and preprocess data you need, like great data and a great
problem, but it’s just too costly to
• Selecting make sense. Part of being a Product
• Available Manager is figuring out the trade-off,
and for this you need a strong
• Missing relationship with your data scientist.

• Discarding
68
A Product Manager’s Role in ML

Once you’ve decided to move forward with ML, and understand the rough
outline of the process, it’s time to take a look at what part a Product Manager
plays in it.

Firstly, a PM has to formulate the problem. Ask yourself what the problem is,
what the measurable goal is, and figure out what you want to predict. Here’s an
example of what this might look like:

Units per order from category X in the

What is the US has remained flat YoY and engagement
problem? has declined as measured by purchase-
week frequency

What is the Increase unit order rate for category X

in the US by +X% within the next X
measurable goal? months without affecting revenue

What do you want Category X products that are more likely

to be added to a customer cart based on
to predict? items in the customer car

Once you understand the problem and your goal, the next step is to select and
preprocess data. As Rubén says, at this point you have to be pretty in the weeds.
If you don’t have the data right, everything else will be wrong. Choosing the
right data sets and knowing that they’re being used for the right purposes is a
critical PM task.

When it comes to formatting, a PM can expect to have a fairly low level of

involvement, zooming out just to make sure that everything is working the way
it’s supposed to.

69
What a PM can do, by working False Negatives, and Go vs No-Go
together with the Data Scientist, is to Metrics.
get involved with cleaning the data,
namely by having incomplete, noisy, Finally, be considerate of scientist
biased, or inconsistent data removed. time and momentum. When working
with people who have different skills,
In your PM role you should also get it’s important not to expect them to
involved with sampling, by choosing work at your pace or presume to tell
representative data. You can choose them how they should organize their
random data (for which there are time.
pros and cons) or you can use
stratified data. Being transparent about what you
need from them and why you need it,
You’ll need to also check your data for without crossing the boundary of
seasonality, leakage, or biases. telling them how to do their jobs.
There’s also a danger of your data Bringing what you do best to the table
being collected based on a certain and working in tandem with other
trend, which will affect your results. disciplines is a recipe for success.

Treat Your Scientists Right

You’ll need a good working relationship

with your Data Scientist for ML to work.
Firstly, you should treat your ML project
as a partnership. Make sure everyone
knows why you’re making your
decisions the way you are. You should
have a clear problem, hypothesis and
success metric. Start from there and let
everything else come later.

Another key part of the PM<>Tech

relationship is to be willing to make
t r a d e o ff s . R u b é n g ive s u s t h e
examples of Time vs Quality, White
Box vs Black Box, False Positives vs

70
NEW
 

The Product and Who’s Who in Design?
Design Partnership
Design is experiencing a moment
right now. A moment that began with
Abigail Hart Gray —  

the first iMac.This was the moment
Director of UX, Google
that the ugly grey boxes that people
Abigail Hart Gray had a hated looking at but loved working
background as an architect, before on became objects of joy. This is when
moving to a digital agency and then Apple discovered that people liked
eventually becoming a self proclaimed their computer more if it comes in a
Product Design Obsessive at Google. In candy-colored shell.
this chapter, we’re going to explore what
Product Managers can do to help Design To help Product Managers understand
be successful, and therefore the product Design better, let’s go through the
can be successful. different types of designers that make
up the team.
Going over a brief history of product
design, it’s clear how the early way of UX Designers: Wireframes are their
thinking (introducing design in the bread and butter. They might work
last stage) was detrimental to the under different names like
product. Interaction Designers or Information
Architects. They focus primarily on
Not having design thinking involved i n f o r m a t i o n h i e r a r c hy , fl o w s ,
early on meant the engineer would interactivity, and helping the product
hand over to the designer and team map all of that out.
essentially say, ‘pretty it up’ with no
room for them to ask questions and Visual Designers: The ones who make
recommend changes. it a branded experience. A great
visual design team helps make your
Things are different at Google now, as product unique, and make it yours
Design has a seat at the table from instead of someone else’s.
beginning to end.

72
Content: The people who write the how to know if you’re interacting
words on the page. Often overlooked with your designers well.
but incredibly important. If your
users don’t understand something According to the study, playing the
about your product, or get confused long game means you are helping
trying to use it, the directions the your designers to be business owners
content team put together will fix the with you. 41% of companies sit on the
problem. bottom rung of the graph, in the zone
of ‘we know we need Design, we
Research: The people who tell you the can’t get away with not having it, and
What, Why, and How to make sure they do what we want with no push
the product is successful. back.’ This counts as low maturity in
terms of how the company has
adopted design.
When you have a really good design
team, it helps the Product Managers
to be more efficient. Most product When a company has design
people are incentivized by number of maturity, they experience a drastic
launches. Launches are great, but if increase in revenue, cost savings,
the product flops when it’s in the time to market and valuation. This
market, if people aren’t engaging proves that design maturity is good
with it, and it isn’t doing anything to for business. InVision’s investigation
increase revenue, then it’s not worth looked at companies across industries
much. and ranging in size, so the data can be
applied very broadly.
So if you do it right up front (with a
great Design team) you’ll be much Harvard Business Review also did a
more efficient in the long-run. It can lengthy study over 10 years, showing
feel counterintuitive, but you’re that companies which were design
going slow to go better. driven - which they defined in 6
different dimensions - outperformed
the stock performance by 228%.
What Does Great Look Like?

The Perfect Triangle of Data

The InVision Design Maturity Report
is a comprehensive study looking at
what a design team can bring you and How can designers do more to work
with product? Firstly, focusing on

73
analytics is a great building block for worked at Northwestern Mutual, she
ensuring your designers can be good and the product team went after the
for your business. After all, numbers customer dashboard, which at the
tell good stories. They tell you where time brought in no revenue as not
to look. They tell you what your next many people were on digital at the
release should be. When you begin to time.
measure everything, you can find
what’s important and design towards With the improved design, the teams
that. created value where there was none
before. The new dashboard got people
Let’s moves on to what analytics can to do some of the things the company
give you: cared about, like signing up for e-
billing instead of paper, aggregating
their accounts from outside, and
• The What: what people are doing. people came back.
• The Why: qualitative reviews done
by different researchers (which is
There was almost no change in
why design teams can sometimes be
functionality, all they did was make
quite big).
clickable things look a little more
• The Will: predict the future with clickable. She did the same thing
A/B testing, surveys, concept again at AOL, by changing the design
testing at a larger scale, etc. of the mobile app, which had a direct
impact on how people used it.
These three things create the perfect
triangle of data which will take you to Discerning between designers who
market with the confidence that your will help you move up the maturity
launch will be successful. ladder, and the designers who just
want to make things beautiful,
involves more data.
Start Small
Often engineers will have much more
You need to start with something information than the designers, and
small, which as mentioned before can you need to share that information
feel counterintuitive. Some designers with them. The designers who take
will want to go after the biggest that and use it to make decisions are
impact project, but starting out like the ones who will drive your
this will be a flop. When Abigail business. Look for the thing that has

74
potential but is not performing at all,
and you’ve found your place to start.

Data-driven design is exciting and

the key to success, whether you’re in
a big company or a teeny-tiny
startup. The partnership of Product
and Design is what drives success.

75
NEW
Why Customers Come 

The Evolution of Product-Driven
Back: Product-Driven Marketing
Growth
It’s easy to see how far product
Satya Singh —  
 marketing has come. In the 80s, tech
Principal Product Manager, Expedia companies stuck to Field Sales,
generating as many leads as possible
While all Product Managers in the hopes that a small percentage
have heard of internal selling in terms would convert to customers. But this
of products, Satya Singh is all about was a very high bandwidth and high
external selling, and thinks that cost affair. It evolved into Inside Sales
growth marketing should be (which Salesforce did very well by
embedded in the product to begin cold-calling and creating prospects)
with. and eventually became marketing.
This evolution slides down the scale
As a Product Manager, Satya has towards being low-cost and low-
always battled with the question: bandwidth, until we reach product-
why do products fail? Especially in driven marketing. This is the future of
tech where products experience effective marketing.
massive growth, then stagnate and
sometimes fall. Friendster, MySpace, Product-Driven marketing essentially
Google+...all really good products means building a product which
when they started but when a markets itself, through word of
disruptor entered the market, their mouth and customers creating
growth marketing failed and they virality. In this way, the product is
couldn’t sustain the number of users growth-enabled. To be successful,
they needed to survive. you have to build this approach from
the very beginning, you have to
So what are the secrets to growth experiment, fail fast, test, and learn.
marketing, and how can Product As opposed to the old days of
Managers use it to keep their releasing a product and then saying
products in the game even when a “OK, great, now let’s go and get ten
disruptor appears? million subscribers somehow.”

76
The Attention Economy out of a product. Growth marketing is
attracting more engaged customers to
your product.”
It’s no secret that people’s attention
is like gold dust on the internet,
wh i c h g ive s u s Th e At t e n t i o n Blurring the Lines
Economy. Not just for marketers, but
everyone involved in product. If your Touching briefly on his engineering
product doesn’t grab your users/ background, Satya explains that if
target market’s attention from the yo u h ave a s ol i d e n g i n e e r i n g
beginning, it’s bound to fail. A survey background and can understand the
by The Mobile Intelligence Report chaotic nature of business, then you
shows that: can be a great Product Manager.
• 23% of users abandon an app after
one use Yo u a l r e a dy u n d e r s t a n d m a ny
important aspects of Product
• 77% of users on average are lost
Management, which involves much
after 3 days
more than simply building the
• 90% of users on average are lost product. It involves marketing,
after 30 days pricing, and sales. Some companies
still view these pillars of product in a
Some products lose their attention- very siloed way, but the truth is that
grabbing appeal halfway through, the lines between them are blurring.
some never had it to begin with, and
some lose it further along the product Nowadays, you have to build your
lifecycle. Satya’s goal is to make sure product in the right way from the
it is ingrained throughout the beginning. When your users first
journey. come to your product, they have to
be hooked and understand how it
To make it even simpler, Satya gives us solves their problem.
the example he gave his grandmother
when he was trying to explain to her
How Products Become Habits
what he does.

Slack is a great example of how a

“Just like Finance owns the flow of
product becomes a habit, which is
cash in and out of a company, Growth
ultimately the goal of Growth
owns the flow of customers in and

77
Marketing. The two aspects of habit- effect of seeing that this is a real
forming products are the emotional person who you probably know in
and the social. Not going on Slack for real life, and knowing that they are
a day leads to FOMO (fear of missing genuinely trying to raise money for
out) for many people, as they don’t charity.
know what their colleagues are
doing. The communities people build
on Slack help to create the social Can You Manufacture Virality?
aspect, further tying them to the
product. One thing a Product Manager can do
to build in Growth Marketing is to
This network effect shows very enable virality both internally and
clearly that Slack is a product with externally. In the attention-grabbing
marketing built into it, as users invite economy, you can focus on collective
their community members, team intelligence, and think about how to
members, and friends to it. The embed growth loops within the
investment of the team in Growth product itself.
Marketing led a platform which
essentially sells itself, rather than the For example, when Pinterest started
alternative of viewing marketing as a they tapped into the Facebook
separate channel. market, where they let you log in
with your Facebook account. They
On the other end of the spectrum, immediately saw the value in this and
and further evidence of the blurring benefitted from a network effect,
lines, we see marketing campaigns which led to virality and lots of users
which become products. For example joining in.
the Ice Bucket Challenge started as a
few videos on Facebook, before Once they understood that the
generating a million videos on 4 Facebook algorithm had changed and
weeks along with more than $90 they were starting to lose numbers,
million. they switched to Google and focused
more on their content and SEO. These
They did this through reciprocity and are the key aspects of Growth
social proof. The reciprocity effect Marketing - experimentation and the
came from users sharing their videos product acting as its own marketing
and encouraging others to do the channel.
same. The social proof was a side

78
What Does a Growth Loop Look who log in regularly. Loyalty is also
r e wa r d e d b y s o m e c o m p a n i e s
Like? through a points system, which can
be used for discounts.
A good growth loop is what keeps
your users coming back for more. This So You Want to Be a Unicorn
is the key to keeping users ‘hooked’
and turns your product into a habit. It Growth PM?
can start small like a notification,
conscious or unconscious. Even if you’re acquiring more and
more customers, a high churn rate
For example, users will sometimes looks bad and makes your growth
open WhatsApp without a notification, unsustainable. So how can you make
just in case there’s something they’re sure your customers come back?
missing out on. The lottery effect of not
knowing what you’re going to get, but You have to sell more to existing
anticipating some kind of value, keeps customers. Product led growth is
users hooked on products. more important than inorganic
acquisition and retention. You can
Social networks are the perfect market yourself on every channel
example and beneficiaries of the available, but without providing
lottery effect. The more time users value and building in organic growth,
spend using a social network, the it’s hard to have your product be a
more value they get out of it every sustainable marketing channel.
time they open the app, as they’ve
been slowing building their online PMs are generalists, where they look
ecosystem. It’s important to use the at lots of different things and
effect positively, as it can be used understand what’s going on at a
negatively. broader level. But the secret to being
a unicorn PM is to have deep
Some companies do this through knowledge of a particular area. For
rewards and gamification, where as example, you could learn a lot about
the user moves along the journey SEO or PPC on top of your general
they receive bonuses or points. For understanding of UX, UI, Analytics,
example Snapchat has streaks which etc.
users lose if they don’t use the app for
a day, therefore rewarding the users

79
NEW
 

AI for Fun and Profit When you set out to build an AI
system you need to know what
Jason Nichols —  
 question you’re asking, and one
Director of AI, Walmart question per model is key.

Most AI problems can be

broken down into four groups. You’re When you’re training an AI model
either trying to classify something, you need to use a loss function, which
takes into account many different
count something, find out what’s
potential objectives for your data.
important about something, or group
things together. It’s important to
think about what model you want to If you want to ask ‘how many items
use, why you want to use it, and are there?’ don’t add additional
whether or not it’ll answer your properties like ‘how blue is it?’
question, which at the end of the day
is what you want AI to do.
Step 2: Consider how AI learns

Step 1: Ask your question • Different models have different

ways of learning:

Different models answer different • Some are supervised and need

kinds of questions. They all have labelled data
different KPIs, you’ll talk about them • Some are unsupervised and
in different ways, and their properties determine their own labels
are different:
• Some take feedback from their
• Regressors answer “How many?” environment and learn via
• Classifiers answer “What’s that?” reinforcement

• Detectors answer “What are ‘these’ • Models can also have online and/or
in this?” offline learning modes (Siamese
Networks use both: Training is
• Dimensionality
Reducers answer
“What makes these things offline but enrollment is online.)
different?”

• Clustering Algorithms answer

“How can I make groups of these?”

81
Once you know which question Can you make use of a Knowledge
you’re asking, the next step is to ask Base?

yourself how your machine will The concept of a Knowledge Base in
learn: AI is very old. You can think of AI
models as very stupid rats. They have
Can humans provide feedback?
 very little neural capacity, they can’t
You need to know whether you’ll be think too hard. So if you’re able to
getting that real-time feedback or give them flashcards that they can
not. reference, they tend to do very well.

Do you have downtime to train?
 How will you know if the model is
If your system has to be online 24/7 meeting the business needs?

and can never go offline you have to This links back to your CI/CD. If you’ve
figure out how you’re going to handle documented your requirements and
that. A lot of Deep Learning models built them into your production
require intensive GPU training. While pipeline whilst keeping an eye on your
it can be done in the background KPIs, you’ll get results.
separately from your production
stream, it’s still something to factor
Step 3: Implementing your AI
in.

How will you build CI/CD?
 Once you’ve got the question and
model type from Step 1, and the
This is more of an engineering
training method(s) from Step 2, it’s
c h a l l e n g e , b u t i t ’ s d e fi n i t e l y
time to think about how you’ll
something for PMs to consider. What
implement this service.
are the tests that you want the new
model to pass? What are the KPIs that
are absolute blockers to releasing In the AI lifecycles they use at
production? 
 Walmart, they have a System Checklist

 based on Agile methodologies:
Without capturing those requirements
and building them into your
• Feature selection
production pipeline, you’ll find • Model architecture
models sneaking in which do slightly
better than previous versions, but • Initial training & transfer learning
which fail catastrophically in terms of • Model persistence
business objectives.

82
• Evaluation and CI/CD Annotators and Algorithms are just
different Agents.
• Inference
• Logging & Sampling Humans are lazy, sloppy, and
imprecise. And worse...they build
• Annotation
algorithms!
• Cross validation
• Source rating There’s a lot of work being done to
understand the source of error in
• Aggregation annotation, the biggest being fatigue,
• Normalization and sanitization coordination, misaligned incentives,
and unconscious bias.
• Training
• Knowledge base update How Can Product Drive AI?

Ground Truth
The big things for PMs to understand
to help make AI effective are that the
Jason quotes Voltaire, saying: costs and benefits associated with the
confusion matrix define business
value.
“Doubt is unpleasant,
but certainty is PMs should also communicate
probabilities and confidences to
absurd.” stakeholders so they can make
informed decisions.

When it comes to Ground Truth...there

is none. You should also make sure your work is
hypothesis driven and experimentally
validated, and think in terms of ROI
There is always a margin of error in both for improving features and
every human annotation, and if you confidence.
don’t take that into consideration as
you begin to build these models, you’re
going to have a bad time. Another way for PMs to boost their AI
knowledge is to learn the basic terms.

83
Here are the most commonly misused Error and Confusion: What PMs
or conflated terms:
Need to Know
• Accuracy: When you ask the model
for an inference, what percentage of
the time is it right?

• Precision: When the model says ‘X’,

how often is that correct? 

If there is only a 10% incidence rate
• Recall: When ‘X’ occurs how often (represented by the yellow box) and a
does the model catch it? 90% negative rate, if the model is
right 80% of the time, that means
• Incidence rate: How often does ‘X’ that its maximum precision is 33%. It
occur?
can never achieve a precision rate
• Error rate: How often is the model higher than this, because it’s going to
wrong? (this is the opposite of be wrong 2/10 times.
accuracy)
And most likely, those two times are
Incidence rate is one of the most going to be when an incidence
important things to be sure of as a doesn’t occur, so it’s going to say that
Product Manager. there were three, when there was
only one.

If you don’t know what the incidence

rate is, it’s very difficult for engineers This is a mathematical relationship that
to adequately design and test systems cannot change, despite some people
and build the training pipelines. saying “just tighten up the precision,
d o n ’ t wo r r y a b o u t a c c u r a c y ! ”
Understanding this relationship is
Before you approach AI, you should
incredibly important.
do at least some base research to
approximate this incidence rate.
It’s also important to visualize a
confusion matrix, which essentially
tells you ‘When I say X, how often do
I mean Y?’ This is the key thing to
understand for a PM, as there are
different costs associated with
different confusions.

84
Good Statement vs Bad Statement something simple, their
understanding gets even simpler!

To g i v e u s a m o r e c o n c r e t e
The first statement gives them all the
understanding of how to talk about
information they need to know, the
precision and accuracy, and keep your
second is not a meaningful statement.
language more grounded in statistics,
What they’ll take away from the
Jason shows us an example of a good
second statement is ‘yeah it’s good.’
statement versus a bad statement.

✔ “We believe the model’s precision Good Requirement vs Bad

is between 80 and 90% with 95% Requirement
confidence based on a production
experiment with 500 examples.”
✔ “The business needs the systems to
• Measured in production have a minimum precision of 0.95
• Real metric and recall of 0.90, a maximum
latency of 5s and process 10 streams
• Communicated uncertainty per GPU.”
• High and transparent sample size • Tells me false positives are worse for
the business than false negatives
• Replicable
• Te l l s
me about the compute
✘ “The model’s accuracy is 99%” available

• No it isn’t • A real definition of done

• What was the sample size? • Can still be refined by adding
information on data and mission
• How does it generalize?
• How did you measure that? ✘ “Just make it right.”

Why does this matter so much? When

•I can make anything right with
enough data and compute
communicating complicated
concepts in AI, people will always • Need to understand tradeoffs,
take away a simplified version of what constraints, overall mission
you said, so if you give them

85
Now, you’ll have everything you need
to successfully build products with
ML (chapter and AI. Make your
decisions safe in the knowledge that
you’ve been instructed by the best.

86
 THE SPEAKERS




Dan Olsen Ketan Nayak

Author of The Lean Playbook Product Manager at Dropbox
Watch the talk here Watch the talk here

Tyler Odean
Josh Elman

Director of Product at Reddit
VP of Product at Robinhood

Watch the talk here
Watch the talk here

Amrit Pal

Product Manager at Square
Watch the talk here
 Byron Jones
 Gayle McDowell

Director of Product at Optimizely Author of Cracking the PM Interview

Watch the talk here Watch the talk here

Helen Sims

David Breger

Product Manager at Airbnb
Product, Messenger at Facebook
Watch the talk here
Watch the talk here

Kenneth Berger

Fmr Product Manager at Slack
Watch the talk here
 




Gibson Biddle Joris Van Mens

Former VP of Product at Netflix Product Manager at Google
Watch the talk here Watch the talk here

Mariano Capezzani
 Rubén Lozano

Head of Group PM at HSBC
 Product Manager at Google
Watch the talk here Watch the talk here

Abigail Hart Gray 


Director of UX at Google
Watch the talk here
 Satya Singh
 Jason Nichols

Principal PM at Expedia Director of AI at Walmart

Watch the talk here Watch the talk here

Are you looking for more resources? 


VISIT PRODUCTSCHOOL.COM