[Don’t] listen to your customers

MARCH 31ST, 2020

Kristen Berman

Dr. June Park John

Today’s product and design leaders often rely heavily on the word of their customers when building
their product road maps; whether it’s a customer survey or a phone interview, loads of qualitative
data through these methods is being collected and used to dictate how to design and develop
products. Seems like a foolproof plan, right?

Actually, no—a sole reliance on customer input

and feedback is built on an antiquated model of
human decision making that assumes humans
are rational.

Let’s take a look at why.

The standard advice for getting inside users’

brains is, “talk with them directly.” As the Ideo.org
design guide says: “There’s no better way to
understand the hopes, desires, and aspirations of those you’re designing for than by talking with
them directly.”
In some cases, this works. In many others, however, it fails fantastically. Why? Much of our
behavior, as proven time and time again in psychological research, is shaped by mental gymnastics
we’re not even aware of. In order to truly understand user motivation, we need to often go beyond
the interview and dig deep into observation and environment.

When asking questions fails

Imagine you’re a researcher at a company. You’ve been tasked with increasing employee
participation in retirement savings programs. Your company already tried doing in-depth financial
education to no avail; only 47% of their employees are signing up for sponsored retirement plans,
which is a big deal when you consider similar companies boast 90% participation. Your company
wants to do away with this disparity, and it’s your job to figure out what’s causing it.

If you’re following the typical PM and researcher script, you’d start with the customer. You’d ask
people why they are or are not saving for retirement.
We did this. We asked over 900 people nationwide why they were or weren’t saving for retirement.¹
We asked in different ways to make sure we represented how a typical research would ask the
 question. For example, we gave some participants
multiple-choice questions (a quantitative, survey-
based research method) and others open-ended text
responses (a qualitative, interview-based method).

Surprisingly, regardless of how we asked the question,

their answers did not reflect reality. The answers were
not only misleading, they completely missed the core
insight on what actually drove them to save or not
save for retirement.

What people said:

86%² of people who were saving for retirement cited how important it was to have money for the
future, saying things like “I want to be able to have something to fall back on” and “I want to be
prepared and live comfortably.” Whereas people who were not saving for retirement said things like:
“I can barely afford to pay bills.” 64% of this group blamed not having enough money.

These seem like reasonable, intuitive responses. However, they are false.

The real reason:

The real reason people save for retirement or (don’t save) is because they’ve been defaulted into a
savings plan. Companies who automatically enroll employees in a retirement savings plan increase
savings rates by 50 percentage points.³

It has nothing to do with a preference for the future or not having enough money.

But some people probably got it correct, right?

Right?! Not really.
How many employees were able to articulate why they choose to save or not save for retirement?
Across all conditions, only 6.2% of participants mentioned anything close to enrollment form design
or default enrollment. Getting granular, while 12.3% of multiple choice respondents cited default
enrollment as the reason they’re saving, only 0.2% of those answering open-ended questions
mentioned it. That’s shockingly low.

If we relied solely on data from these traditional qualitative and quantitative research methods, we
would likely spend a lot of time and money building solutions that would miss the mark, and for sure
be less effective than just designing a default enrollment.
Put as a more general takeaway, people aren’t generally aware why they really make the decisions
they do. They place far too much emphasis on their own attitudes and beliefs when they should
really be giving credit to context and environment.
Imagine you could only trust 6% of the people
you brought in for research. What’s Going On?
Decades of work in psychology show that much of human behavior is guided by mental processes
outside our conscious awareness.⁵⁶ Simply put, we are unaware of what goes on in our brain, our
susceptibility to biases, and how we make decisions.⁷

Given we’re typically incognizant of our decision-making processes, you’d imagine questions about
our decision-making should be met with answers like, “I don’t know” or “I’m not sure” or “I’m not all
that confident.” What happens, however, is the exact opposite.

In a study we conducted with 496 people, we explicitly asked whether participants thought a default
setting would influence their choices for organ donation and retirement savings, and how confident
they were about this.

If people were aware of how much the environment impacts their decision making, we’d expect
people to have lower confidence in their answers. Not the case. 71%⁸ of people⁹ who said the default
would not impact their retirement or organ-donation decision were “very” or “completely” confident in
their answer.

Not only do people avoid answering questions with uncertainty, they’re actually confident in
their incorrect answers.

This in mind, things don’t look good for traditional approaches to studying people’s behavior. When
you ask people questions about why they did something, it seems they unwittingly make up stories
and justifications. And perhaps more concerning for researchers, they’re confident in the story they
made up!

This may be hard to believe. But researcher Petter Johansson and his colleagues got a little more
extreme to prove the point.¹⁰ They showed participants two female faces and asked which face was
more attractive. After the participant pointed to a face, the researcher secretly swapped photos to
the one they didn’t choose (think David Cooperfield). Then, the participant was asked to explain why
they chose the face (which they didn’t actually pick). A majority of participants didn’t even notice the
face swap. And furthermore, they went on to explain in great detail why they preferred that face.Most
participants made up the reasons—good reasons—for liking a face that they didn’t actually choose.
Only a third of people realized they had been tricked.

Jonathan Haidt, Dan Dennet and Robert Kurzban give this phenomenon a name — they call it the
“Press Secretary.”¹¹ The brain’s Press Secretary must, like a president or prime ministers’ press
secretary, explain our actions to other people. “I don’t know” is simply not an acceptable answer.
Many times the brain’s Press Secretary is correct, but other times, it’s just guessing, piecing together
clues in order to give a plausible answer. The tricky thing is, we don’t know when our Press
Secretary is using facts and when it’s just guessing.

This is not a new theory. In 1931, Norman Maier asked participants in his University of Michigan lab
solve a puzzle most people were stumped by. When participants were out of ideas on how to solve
the problem, Maier would ‘accidentally’ brush a rope, part of the problem’s solution, and set it
swinging. Maier’s seemingly innocent gesture triggered participants to find the right answer. When
asked, however, how they ended up solving the riddle, they failed to mention Maier’s move.¹² In
1977, the Press Secretary concept continued to popularize when Nisbett and Wilson put forth a
seminal paper that argued people may be “unaware of the existence of a stimulus that importantly
influenced a response”, or plainly put, people have no idea what causes them to act in a certain way.

Conclusion? We just can’t ask “Why”

Your customers (scratch that, all of us!) are not aware of all the forces at work on us and thus
answering questions about them is almost laughable. One more example that should hit home.

When buying a new home, people don’t realize how much they’re influenced by the price of houses
in the city they are moving from.¹⁴ Regardless of income, if someone moves from San Francisco (a
super expensive city) to another city with a lower cost of living, research shows that they will have a
high reference point on what a house actually costs. This high reference drives them to buy a more
expensive home than if they’d moved from a cheaper city.

However, when asked why you bought your home,

our studies show people cite their income, style
aesthetic or commute length. People do not say:
“because I was anchored high.” In this case, it’s
easier to see the hilarity of asking people to
articulate their motivations—it’s unlikely people
even know what a reference price is, thus how
could they say it’s the primary influence?

We’re blind to how much our environment affects us, making answering questions about it nearly
impossible.

Let’s review:

1. Your customers tend to make up stories to explain their behavior. This is the Press
Secretary.
2. These stories tend to attribute behavior to attitudes, beliefs and preferences (e.g. “I care
about my future”).
3. The stories tend to discount the choice environment’s influence on behavior (e.g. we don’t
say, “the default made me do it”)¹⁵
4. We’re not aware that any of this is going on. If anything, we are overconfident in our ability to
reflect accurately on our behavior.
When you are seeking to understand customer behavior, you cannot rely solely on what people tell
you. In fact, you may be misled if you do. Instead, you need to study what people actually do – vs
what they say they will do.
References
[1] Multiple Choice Survey demographics: (454 respondents: open response) 48% female, 72%
White, 46% between 18–34, 41% less than $35,000 income, 60% employed full time. Open Text
Survey demographics: (446 respondents: multiple choice): 36% female, 75% White, 57% between
18-34, 40% less than $35,000 income, 69% employed full time, Combined:(900 respondents): 42%
female, 73% White, 51% between 18-34, 41% less than $35,000 income, 65% full time.
[2] 88% said “Having money in the future is important” for open text and 85% said this for multiple
choice. 60% said “Not enough money to save” for open text and 67% said this for multiple choice.
[3] Beshears, J., Choi, J. J., Laibson, D., & Madrian, B. C. (2009). “The Importance of Default
Options for Retirement Saving Outcomes: Evidence from the United States.” In Social Security
Policy in a Changing Environment (pp. 167–195). University of Chicago Press.
[4] This represents the combined:(900 respondents Mturk survey: 42% female, 73% White, 51%
between 18-34, 41% less than $35,000 income, 65% full time. “It’s the right thing to do” was 88% for
open text and 84% for multiple choice (N= 474) “I don’t feel comfortable” was 18% for open text and
41% for multiple choice (N= 405). Filtered out “I don’t know”.
[5] Tversky, A., & Kahneman, D. (1974). “Judgment Under Uncertainty: Heuristics and Biases.”
Science, 185(4157), 1124–1131.
[6] Morewedge, C. K., & Kahneman, D. (2010). “Associative Processes in Intuitive Judgment.”
Trends in Cognitive Sciences, 14(10), 435–440.
[7] Nisbett, R. E., & Wilson, T. D. (1977). “Telling More than We Can Know: Verbal Reports on
Mental Processes.” Psychological Review, 84(3), 231.
[8] Response breakdown: For the question on organ donation, 11.49% say they are “Not at all
confident” or “Slightly confident” about whether a default would impact their decision. For the
question on retirement savings, 13.51% say they are “Not at all confident” or “Slightly confident”
about the impact of a default.
[9] Response breakdown: 73% for organ donation and 67% for retirement
[10] Johansson, P., Hall, L., Sikström, S., & Olsson, A. (2005). “Failure to Detect Mismatches
Between Intention and Outcome in a Simple Decision Task.” Science, 310(5745), 116–119.
[11] The Righteous Mind: Why Good People Are Divided by Politics and
Religion, https://righteousmind.com/author/jonathan-haidt-2/page/11/
[12] Maier, 1937. “Reasoning in Humans. II. The Solution of a Problem and Its Appearance in
Consciousness.”
[13] Festinger, Leon, 1919–1989 (1962). A Theory of Cognitive Dissonance. Stanford, California.
ISBN 0804709114. OCLC 921356
[14] Simonsohn, U. and Loewenstein, G. (2006), “Mistake #37: The Effect of Previously Encountered
Prices on Current Housing Demand*.” The Economic Journal, 116: 175–199. doi:10.1111/j.1468–
0297.2006.
[15] Ross, L., & Nisbett, R. E. (1991). McGraw-Hill Series in Social Psychology. The Person and the
Situation: Perspectives of Social Psychology. New York, NY, England: Mcgraw-Hill Book Company.