Marketing

by Experiment
B y Yo r a m ( J e r r y ) W i n d

Adaptive experimentation
ensures that research is
fast and flexible.

10 Spring 2007
 A company sets a $100 million
marketing budget for a product launch. A year later, it has spent
the budget, and management is able to assess the results. Or is it?
It has detailed data on every aspect of the launch campaign, but it
still cannot answer a very fundamental question: Was this the optimal
budget? (What would have happened if the company had spent $50
million or $150 million?) With a one-time test, management has little
guidance in setting the next budget.
Hauke Dressler/Getty Images

marketing research 11
 tising, but what combination of “first life” and Second Life
Executive Summar y approaches are most effective?

Rather than seeking single answers to simple questions, In an environment of discontinuous change, research can
researchers must experiment with diverse strategies. answer questions like these only through trial and error.
Adaptive experimentation permits this process, but in
“Adaptive experimentation” is a philosophy that integrates
a systematic manner. Adopting the philosophy of adaptive
research and action. This approach is important in today’s experimentation is the best way to become a learning organi-
swiftly changing business environment, where researchers zation that continuously challenges its strategies, measures
must challenge their assumptions and devise fresh tactics. their impact, improves them, and encourages innovation.
By becoming advocates of adaptive experimentation and In this environment, the effectiveness of specific marketing
research approaches might erode over time, as the nonre-
faithfully implementing it, they can secure a seat at the table
sponse problem in surveys illustrates. With the proliferation of
and influence critical business decisions. telemarketing activities, and the volume of polling, some sub-
jects are just refusing to be interviewed. This problem is partic-
ularly troubling in critical segments such as busy physicians
Suppose that instead, management had set up a carefully and high net-worth individuals. As the value of certain tactics
designed experiment to test different levels of spending. In decline, researchers must find new ways to gain insights into
one market, it would invest at a level comparable to a $100 market needs, wants, preferences, and responses to marketing
million launch; in a second market, it would invest at a level strategies. To do this and learn about what works, they should
comparable to a $50 million launch; and in a third market, it experiment with diverse marketing research methods.
would invest at a level comparable to a $150 million launch.
At the end of the year, they would be able to compare the Adaptive Experimentation
results of the different markets, and then use this learning as Both the development of strategy and the design of market-
the foundation for the next set of experiments in the following ing research could benefit from adaptive experimentation.
year. If the $150 million budget level happens to emerge as the In contrast to discrete, one-shot tests and research, adaptive
hands-down best option, then the company could invest 80% experimentation is rigorous, multifaceted, and continuous. It
of its total budget in that strategy on the next round. But it can pursue several hypotheses simultaneously or sequentially,
still might devote the remaining 20% of its total budget to using the results of each round of testing as the foundation for
other experiments, looking at even higher or lower levels or the next experiment. Adaptive experimentation integrates
experimenting with other variables. research and action. It is not a technique, but a philosophy
A single test offers a single answer. The management that had and method for learning and managing evolving enterprises.
pursued the single budget for the product launch wasted a year Although serendipity can be important in developing ideas,
while discovering very little. Multiple strategies, as part of a rigor- adaptive experimentation is not random. It consists of care-
ous experimental design, offer opportunities for greater learning. fully designed experiments that test various marketing and
This approach of “adaptive experimentation” is particu- business strategies across different geographies and product
larly important in a rapidly changing business environment, categories. These strategies can include assorted advertising
where researchers should challenge assumptions about what levels, media mixes and messages, sales force sizes, organiza-
works and come up with other approaches. Because of market tions, pricing options, and many other dimensions.
shifts and imitation, a winning strategy might last for only Adaptive experimentation can start with a simple design
three to six months. This means that the experimentation must (e.g., testing marketing budgets), and then researchers add
be continuous to identify the next strategy. Changing technol- more complexity in subsequent rounds of experiments—such
ogy and business models raise new questions about current as exploring marketing mixes and other variables, and using
marketing thinking: Latin square or alternate experimental designs. Adaptive
experimentation can employ methodologies such as the
• If TiVo users are no longer looking at TV commercials, Multitrait-Multimethod Matrix (MTMM), as part of the
then how can companies best get their advertising messages portfolio. D. Campbell and D. Fiske developed the MTMM
across? approach to test the validity of various methods, such as
pen-and-paper tests, direct observation, and performance
• How can companies tap into the rising interest in social
measures. (Read their article, “Convergent and Discriminant
networking Web sites such as MySpace?
Validation by the Multitrait-Multimethod Matrix,” in the
• Does the existence of new media channels make a $2.6 mil- March 1959 issue of Psychological Bulletin.)
lion investment in a 30-second Super Bowl advertisement A rigorous design increases the efficiency and effectiveness
more or less valuable? of the experiments, reducing the number of them required to
gain insights. Rather than using full factorial testing,
• The expansion of simulated environments such as Internet- researchers can move to fractional factorial testing to identify
based Second Life has created parallel universes for adver- a subset of meaningful combinations. By using such efficient

12 Spring 2007
approaches, researchers can more easily engage in continuous design consultancy IDEO employs it to develop new products
testing and create a nimble marketing strategy. through what it refers to as “enlightened trial and error.” Yet
despite the successes in direct mail and new-product develop-
Strategy as Experimentation ment, researchers have underused adaptive experimentation.
Adaptive experimentation can offer a crucible for testing The time has come to expand the use of adaptive
both marketing strategy and new business models, as pioneer- experimentation. For instance, a pharmaceutical company
ing companies in diverse industries and contexts demonstrate. designed experiments to test the optimal level and mix of pro-
Researchers almost universally recognize the power of adaptive motional activities, increase patient compliance, connect with
experimentation in direct mail. The cost of designing tests is untapped physician segments, and achieve other objectives. To
very low, and they can easily measure outcomes. With large test its strategy for promotion, the company used a multicell
mailing lists, there are abundant opportunities to develop adaptive experiment. For 16 weeks, it used varying levels of
multiple experiments. This makes direct mail particularly detailing, sampling, advertising, and medical education in dif-
well-suited to low-cost experimentation. Direct marketers ferent regions. The company then adjusted the approaches in
test many approaches, see what messages and designs work each of the markets, based on the initial experiments—quickly
best, and then continue to refine their hypotheses on the identifying the most effective tactics for specific areas.
next round of tests. Companies can also test bolder strategies for new offerings
Early research with direct mail, for an insurance company, and revenue models, in an integrated fashion. A pharmaceuti-
found that it was able to triple its response rates through well- cal company wanted to experiment with a shift from a brand-
designed experiments. And a major financial services company centric strategy to a customer-centric one: from selling
used a set of adaptive experimentation to test hundreds of branded drugs for treating a single illness to addressing the
credit card offerings every month. It created tests focused on concerns of patients suffering from more than one ailment.
various messages and execution, then assessed the results and For example, many patients with diabetes might also have
designed new ones. problems with cholesterol.
Researchers have demonstrated the power of adaptive The experiments examined strategies that combined dia-
experimentation in other areas. For example, California-based betes with cholesterol, breast cancer with depression, and HIV
with depression. This led to a set of experiments with product prototype might tip the hand of the company to competitors,
and service offerings, information and education, psychology often undermining the advantage of the strategy.
and social support, business relationships, pricing and revenue Addressing real-world applications. Some have criticized
models, and customization—plus the creation of virtual com- marketing research approaches that fail to deliver real-world
munities. They also examined integrated marketing, localized results. Adaptive experimentation is grounded in actual imple-
and global strategies, and e-business opportunities. Through mentation, so results are more likely to be valid and generaliz-
careful experimentation, the company could test a unique way able in practice.
of looking at its business.
Experimentation in Research
Benefits of Experimentation Marketing researchers ought to see adaptive experimenta-
Adaptive experimentation offers many benefits over the tion as a key tool and as an effective philosophy for guiding all
more common implementation of a single strategy. research. It is also very useful in addressing complex decisions
Stimulating breakthrough ideas and a culture of innova- that do not lend themselves to standard marketing research.
tion. Adaptive experimentation sees the entire organization as With any given approach, researchers can use experimenta-
a laboratory, which promotes discontinuous thinking. This tion to refine it. Experimentation can improve the quality,
leads to breakthroughs in strategy, rather than minor tweaks effectiveness, and efficiency of research tools and approaches.
to the next ad campaign. When researchers see adaptive exper- For example, in using panels, which incentives are most useful,
imentation as a way of life, employees realize that it is all right which ways of identifying new panel members lead to the best
results, and how are online and offline panels different?
A set of experiments could lead to strategies for improving
M a n y c o m p a n i e s a re response rates or other outcomes.
Adaptive experimentation can test combinations of tools.
too timid in their thinking Researchers can experiment with different mixes of marketing
research tools, such as surveys based on convenience samples,
and initiatives. conjoint analysis, and tracking surveys. What combination
yields the best results for a specific company or research
challenge?
to fail. Failure is part of experimentation; the ability to fail is Adaptive experimentation can be a guiding philosophy for
vital to learning. Experimentation in a company without this research. It encourages researchers not to become overreliant
mind-set can be very disruptive. And without a culture of on a single research tool or set of tools. Instead, it promotes
experimentation, organizations might kill or not appreciate the development of a portfolio—and researchers continually
successful experiments because they represent a threat to the sharpen, test, and expand that with each round of experi-
status quo. ments. Researchers can apply this mind-set to all research
The sobering fact is that the leading companies in their activities.
industries don’t develop many innovations (e.g., the iPod came
from a computer company, not a music company). This is an Obstacles and Opportunities
indication that many companies are too timid in their thinking Given its benefits, why don’t researchers employ adaptive
and initiatives. They are not taking bold leaps to challenge the experimentation more broadly? There are a number of obsta-
views of their industries and come up with radical new ideas cles to adopting it. Cost is a concern, because it is typically
and approaches. Adaptive experimentation can help identify, more expensive to design and implement multiple strategies
develop, and implement these. than a single strategy. And experimentation requires a
Focusing on measurement and improvement. An attitude of process for measuring results. There is also resistance from
experimentation encourages measurement across the organiza- those involved in experimental cells that do not perform well.
tion, to understand results. Researchers create experiments To overcome such resistance, particularly in a sales force, the
with clear measurements of the effectiveness of the approaches solution might be to adjust incentives. For example, a com-
that they test, including return on marketing investment. pany can guarantee salespeople a level of income regardless of
Companies can also integrate the results of these experiments the results of their experimental cells. This protects them from
into corporate dashboards. This mind-set of measurement feeling penalized when the company places them in a cell it
leads to research-based actions. And advances in database expects to have lower performance (e.g., where marketing
marketing allow for better analysis of adaptive experimenta- spending is reduced). But this also raises the costs of imple-
tion’s results, at the individual and segment level. mentation.
Confusing competition. Unless competitors know the mas- In addition, companies face opportunity costs because of
ter design when a company is engaged in a number of experi- the potential for failure of experimental options. These costs
ments, they have no way of inferring the underlying strategies can be particularly great for experiments that are risky, such as
from the company’s activities and results. Competitors might using no advertising in a given cell. Some extreme experiments
be able to see the individual experiments, but still not know might even hurt overall brand equity. And certain strategies
where they are leading. In contrast, a typical test market or might have unintended negative consequences, so one should

14 Spring 2007
try to avoid such experimental options. In this case, Adaptive experimentation can also allow a large company
researchers need to take immediate action to protect the to act more like a small one. Quick, little experiments allow
brand—which can also be quite expensive. Furthermore, man- the company to fail fast and cheap and test innovative strate-
agement might be concerned about disruption of the business, gies without risking everything. Valid and reliable testing at
because companies can find it challenging to execute capably low cost also permits lower-level managers (particularly those
against the current strategy and experimental ones—while of regions or countries) to experiment without having to jump
continuing to test and learn. through bureaucratic hoops for approval. This streamlines the
But again, cost is not the only obstacle to adaptive experi- process of innovation, and allows for more widespread and
mentation. Perhaps the biggest barrier is the mind-sets of diverse experiments.
researchers and managers, which can limit their thinking and
creativity. For more information read The Power of Beginning to Experiment
Impossible Thinking (Wharton School Publishing, 2004) by Senior executives make many major business decisions
Yoram (Jerry) Wind and Colin Crook, with Robert Gunther. without the benefit of marketing research. This is unfortunate,
The current mind-set sees marketing research as a precise because marketing research is often directly relevant to
application of a set of tools, rather than a philosophy of inves- launching business models, growth plans, or other strategic
tigation. Management treats marketing research as a practical initiatives. With a focus on business challenges, adaptive
and applied engineering discipline. Although this engineering experimentation can raise the stature and usefulness of mar-
view is important, it is too narrow. Management needs to keting research within the organization.
observe marketing research through the lens of broader scien- With corporate demands for growth and innovation,
tific inquiry, in which researchers raise and test big questions. marketing and marketing research have to deliver more. At
Challenging the mind-set of continuing with the current, the same time, marketers have been asking for a seat at the
known, and comfortable is a must. table to influence key business decisions. Becoming champions
In applying adaptive experimentation, researchers need of adaptive experimentation and implementing it on an ongo-
to weigh the costs and added complexity against the results ing basis is the best way to ensure that they get this seat.
and benefits. Adaptive experimentation might be more costly Adaptive experimentation will make marketing research
in the short term, but it produces the most value in the long more applicable to top management wrestling with the com-
term. The benefits, as discussed, are compelling. It is time to pany’s major strategic issues. It can offer insights and answers
apply experimentation more broadly to the philosophy and for these decisions—such as whether to continue using TV
practice of marketing research. With advances in data avail- advertising, enter new distribution systems, or launch market-
ability and measurement, all organizations can adopt adaptive ing and cobranding initiatives. This creates an opportunity to
experimentation. expand the application and impact of marketing research,
Cost is a particular concern for small companies, which increase its value, and raise its stature in the organization.
might have limited resources to engage in adaptive experimen- This will also help bridge the silos of marketing and other
tation. But given (1) the risks of the wrong marketing and disciplines, creating a more integrated approach.
business strategy, (2) that they can rarely afford the time Looking at the organization more broadly, adaptive
involved in standard research, and (3) the likelihood that experimentation coupled with flexible corporate strategy is
larger competitors will move to preempt if their directions are an integral part of creating “adaptive enterprises.” In Adaptive
clear, small companies might have more compelling reasons to Enterprise (Harvard Business School Press, 1999), Stephan
employ adaptive experimentation than large firms do. Small H. Haeckel characterizes this as a shift from “make and sell”
companies, however, do have to be very attentive to reducing organizations to “sense and respond” organizations. The
risks and costs as they proceed. They can achieve this by effec- adaptive organization uses feedback loops for sensing and
tively managing the learning organization. interpreting, deciding and acting, and continuing to refine the
strategy and adapt to changing conditions.
With burgeoning new technologies and an avalanche
of data, the potential for new approaches to marketing
Custom Reprints research is enormous—as is the need to develop them. We
need to add adaptive experimentation to our portfolio of
Custom article reprints offer information at a glance at trade
approaches, and adopt it as a core philosophy for how we
shows, conferences, and seminars. They can also be a great way
conduct marketing research. We need to start experimenting
to share with your colleagues some of the cutting-edge research, now—and keep experimenting—to continuously improve
opinions, and insights featured in Marketing Research. our strategies. G
G

To order reprints of any article in Marketing Research,

Yoram (Jerry) Wind is the Lauder Professor and professor of
contact Sheridan Reprints at 800-352-2210. For subscription marketing at The Wharton School of the University of
information or to order single copies, call 800-262-1150. Pennsylvania in Philadelphia. He may be reached at
[email protected].

16 Spring 2007