Test and learn
Test and learn is a set of practices followed by retailers, banks and other consumer-focused companies to
test ideas in a small number of locations or customers to predict impact. The process is often designed to
answer three questions about any tested program before rollout:
1. What impact will the program have on key performance indicators if executed across the
network or customer base?
2. Will the program have a larger impact on some stores/customers than others?
3. Which components of the idea are actually working?
Historical context
Test and learn has been systematically applied as far back as 1988 by Capital One. Capital One has been
aggressive about testing since the firm was founded, testing everything from product design to marketing to
customer selection to collection policies.[1] In a single year, the company performs tens of thousands of
tests, allowing it to offer thousands of different types of credit cards to customers, based on the knowledge
gained from their tests. Richard Fairbank, CEO of Capital One, called test and learn "a marketing
revolution that can be applied to many businesses".[2]
Examples
Retail banks running different commercials in different markets.
Fast food restaurants launching a new menu item or product for a limited time in select
locations
Big-box retailers evaluating “store of the future” programs
Companies evaluating pricing strategies
Retailers testing merchandising layouts across their door base
The UK government, through research projects such as the College for Teaching and
Learning's Close the Gap: Test and learn [3]
Test and learn in practice
Woolworths uses a test-and-learn approach to identify which stores should undergo
renovations. [4]
Wawa Food Markets, a Mid-Atlantic convenience store chain, uses Test and Learn across
several aspects of its business:[5]
Measuring the effect of introducing a new product
Determining the impact of adding additional staff
Understanding how the opening a new location would impact existing nearby locations
Subway ran a promotion that lowered the price of its foot-long sandwiches to $5. Before
deciding to launch the promotion nationally, Subway ran a test that compared locations
offering the $5 foot-long sandwiches to a group of similar control restaurants that were not
offering the promotion.[1]
Kraft Foods, a manufacturer and distributor of packaged foods, uses testing to optimize the
placement of its products on grocery store shelves:[6]
Determining the effect of a coffee aisle re-design on sales of premium coffees
Measuring the impact on product sales of a change to in-store sales representatives'
responsibilities
Pier 1 Imports, a recently bankrupt retail chain specializing in home decor and furnishings,
uses testing to measure the effectiveness of search advertising campaigns.[7]
Test and learn is also methodically used online to test impact of various web page layouts
on user behavior. Tools such as Google Website Analyzer provide every site with the ability
to test various elements and their effect on user engagement.
Test and learn software
There are a variety of software tools available today to support systematic testing within an organization.
However, there is currently no one software package that covers all types of tests, and in some cases
significant knowledge of statistics is still required for effective analysis.[1]
Many companies, including Capital One and eBay, have developed experiment management software
within the company. Alternatives to internal solutions include broad statistical analysis software tools such
as SAS or test and learn focused software from Applied Predictive Technologies, Trial Run and
MarketDial. There are also a variety of more narrowly tailored products, focused on particular problems or
fields. In web analytics, for example, firms such as Omniture, WebTrends, and Google have created
specialized software.[1]
Criticism
Companies wanting to conduct Test and Learn must mobilize, track and analyze site and customer data at
very granular levels. This can often be a massive and expensive undertaking. Increasingly however, an
exponential increase in computing power, and corresponding decrease in its cost, have made testing more
accessible. In the Harvard Business Review February 2009 article "How to Design Smart Business
Experiments",[1] Thomas Davenport discusses the Test and Learn Process, and describes how several
companies have overcome these and other barriers to testing.
See also
Clinical trials
Scientific method
Design of experiments
Evidence-based management
Evidence-based practices
Business process reengineering
A/B testing
References
1. Davenport, Thomas H. (February 2009). "How to Design Smart Business Experiments" (http
s://hbr.org/2009/02/how-to-design-smart-business-experiments). Harvard Business Review.
2. Fishman, Charles (1999-04-30). "This is a Marketing Revolution" (http://www.fastcompany.c
om/magazine/24/capone.html). Fast Company. Retrieved September 22, 2009.
3. "Closing the gap: Test and learn" (https://www.gov.uk/government/publications/closing-the-g
ap-test-and-learn).
4. "High-tech secret behind the future of your local Woolworths" (https://www.9news.com.au/tec
hnology/woolworths-refurbished-stores-decided-by-computer-software-coles-stores/6d512a
df-baa4-4320-b817-457a2f8d5897). 9News. January 23, 2020. Retrieved July 6, 2020.
5. Fleenor, D. Gail (June 2009). "These Tests Are Positive" (https://web.archive.org/web/20091
012122924/http://www.stores.org/InformationTechnology/2009/06/Edit03.asp). STORES
Magazine. Archived from the original (http://www.stores.org/InformationTechnology/2009/06/
Edit03.asp) on 12 October 2009. Retrieved 18 September 2009.
6. Angrisani, Carol (December 22, 2008). "Coffee Aisle Revamp Lifts Sales" (http://subscribers.
supermarketnews.com/Grocery_Center_Store_Brands/coffee_aisle_revamp_1222/).
Supermarket News. Retrieved September 22, 2009.
7. Wong, Elaine (June 24, 2009). "Case Study: How Search Ads Helped Pier 1 Stay Afloat" (htt
p://www.brandweek.com/bw/content_display/news-and-features/direct/e3i048f01beefa084a
38c84fb0602a1b651). Brandweek. Retrieved September 22, 2009.
Retrieved from "https://en.wikipedia.org/w/index.php?title=Test_and_learn&oldid=1087903566"
