MN-416-E

February 2023

How to Conduct a Good Experiment Online

Authorised for educator review use only by Fayola Nicholas, Arthur Lok Jack Graduate School of Business. Expiry date 4-Apr-2024
Ewa Kochanowska
Elena Reutskaja

Usage permitted only within these parameters otherwise contact [email protected]

Collecting your own data is a great way to gain insight into a topic of interest. Whether you want
to test a new idea before implementing it, measure your customers’ attitudes toward a new
product, or perhaps find the best way to convey your brand values, experimentation can show
you what works and what doesn’t. Experiments can also indicate which of several ideas works
best. With so many options available for online data collection, the answer to your next business
question is, quite literally, at your fingertips.
Educational material supplied by The Case Centre
Copyright encoded A76HM-JUJ9K-PJMN9I

Conducting a good online experiment, however, is both an art and a science. In this note, we
discuss the most common steps in running an experiment online, from designing your study
through recruiting participants to collecting your data. This is by no means an exhaustive list,
and the exact steps you need to take will highly depend on the specifics of your particular study,
but it covers most of the points you need to consider when collecting data online.

1. Ethical Considerations
Before you begin designing your study, it is important to keep ethical considerations in mind when
running experiments. This involves following a set of guidelines and principles to ensure the experiment
is conducted in a manner that protects the rights, welfare, and dignity of human participants.
Below are some key principles that you should take in account:
1. Obtain informed consent: Participants must be informed of the purpose and procedures
of the study, the risks and benefits involved, and their right to withdraw from the study
at any time.
2. Protect privacy and confidentiality: Participants' personal information and identities
must be kept confidential, and their privacy must be protected.
3. Minimize harm: You should ensure that participants are not subjected to physical or
psychological harm during the experiment.

This technical note was prepared by Ewa Kochanowska, doctoral student, and Professor Elena Reutskaja. February 2023.

Copyright © 2023 IESE. To order copies contact IESE Publishing via www.iesepublishing.com. Alternatively, write to
[email protected] or call +34 932 536 558.
No part of this publication may be reproduced, stored in a retrieval system, used in a spreadsheet, or transmitted in any form
or by any means - electronic, mechanical, photocopying, recording, or otherwise - without the permission of IESE.

Last edited: 28/2/23

Distributed by The Case Centre North America Rest of the world
www.thecasecentre.org t +1 781 239 5884 t +44 (0)1234 750903
case centre All rights reserved e [email protected] e [email protected]
 MN-416-E How to Conduct a Good Experiment Online

4. Ensure fairness: Ensure that all participants are treated fairly and that biases are avoided
in the selection and treatment of participants.
5. Monitor and report adverse events: You should monitor participants for any adverse
events that may occur during your experiment.
Following these guidelines will help you conduct experiments in an ethical manner and ensure
that your participants are treated with respect, dignity, and fairness.

2. Research Design

Authorised for educator review use only by Fayola Nicholas, Arthur Lok Jack Graduate School of Business. Expiry date 4-Apr-2024
A good research design is the most important part of any study. It determines everything else
down the line: your participants’ user experience, the quality of your data, and whether the type
of data you collect actually answers your question. If your study is poorly designed, there is no
going back to fix things, so you’ll have to start over from the beginning. Therefore, it is crucial to

Usage permitted only within these parameters otherwise contact [email protected]

spend a substantial amount of time up front to think through your study’s design in depth.
Designing a good experiment probably consumes more time and effort than any other part of this
market research method. Research design is an incredibly broad topic, and discussing all its aspects
is beyond the scope of this note, but the following general principles apply to every online study.

2.1. Research Question

Educational material supplied by The Case Centre
Copyright encoded A76HM-JUJ9K-PJMN9I

A good study design starts with a well-defined research question. To illustrate, we use a running
example throughout this note. Let’s say you run a coffee shop and want to reduce the use of
disposable cups by introducing your own branded reusable coffee cup for sale. Perhaps you want to
know whether people will be more likely to buy this cup if the impact of using it is presented in terms
of losses instead of gains; for example, every time customers do not bring the cup, you charge them
30 cents for a disposable cup (framed as a loss of money), while every time they do bring the cup,
you give them a 30 cent discount (framed as a monetary gain). Your research question will usually
(but not always) be structured in terms of some variable (x) influencing some other variable (y).1 We
will call variable x an independent variable and variable y a dependent variable. Both x and y should
be specific and well defined. In our example, x is how the impact of the reusable cup is framed (loss
vs. gain) and y is the likelihood of buying the cup. The full research question will be, Are people more
likely to buy the reusable cup if the impact of using it is framed in terms of losses or gains?

2.2. Hypotheses
Now that we have a research question, we can form a testable hypothesis. This takes the form
of a statement rather than a question and makes a prediction about the result you expect to
see. Hypotheses can be directional or nondirectional. A nondirectional hypothesis simply states
that there will be a difference between the two versions you are testing. In our example, a
nondirectional hypothesis might take the following form:
H1: The likelihood of buying the reusable cup will differ between people who see its impact
framed as a loss and those who see its impact framed as a gain.

1 A question concerning how x influences y is a relationship-based research question, i.e., one that examines the relationship

between two variables. Other common types of research question include descriptive research questions (e.g., How
regularly do people bring reusable cups to coffee shops?), and comparative research questions (e.g., What is the difference
in reusable cup usage between people under and over 40?).

2 IESE Business School-University of Navarra

 How to Conduct a Good Experiment Online MN-416-E

You’ll notice that we did not specify which group will show a higher likelihood of buying the cup,
merely that there will be a difference between the groups. In many cases, you may not be able
to make an informed guess about which direction your results will go. This is when
nondirectional hypotheses are useful. In many situations, however, you have a hunch about
what your results will be, and you just need data to test it. For example, research in psychology
and behavioral economics suggests that people are loss averse, meaning that they prefer
avoiding losses to acquiring gains of equivalent value. In other words, the pain of losing $10 is
psychologically more powerful than the satisfaction of gaining $10. With this in mind, we might
expect that framing the impact of using the reusable cup as a 30 cent loss of money will be more
effective, so we can write a directional hypothesis as follows:

Authorised for educator review use only by Fayola Nicholas, Arthur Lok Jack Graduate School of Business. Expiry date 4-Apr-2024
H1: People for whom the impact of the reusable cup is framed as a loss will be more likely to
buy it than those for whom its impact is framed as a gain.
Every hypothesis, directional or not, comes with its opposite: the null hypothesis (or H0). In a null
hypothesis, you state that whatever you predicted in your experimental hypothesis will not be

Usage permitted only within these parameters otherwise contact [email protected]

the case. Our example would propose the following null for the nondirectional hypothesis:
H0: There will be no difference in the likelihood of buying the reusable cup between people
who see its impact framed as a loss and those who see its impact framed as a gain.
The following represents the null for the directional hypothesis:
Educational material supplied by The Case Centre
Copyright encoded A76HM-JUJ9K-PJMN9I

H0: People who see the impact of the reusable cup framed as a loss will not be more likely to
buy it than those who see its impact framed as a gain.
Having a null hypothesis is important if you want to run a scientifically sound experiment. The
idea behind it involves discovering something by disproving the opposite, as it is easier to use
data to disprove than to prove something. A much-used analogy is that of a black swan. The
hypothesis that all swans are white is easier to disprove by finding a single black swan than by
examining every swan on the planet to see whether they’re all white.
The null hypothesis, then, is the state of the world you believe in until you find convincing
evidence to the contrary. In our example, if the data you collect reveal that people who see the
impact of the reusable cup framed as a loss indeed have a higher likelihood of buying it, you can
reject the null hypothesis and conclude that your data support H1. If, however, the data reveal
no difference between the two framing conditions, you cannot reject the null hypothesis. Note
that this does not necessarily mean that framing has no effect on the likelihood of buying the
cup; a lack of evidence means only that you haven’t proven that the effect exists. It may still
exist, but your experimental data have failed to reveal it. Another study might generate data
indicating the framing does have an effect, thereby rejecting the null hypothesis, but we will
continue to believe in the null until we find that evidence.

2.3. Measurements
Now that we have a pair of testable hypotheses, H0 and H1, we need to define how variables x
and y will be measured. The specifics very much depend on what you’re researching, but it is
important that you define exactly how your variables are measured so that the collected data
will answer your research question. For the x variable in our example, you can design your
experiment so that customers are randomly divided into two groups. Half the participants are
assigned to the “loss condition” group (who will see the cup described in terms of a 30 cent loss),
while the other half are assigned to the “gain condition” group (who will see the cup described

IESE Business School-University of Navarra 3

 MN-416-E How to Conduct a Good Experiment Online

in terms of a 30 cent gain). It is important to assign people to the conditions randomly, so that
other factors that might affect cup purchases are equal between two groups (e.g., day of the
week, weather, time of day). For example, it might be that you give a loss description to every
odd-numbered respondent who enters your experiment and give the gain description to
every even-number respondent. As the assignment is done randomly, one would expect that
both groups experience similar external factors that influence customers identically. Thus, the
only variable that differs between these two groups is x, your independent variable, and, if you
find a difference between the two groups, you can conclude that it is due only to variable x and
not to any other external factors, which are similar across the two groups.

Authorised for educator review use only by Fayola Nicholas, Arthur Lok Jack Graduate School of Business. Expiry date 4-Apr-2024
For the y variable, you could ask people how likely they would be to buy the cup using, say, a
scale of 0 (not likely at all) to 100 (extremely likely), or you could actually let people buy the cups
and measure the purchase rate for each group. Whatever your variables, make sure you phrase
your experimental questions in such a way that people understand what you are asking, that
different groups are identical in everything but your independent variable x, and that the

Usage permitted only within these parameters otherwise contact [email protected]

experiment gives you the data you need.

3. Analysis Plan
A well-designed study should also come with an analysis plan in which you specify exactly how
you will analyze your experimental data to test your hypotheses. It may seem premature to think
Educational material supplied by The Case Centre

about analysis before even collecting any data, but doing so will provide a check of your research
Copyright encoded A76HM-JUJ9K-PJMN9I

design and ensure that the resulting data structure will be able to answer your research
question(s). Thus, using a “backwards” research method—first considering how you will use your
data, how the data should look, and which analysis method you will adopt, all before even
designing the study—is a good strategy that could save you a great deal of effort and money.
You might draft an analysis plan based on the data you expect to get from your study design, only
to realize that it doesn’t actually answer your questions. For example, let’s say you also want to
know how big a discount (or charge) you should introduce for bringing (or not bringing) the reusable
cup at your café. The existing study design, which has a fixed 30 cent discount/charge, will not
answer that question. Thinking about the analysis before you actually conduct the study allows you
to go back to the design and revise it to meet your requirements. For example, you could introduce
additional conditions (in which some customers see the impact of the cup framed as a 20 cent
loss/gain and others as a 40 cent loss/gain) and test whether this changes the likelihood of buying
the cup. Of course, once you run the study and get to play with the data, you will inevitably do some
secondary analysis that was not in your original analysis plan, but you should definitely specify at
the start what analysis you will perform to test your main hypotheses.
The analysis plan is also a good place to specify any data exclusions up front. Examples of data
you may wish to exclude include:
• incomplete responders
• straightliners (people who give identical answers to a battery of questions that have the
same response scale, e.g., answering “Strongly agree” to every question in an attempt
to quickly click through the survey)
• outliers by some metric (e.g., those whose response to a question is more than three
standard deviations below or above the sample mean)
• attention check fails (more on this in part 5 below)

4 IESE Business School-University of Navarra

 How to Conduct a Good Experiment Online MN-416-E

Specifying data exclusions up front shows others that you did not manipulate the data post hoc
to fit your predictions. For example, specifying how outliers will be defined a priori means that
you won’t be tempted to reclassify unusual responses as outliers just because they fall outside
what you predicted. Of course, you can’t foresee and plan for every data exclusion rule, but you
should pre-specify any hard criteria that you definitely want your data to meet.

4. Stimuli
Now that you have an overall study design and analysis plan, you can consider how you want
your experiment to look by designing experimental stimuli. This may include:

Authorised for educator review use only by Fayola Nicholas, Arthur Lok Jack Graduate School of Business. Expiry date 4-Apr-2024
• writing questions your participants will answer
• choosing response scales
• preparing instructions, written descriptions of tasks, etc.

Usage permitted only within these parameters otherwise contact [email protected]

• selecting any visual, auditory, or other stimuli
When deciding how you want your study to look, it may even help to sketch it out with pen and
paper, especially for more complex designs that include visual or interactive stimuli.
Depending on what you are testing, there may be standardized tasks or questionnaires that you
Educational material supplied by The Case Centre

can adapt for your study. For example, existing experimental paradigms, such as the ultimatum
Copyright encoded A76HM-JUJ9K-PJMN9I

game, dictator game, or prisoner’s dilemma, can be used to assess people’s decision-making
strategies and attitudes toward cooperation and competition. Similarly, you can use standardized
questionnaires to assess measures such as personality traits, risk attitudes, tendencies to engage
or not engage in various behaviors, etc. For visual and auditory stimuli, a number of existing stimuli
databases offer standardized content at no cost (e.g., video clips or pictures of faces, objects,
scenes, and the like). When designing your study, it is always good to check whether any existing
measures or stimuli that have already been used and validated by other researchers can be
adapted for your purpose, ensuring that the scales and variables you use actually measure the
construct you want to measure.

5. Program the Experiment

Now you have everything you need to actually program the experiment. There are a number of
platforms available for online data collection, and we briefly describe the three most popular below.

Qualtrics
This is probably the most popular platform for collecting data online. Its focus is predominantly
survey-type data (multiple choice answers, free text, rating scales, etc.), but you can also present
media to participants and set up randomization rules to create experimental conditions by
assigning participants different sets of questions to answer or various stimuli to hear, to read, or
watch. Qualtrics has a graphical user interface, so no programming knowledge is required to get
a study up and running. For relatively simple study designs, it is a great tool to quickly collect
data, but it is not designed for complex experiments. For anything with a more complex design
(for example, precise timing of stimuli presentation, collecting reaction time data, or allowing
participant interaction), other tools are more suitable.

IESE Business School-University of Navarra 5

 MN-416-E How to Conduct a Good Experiment Online

Gorilla
This online experiment builder can be a good option for more intricate designs that Qualtrics cannot
handle. In addition to survey questions, you can program more complex interactive tasks as well as
game-like experiments. Many popular tasks used in psychology, economics, and other fields have
already been programmed by other researchers, so you can download them and adapt them to
your needs. Gorilla supports several input methods (mouse, keyboard, voice recording) and offers
mouse-tracking and even basic eye-tracking. It also offers custom stimuli presentation times and
the ability to record reaction times. The graphical user interface, while more complicated than that
of Qualtrics, requires no programming knowledge to run experiments.

Authorised for educator review use only by Fayola Nicholas, Arthur Lok Jack Graduate School of Business. Expiry date 4-Apr-2024
Inquisit Web
This is the most advanced of the platforms reviewed here. Inquisit Web allows you to create an
online experiment that is the most comparable to an in-person lab study. Instead of opening the

Usage permitted only within these parameters otherwise contact [email protected]

experiment in a browser tab, participants view the study in full screen mode, so it is a good
option if you are concerned about distractions. It offers the highest precision for stimuli
presentation times and participants’ reaction times (down to millisecond accuracy). For these
reasons, it is the platform of choice for complex experimental paradigms that demand maximum
customizability and high control of tasks. There is no graphical user interface, so you must learn
the basics of coding in Inquisit to run studies. However, as with Gorilla, many popular tasks have
Educational material supplied by The Case Centre
Copyright encoded A76HM-JUJ9K-PJMN9I

been pre-programmed by other researchers, so, depending on what you are testing, there may
be templates available to download and adapt to your needs.
It is important to note that, while setting up an account with some software is free, actually
collecting the data using such tools is not free. Users should generally pay for the use of such
software or buy a long-term license to use these tools.
There are other platforms beyond the three reviewed here as well as the option of coding and
hosting the experiment yourself using bespoke software. Whichever option you choose, several
things demand consideration when you program your study. You must determine the order of
the questions and tasks presented to participants and make sure the experimental flow gives
them the intended user experience. You will need to set up randomization by deciding who sees
which version of the experimental stimuli. In our coffee shop example, this would mean setting
it up so that half the participants see the impact of the reusable cup framed as a loss and the
other half as a gain. It is also good practice—especially for longer studies—to include a few
attention checks in your experimental flow. Attention checks are questions or tasks that are
designed to determine whether a participant is paying attention to your study or is being
distracted by other things. They are a good way to identify unmotivated participants (or bots!),
who can harm your data quality by speeding through your study, not reading instructions
carefully, answering randomly, or skipping questions. Examples of attention checks include:
• factual attention checks that ask participants a simple question with only one correct
answer (e.g., Which of the following is a fruit? a. milk, b. pasta, c. apple, d. bread)
• instructing the respondents to pick a specific answer (e.g., To ensure you are paying
attention, please answer “Strongly agree” to this question)
• free-text attention checks that require participants to type in their response (e.g., Please
enter the word “apple” into the box below)

6 IESE Business School-University of Navarra

 How to Conduct a Good Experiment Online MN-416-E

The free-text attention checks are especially useful for telling apart bots and humans, although
you can also embed a CAPTCHA test in your experiment for that purpose. 2
After you have set up your attention checks, you need to determine the criteria for excluding
participants based on their answers. It is usually advisable to give participants a little room for
error, as everyone can make a mistake and answer a question incorrectly. For example, you can
decide to exclude those participants who fail, say, three or more of the four total attention
checks, thereby allowing those who happened to press the wrong button on one of the
questions but are generally paying attention.
Once you have programmed a version of your study that pleases you, you need to test it, a very

Authorised for educator review use only by Fayola Nicholas, Arthur Lok Jack Graduate School of Business. Expiry date 4-Apr-2024
important step that should not be skipped. Testing involves going through the study as a
participant, timing how long it takes you to complete the study, and checking whether the
randomization works and all the instructions and task descriptions make sense. It is also good
practice to ask a few other people to do the same and give you feedback, and it is especially
good to get feedback from people who are not familiar with your study, as they will approach it

Usage permitted only within these parameters otherwise contact [email protected]

with the same level of prior knowledge as your participants. When testing your experiment,
don’t forget to look at the data output generated by your test participants to make sure that all
the variables you need are being recorded properly.

6. Participants
Educational material supplied by The Case Centre
Copyright encoded A76HM-JUJ9K-PJMN9I

Now that your online experiment is ready, all that’s left to do is find people to participate in it.
The first step is defining the population of interest. In our coffee shop example, this could mean
the general population or, if you want to be more specific, coffee drinkers or people who visit
coffee shops. Note, however, that the more specific you get when defining your population of
interest, the trickier it becomes to source your sample. There are a number of platforms and
panels for sourcing participants online, the most popular being Amazon Mechanical Turk,
Prolific, CloudResearch, Qualtrics, and Dynata. In addition to demographic information about
their participant pool, they sometimes collect data on other variables that may be of interest,
such as income, education, consumption habits, etc. It is important to note that data quality can
vary on such platforms, so, if you choose to run your study there, consider implementing
adequate attention checks and bot checks like the ones discussed above.
Another option for those affiliated with a university is to source participants from undergraduate
research pools comprised of students enrolled in courses offering course credit for participation
in research studies. This can be a good source of sample if you are not particularly concerned
about data from a specific demographic or if college-aged adults are your population of interest.
If you need data from a more specific population, however (e.g., small business owners or
graphic designers), you may need to get more creative when sourcing your sample. You can
advertise on classified ad websites, such as Craigslist, or on social media. You can also reach out
to people who manage newsletters that target those specific individuals and ask whether would
include an ad for your study in their mailout. Sometimes, it is possible to partner with companies
to survey their employees. For example, if your study is about testing an idea for a new human
resources (HR) software that you hope to market, you might be able to use your network to
reach out to a few HR departments and have their employees complete your study—especially
if the output of the study benefits the employees and the company. Finally, you could also go
2 You
often encounter a Completely Automated Public Turing test to tell Computers and Humans Apart (CAPTCHA) online
when you are asked to “prove that you are a human.”

IESE Business School-University of Navarra 7

 MN-416-E How to Conduct a Good Experiment Online

the old-fashioned way and post printed ads in public places, on community noticeboards, etc. In
our reusable coffee cup example, this could be a particularly good option, as you could advertise
in coffee shops to reach your specific population of interest.
No matter how you decide to source participants, make sure they represent the population you
are interested in. When you collect your data, your sample may well be unrepresentative of the
country or city you live in, but the participants should be representative of the population you
are interested in. In our example above, the customers of your store are the sample you are
interested in. If these are young college students with some specific habits, then the participants
of your experiment should represent exactly this particular group of people for the data to yield

Authorised for educator review use only by Fayola Nicholas, Arthur Lok Jack Graduate School of Business. Expiry date 4-Apr-2024
actual insights for your particular business.
Once you have chosen how to source your sample, you must decide on the compensation for
participating in your study. Compensation is usually in the form of cash or gift cards (or course
credit in the case of undergraduate research pools) but can also be other gifts, product
giveaways, or nonmonetary incentives, depending on your particular study. When choosing the

Usage permitted only within these parameters otherwise contact [email protected]

level of compensation, you need to determine a fair amount for the time and effort your
participants will spend on your study. While your level of compensation should be fair and
proportional to the work required, it is generally not advisable to focus on compensation as the
primary means of enticing potential participants, because setting unusually high incentives may
unduly influence respondents to take part or remain in a study, thus potentially taking advantage
of financially vulnerable populations and/or compromising your data quality. The main focus for
Educational material supplied by The Case Centre

the respondent should be participating in research that they and others can potentially benefit
Copyright encoded A76HM-JUJ9K-PJMN9I

from while being compensated for their time and effort and reimbursed for any additional
expenses, if applicable.

7. Analyzing and Interpreting Your Data

You have run your study and it’s time for the best part: analyzing your data and interpreting the
results. Because you have thought through your study design and tested it up front, you should
be getting the right data structure with all the variables of interest recorded properly. The first
step, then, is to do some data cleaning. This involves checking the data quality and implementing
any data exclusions that you pre-specified in the analysis plan (incomplete responses,
straightliners, attention check fails, etc.). For the next step, it is always useful to look at your
data descriptively: calculate means and make tables, charts, etc. This will give you a general idea
of your data’s look and feel.
Now, you’re ready to analyze your data. You already have an analysis plan, so this will be
straightforward; you perform the necessary analyses to determine whether or not you can reject
the null hypothesis and conclude that your data support H1. Don’t forget to visualize your results,
as it’s a great way to make findings impactful and easy to understand. Did the study answer your
research question? Are there any secondary analyses you can perform that you didn’t think of
beforehand but that provide additional insight on your topic of interest? Did the data give you
some answers but also generate new questions, inspiring you to conduct a follow-up experiment
with your new knowledge? Are there any actions you can take or business decisions you can
implement based on your findings? Whatever your output, remember that research is an
iterative process, and cycles of testing and implementation are often the norm when backing up
business decisions with empirical data. Happy testing!

8 IESE Business School-University of Navarra