Journal of Statistics and Data Science Education

ISSN: (Print) (Online) Journal homepage: www.tandfonline.com/journals/ujse21

Five Hands-on Experiments for a Design of

Experiments Course

Victoria Woodard

To cite this article: Victoria Woodard (2023) Five Hands-on Experiments for a Design of
Experiments Course, Journal of Statistics and Data Science Education, 31:3, 225-235, DOI:
10.1080/26939169.2023.2195889

To link to this article: https://doi.org/10.1080/26939169.2023.2195889

© 2023 The Author(s). Published with

license by Taylor and Francis Group, LLC.

Published online: 21 Apr 2023.

Submit your article to this journal

Article views: 2941

View related articles

Citing articles: 1 View citing articles

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=ujse21
JOURNAL OF STATISTICS AND DATA SCIENCE EDUCATION
2023, VOL. 31, NO. 3, 225–235
https://doi.org/10.1080/26939169.2023.2195889

Five Hands-On Experiments for a Design of Experiments Course

Victoria Woodard
Applied and Computational Mathematics and Statistics Department, University of Notre Dame, Notre Dame, IN

ABSTRACT ARTICLE HISTORY

In many collegiate level statistics courses, the focus of the learning outcomes is often on the analysis of Received June 2022
data after it has been collected. Students are provided with clean datasets from previous studies to practice Accepted March 2023
statistical analysis, but receive little to no application as to the amount of time and effort that goes in
to collecting good data. To account for these deficits at the author’s institution, a design of experiments KEYWORDS
Completely randomized,
course was created that provided students with a more hands-on learning experience to the statistical
Factorial; Matched pairs;
process, especially as pertains to data collection. This article focuses on five of the experiments that students Nested; Split plot
designed and implemented during the course, and some suggestions to instructors that may wish to use
these experiments in their own courses.

1. Introduction important statistical ideas” (American Statistical Association

2016, p. 18). As such, this course not only taught the stu-
For students graduating with degrees in statistics, being able to
dents how to design the experiments, it provided them with
just conduct a statistical analysis is not enough. It is also imper-
opportunities to implement their designs and collect data, then
ative that our students can work through the entire investigative
perform an appropriate statistical analysis on that data (Blades,
cycle, from posing a problem, to collecting data, conducting
Schaalje, and Christensen 2015). This process was done in an
the analysis and providing a solution to the problem (American
attempt to teach students how difficult, and potentially messy,
Statistical Association Undergraduate Guidelines Workgroup
data collection could be and ways to help ensure that data is
2014; American Statistical Association 2016). Lee et al. provide
clean after collection.
a comprehensive collection of researchers that have proposed
The aim of this article is to describe some statistical experi-
frameworks for what this process might look like (2022). While
ments that were used in this course that fostered active learning
each of these have their own nuances of this process, each
and gave students the opportunity to focus more on the Plan and
includes an element of planning and implementing data collec-
Data phases of the investigative cycle than may be available in
tion. For example, one of these frameworks, created by Wild
many traditional collegiate level courses.
and Pfannkuch, proposed the investigative cycle (the PPDAC,
Figure 1), which includes defining the problem, planning the
data collection, collecting the data, conducting the analysis, and
creating conclusions (1999). 2. Methods
However, in scanning the course descriptions for classes 2.1. Course Information
offered at the author’s home institution, and having informal
conversations with colleagues, it was found that most courses In the spring of 2021, the Design of Experiments course was
within the department emphasize methods of analysis and only offered as an elective topics course at a private university in
a few mentioned topics for appropriate data collection. As noted the Midwest. The learning outcomes for the course were as
above though, learning statistics is not just about being able to follows:
analyze pre-collected and pre-cleaned data. A critical step in
doing research and conducting a statistical analysis is designing • Understand principles of experimental design that can be
an appropriate experiment that will provide unbiased, authentic used to help answer scientific questions.
data (1999). • Know when it is practical to use different methods of exper-
As such, a design of experiments course was developed imental design.
to help students learn methods for both data collection and • Understand how to analyze and interpret data that has been
analysis. However, just learning about the methods without collected through a well-defined experiment.
practical application is not enough. The GAISE College Report • Know limitations of different types of study design.
recommends that statistics teachers should foster active learning • Possess an introductory understanding of technology used in
as this “allows students to discover, construct and understand statistical analysis.

CONTACT Victoria Woodard [email protected] Applied and Computational Mathematics and Statistics Department, University of Notre Dame, Notre Dame,
IN.
© 2023 The Author(s). Published with license by Taylor and Francis Group, LLC.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution,
and reproduction in any medium, provided the original work is properly cited. The moral rights of the named author(s) have been asserted. The terms on which this article has been published
allow the posting of the Accepted Manuscript in a repository by the author(s) or with their consent.
226 V. WOODARD

Figure 1. Wild and Pfannkuch’s Investigative Cycle (1999, p. 226).

The prerequisite for the course was our institution’s version To ensure students were completing the designated material
of an introductory statistics course, which covers introductory outside of class time, they were required to complete a short
statistics material (sampling distributions, confidence intervals comprehension quiz at the end of each module. The compre-
and hypothesis testing), and regression topics (simple, multi- hension quizzes were generally between five and eight questions
ple linear, logistic and multinomial regression including their given as a Canvas quiz. Students were allowed two attempts for
assumptions for inference). This prerequisite course provided each comprehension quiz and the average of their two scores
students with an opportunity to think about bias in data collec- was recorded in the gradebook.
tion, but offered no specific examples for students to conduct
data collection on their own. 53 students completed the Design
of Experiments course. These students were either applied and 2.2. Module Design
computational mathematics and statistics majors or minors, or The class met twice a week for 75 min. Depending on the
statistics majors. content, a typical module would span two to three of these class
The course was offered in a flipped classroom setting. While periods. The modules included:
a flipped classroom design is not necessary to complete the
experiments described in Section 3, doing so may provide 1. Introduction to Experimental Design – Methods and Ethi-
instructors and students with more time in class to conduct cal Concerns
these experiments. As such, a description of the course as a 2. Randomization in Designs including Matched Pairs
flipped classroom will be discussed here. Designs
Students were provided with guided note outlines to be 3. Completely Randomized Designs
completed outside of class time, by watching ADA compliant, 4. Contrasts
recorded content created by the instructor. This content fol- 5. Factorial Designs
lowed the material outlined in the course textbook: A First 6. Nested Designs/Fixed and Random Effects
Course in Design and Analysis of Experiments (Oehlert 2010). 7. Complete Block Designs
Additionally, the note outlines provided discussion questions 8. Split Plot Designs/Repeated Measures Designs
and practice problems pertaining to the material, but the solu- 9. Designs with Covariates
tions were not provided to students in the course content 10. Propensity Score Analysis
videos. To aid in critical thinking and discussion, students were
In a typical module, the following would occur:
expected to complete the discussion questions and practice
problems on their own while completing the rest of the note 1. Students would complete the assigned note outlines and
outline. Figure 2 provides a condensed example of the note comprehension quiz outside of class. According to course
outlines from the section on contrasts. Note that spaces origi- instructor feedback, these would typically take between 2 and
nally provided to students to write down their work have been 3 hr to complete, depending on the student and the module.
removed for space reasons here. This example demonstrates 2. In class, students would discuss the note outlines in self-
notes where the student is expected to fill in the blank and selected small groups and try to answer questions that they
write out a definition, as well as complete a guided example were not able to complete outside of class. Students also
and then try one on their own. Finally, the student is asked to provided their thoughts on the discussion questions and
prepare some remarks for the class discussion about the term their attempts at the practice problems during this time.
orthogonal, either as a definition, or how it has been used in The instructor would circle the room, answering questions
other classes they have taken. that groups had, and observe common discussion points
 JOURNAL OF STATISTICS AND DATA SCIENCE EDUCATION 227

Figure 2. A condensed example of the guided note outlines used in the course.

groups were making about the notes. The amount of time dataset. The use of groups was intentional to prevent shy
spent during this phase of class was dependent on the level or anxious students from feeling like they had to initiate a
of participation by the students but the class would typi- conversation with a stranger and so that the students could
cally spend between 20 and 30 min in this phase. Once it start reflecting on the experimental design process while they
seemed that the students were getting off task, the instructor were still collecting data. Students were asked to return to the
switched to a whole-class discussion and the common dis- classroom by a designated time. Once all students were back,
cussion points noted during the group discussion time were a debriefing of the data collection process was discussed as
addressed. a class. Here the students provided any comments or con-
3. Techniques for exploratory analysis and statistical analysis for cerns about the data collection process that they observed,
the module were discussed. Since a course covering regres- and ways that they might improve upon the experiment if
sion topics was the prerequisite to the Design of Experiments they were to conduct it again in the future. Depending on
course, some of the topics for analysis were review for stu- the experiment, this phase could take anywhere between 30
dents, especially at the beginning of the semester. As the and 75 min. Each experiment and discussion were always
course went on, a bit more time was spent demonstrating completed on the same day to ensure students remembered
the process of analysis for the different design methods. This the problems or concerns that they had during data collection
meant that the class might spend anywhere between 30 and and could share them with the class.
60 min in this phase. This demonstration usually included a 6. Depending on how much time was left in the allotted time for
practice problem with curated data, which gave the students the module, students worked on the analysis and write up for
an idea of how their data should be formatted when they went their experiments. This time was provided so that students
to collect their own data for an experiment of the same type. could remind each other of important details in the data
R code was also provided in a script so that students could collection process and ask questions about how to conduct
follow along and write their own comments as the topics were the analysis in R if they needed additional help. Most students
discussed. did not finish the write-up in class. They were expected to
4. An introduction to the main problem for the module would complete it outside of class and submit it within a week.
be given and students would work in small groups to start
planning their experimental design. The students would then For those wishing to use the assignments described in Section 3
come together as a whole class and come to a consensus about with their students, but are not planning on using a flipped class-
how data collection should proceed. The instructor had a room design, it may be advantageous to give students the main
flexible intended process in mind for each experiment and problem at the end of a class period where the experimental
attempted to direct the students toward that process through design has just been discussed, and ask them to prepare their
leading questions in both the prompt and in the discussion. own design for the next class. You could then have the whole
This would take approximately 30–40 min. class discussion at the beginning of the next class period to
5. Once a process was established, students were sent out onto solidify the experimental design that will be used and students
campus in groups to collect data and record it in the class can implement the design in the same class period.
228 V. WOODARD

2.3. Assignment Design Aspects design process for the given experiment. As another exam-
ple, Muehlenstaedt and Lanzerath (2019) detailed an excellent
The main problem in each section was designed not only to
experiment to demonstrate split plot designs about a boomerang
give students the chance to work with an experimental design
tin, but I felt that it was too resource intensive and wouldn’t be
based on the current topic being discussed in class, but to
of interest to my students. Finally, Hunter (1977) provided 32
demonstrate other aspects of experimental design that have a
examples of experiments that could be conducted by students,
much broader impact across data collection. While not every
but didn’t provide any details on how the experiments should be
experiment included all of the following, the list given below
conducted.
demonstrates some aspects of the assignment creation that were
Due to the lack of previous examples that I felt I could use,
used in at least one experiment and were found to be helpful to
I ended up creating most of my experiments for the course
promote students’ understanding of experimental design. More
from scratch. While students completed 10 modules during
information about how these items were used and why they were
the semester, this article contains a discussion of only five of
helpful is discussed in more depth in Section 3.
the experiments (main problems) that were given to students
As a whole, the assignments should have students:
during that time. The main problems to be discussed here will
• Propose and discuss multiple experimental designs so as to come from the modules on Matched Pairs Design, Completely
understand potential benefits and limitations of the experi- Randomized Designs, Factorial Design, Random Effects with
ment. Nested Designs and Split Plot Designs. For each of these design
• Provide a list of potential explanatory variables, factors, and types, the reader will be presented with:
factor levels that they believe may affect the response variable.
• The prompts that students were given for the main problem
• Consider variables that may affect the response variable but
of the module.
are of no interest to the research question, and potentially
• Highlights of the discussion to plan the experiment and the
how to hold those variables constant.
experimental design that students eventually decided on.
• Think critically about the research question of interest and
• Comments from the instructor about the experiment.
propose a response variable and how best to measure it.
• Determine how randomization will be consistently per- When reading about the experimental design process, note that
formed in the experiment across all groups collecting data. the term “student” is used for a student in the Design of Exper-
• Consider the protocols and ethical concerns of using human iments course. A “participant” is an individual who was asked
subjects in research as well as challenges in finding volunteers to participate in the data collection process. Unless otherwise
to participate in a study, while still collecting data that is noted, students were not participants in the experiment.
representative of the intended population.
• Critique the use of resources allocated for an experiment
for both cost and usefulness, and suggest alternate tools for 3.1. Matched Pairs Design
measurement and data collection if necessary.
• Consider group sizes (number of students within a group) The first experiment for the course was modified from Randal
and sample sizes (number of participants to include) of the Blackwood’s best practices presentation at the 2016 AP Statistics
experiment and whether the amount of data collected will be reading (Blackwood, 2016). This experiment was used during
enough to provide a meaningful answer to the question that the module on matched paired designs. Matched pairs designs
was asked. are used when two measurements are made on each of the
• Utilize technology, such as Google Sheets or Google Forms, experimental units and a comparison of the two sets of measure-
to organize a class dataset that is easily accessed and fairly ments is desired. Since the data is matched on the experimental
clean. unit, the traditional independence assumption for analysis is
• Work in groups to promote discourse and prevent shy stu- violated. Yet this method of design is often used since it is good
dents from feeling as if they need to talk to strangers outside at reducing the variability between groups.
the classroom.
• Practice technical and scientific writing. 3.1.1. Prompt
Problem: Are dominant arms stronger than nondominant arms?
Supplies: Pool Noodle; Tape Measurer
3. Experimental Designs Plan
When first creating the course, I looked for inspiration for 1. Why might the question above be of interest to someone?
experiments my students could conduct via a Google search. 2. To collect data for this, you will be provided with a pool
While there were several examples given, I didn’t find many that noodle and tape measurers.
would meet the learning objectives I wished to cover, were inter-
esting to my students, and that provided enough guidance that I a. Propose a method for data collection that could be used to
could follow along with the intended experiment. For example, answer the question.
Gelman and Nolan (2017) and Annis (2005) both defined fac- b. What will be your response variable?
torial experiments based on Box’s helicopter experiment (1992). c. What will be your explanatory variable?
While these authors provided excellent examples, I didn’t feel d. How will we apply randomization to this scenario?
like there would be enough discussion about the experimental e. How will you keep track of the data you collect?
 JOURNAL OF STATISTICS AND DATA SCIENCE EDUCATION 229

Data Collection supplies back in a pile, I asked if any of them had noted the label
Analysis and Conclusions on their pool noodle. This was intentional on my part to get
them thinking about sources of variability that might affect the
3. Once we have collected data, the final step of this process is outcome of the results. I have also considered having students
to write a short report about the methods and results that we record the pool noodle label as a precursor to block designs.
have found. To do this please write a report that: For those wishing to conduct this experiment, it is important
to plan ahead and make sure that you have enough supplies.
• Introduces the problem and why it might be of interest to
The choice to have eight pool noodles and thus eight groups
someone.
was completely dependent on how many pool noodles I could
• Discusses the data that was collected.
purchase in January. Smaller groups might mean a quicker
• Describes how we applied randomization in our study.
implementation, however, if groups are too small, then students
• Includes applicable results that we found.
will have to be constantly changing rolls (i.e., spotter, tape
• Interprets results that were found and uses them to answer
measure holder) within the group and this might cause extra
the question.
measurement error in data collection.
As a final note for this experiment, prior to data collection, it
3.1.2. Students’ Experimental Design may also be helpful to set up a Google Sheet and share it with
Students came to the consensus that they should be the partici- all of the students in the course. During our planning phase,
pants, throw the pool noodle once with each arm and measure students stated that they would write the results on a sheet of
how far it traveled on each throw. Because there were eight sets paper and then share it with everyone else in the class when they
of supplies, the students split up into eight groups of either six or were done with data collection. However, when students went
seven students. Each group marked a starting line from which to analyze data, it took up a lot of class time to wait for the one
to throw (typically a crack in the sidewalk) and the participant student to create and share the spreadsheet with the rest of the
was not allowed to cross the line when they threw the pool class. While this may be a good lesson for students to learn, I felt
noodle. One person spotted where the pool noodle landed and it was not a good use of class time.
two students used the tape measure to determine how far the
pool noodle had traveled.
Since this problem was posed in the section about matched 3.2. Completely Randomized Design
pairs experiments, students recognized that each participant
The second experiment to be discussed here is for a completely
needed to throw the pool noodle twice, once with their dom-
randomized design. A completely randomized design is when
inant arm and once with their nondominant arm. To apply
each of the experimental units has the same probability of being
randomization to the scenario, students flipped a coin or used a
assigned to any of the treatment groups. Along with other help-
random number generator on a phone to determine with which
ful tips when designing experiments, Easterling (2004) provides
arm they would throw the pool noodle first.
thoughtful background on appropriate randomization methods
to uphold this definition.
3.1.3. Instructor Comments
Since this was the first experiment that students designed for the 3.2.1. Prompt
course, their description of how to collect the data was originally Problem: Do people remember better if they hear a list, see a list
very limited. Despite giving them several minutes in class to or both?
discuss in their groups, most groups’ plans were akin to “we Plan
will each throw the pool noodle with each of our arms and
record where it lands.” There was no mention of what part of the 1. We would like to use a completely randomized design to
pool noodle to measure after it had landed, or if students were study this problem. In small groups, propose a method for us
allowed to get a running start to throw the pool noodle. I had to collect the data to answer the question. As a class, we will
anticipated this and was prepared to ask them leading questions define the methods we wish to use to collect the data. Take
to eventually get them to the design that was described above. notes on our discussion, you will need them later!
One of the questions that I asked the students during this 2. Identify the explanatory and response variables for this
time was where the data should be collected. It was easy to experiment.
convince the students that the only place that would allow for Data Collection
enough space for the pool noodles to be thrown would be
outside. This was done strategically, knowing that wind would 3 Spend 30 min with a partner traveling around campus finding
likely be a confounding variable in the study. The purpose of volunteers for our experiment (make sure they have not done
choosing this location was so that I could break the design that the experiment with someone else yet). Use the method we
the students came up with so that they would think about such discussed in class to conduct the experiment. Make sure to
things in the future. record your results in our class Google Sheet.
Another point of interest for this experiment was that there
Analyze
were multiple pool noodles that were used. Each pool noodle
was given a unique label, but the students were not specifically 4. Once everyone is back from collecting data, you will indi-
told to mark down the label of the pool noodle that their group vidually conduct the analysis. Make sure to do a preliminary
used. After the students had returned to the room and put their analysis as well as the ANOVA procedures.
230 V. WOODARD

Conclusions data collection process, which demonstrates the layout of the

Google Sheet, is given in Figure 3. The instructor made the Sheet
5. The final step of this process is to present your results. You prior to data collection day and students asked for the addition
will do this by writing a short paper which includes: of the variables “Class” and “Type of Learner” to account for
potential confounding variables. It was helpful to ask for addi-
• Introduction—introduce the problem: what were we try-
tional variables that the students wanted to collect to be able to
ing to figure out and why might it be important?
show them how to set up a drop down menu in Google Sheets.
• Methods—how was the data collected and why did we
One issue that came about with the use of Google Sheets was
collect it in that way? Be VERY specific about this as our
that for the students using the Google Sheets app on their phone,
course really is about how we design our experiments.
the page kept reloading whenever anyone else input data. When
• Results—what did you find from your preliminary analy-
they tried to go to the actual webpage through a browser, their
sis and ANOVA procedure?
phones would just reopen the Google Sheets app. It is recom-
• Discussion/Conclusion—what conclusions can you make
mended that each group take a laptop or tablet with them that
about the question that was asked and what recommenda-
can access the Google Sheets page and does not automatically
tions can you make to those that may be interested in the
open the Google Sheets app. Additionally, students found that
question that was asked?
others would overwrite the data they had already input. As such,
students “claimed” lines on the Google sheet by giving them a
3.2.2. Students’ Experimental Design background color prior to leaving the classroom.
Similar to the design proposed by Lawrance (1996), students Another solution that may be used in this context is a Google
decided that a list of 10 items from the grocery store that should Form. Using this technology would remove the burdens of using
be given to participants. These items were selected haphazardly Google Sheets described above, but it takes the ownership of the
and can be seen in Figure 3. The lists were printed by the data away from the students and puts it into the hands of the
instructor and were shown to the participants assigned to see instructor. Depending on the learning objectives of one’s course,
the list, or see and hear the list. Students found that if they read this may be the more desirable method for data collection.
at the right pace, it took about 20 sec to read the list. This was
the amount of time that participants were given to view the list,
if that was part of their treatment. 3.3. Factorial Design
With prompting, the students also came up with a script so The third experiment to be discussed here was intended to
that each group of students would give approximately the same help students learn more about factorial designs. In this type of
directions to their participants. To randomly assign treatment design, the possible combinations of the levels of two (or more)
to participants, the students decided that they would have their factor variables are used as the explanatory variable groups.
participants roll a die, or use an equivalent random number One of these combinations is then assigned at random to a
generator on their phone. If the die came up as one or two, the participant and the response variable is measured.
participant would see the list. For three or four, the participant
would hear the list. For five or six participants would see and 3.3.1. Prompt
hear the list. Problem: What is the best way to take a selfie?
Plan
3.2.3. Instructor Comments
1. Explanatory variables
In addition to learning about completely randomized designs,
one of the goals for this activity was to introduce students • What are some factors that you think affect the quality of
to technology that can be helpful in the data collection pro- a selfie? What are potential levels of these factors?
cess, especially when multiple people are collecting data for • Which of these factors can we control?
the experiment, as well as to reinforce principles of Tidy Data • Select two factors that you have listed. How many total
(Wickham 2014). groups will you have with the factors you have selected?
A Google Sheet was created and introduced to the data
collection process and shared with the students for this exper- 2. Response variable
iment. Drop down lists were used so that students did not
have to clean the data after they were done with collection. • How can we measure the quality of a selfie?
Students were encouraged to use this tool when they created • Choose a response variable that we can use to measure
their own spreadsheets in the future. Partial output from the this.

Figure 3. An example of using Google Sheets to aid in data collection.

 JOURNAL OF STATISTICS AND DATA SCIENCE EDUCATION 231

3. Design the experiment in the context of the problem, which along with the ambiguity of
what to measure for the response sparked a lot of rich discussion
• How many participants do you think we need in the study? about the data to be collected for the experiment. Aspects of
• How will you ensure that we have a balanced design? using a filter, which app should be used to take the picture and
• How will participants take their selfies? if the selfie could be in a group or alone were all discussion
• Is there anything else that you can think of that will affect points that students brought up. Even if students had settled on
the study? another metric to measure the response, I found the discussion
that students had about the pros and cons of possible variables
4. You have been given access to a class Google Sheet. Decide to be very thoughtful and worth the class time.
what information needs to be collected from each participant Discussion from the debriefing after data collection was also
and input these variables into the spreadsheet. valuable. The students realized that some had thought that high
5. Based on what you have learned from past experiments; is level lighting meant that the participant needed to be outside in
there anything you need to consider for this experiment while natural light, while others thought that it meant just having the
collecting data? lights on in a room. Students learned through this experiment
Data Collection that researchers may define levels of variables differently and in
Analysis and Conclusion particular, subjective wording such as “high” and “low” should
be better defined to avoid confusion in future studies. Addition-
6. For this write up, pretend that you are submitting to the
ally, the students felt that there needed to be more restrictions
school newspaper. Your focus should still be on the design
on the individuals that should participate in the experiment in
of the experiment. However, try to explain it in a way that
the future. Several of the individuals that agreed to participate
someone who has not learned anything about factorial design
in the experiment were satisfied with a single selfie because
might still understand what you did and why you did it.
they “just didn’t care” what was posted on social media. The
students felt that only individuals that were more discerning
3.3.2. Students’ Experimental Design of their selfies should be considered for the experiment in the
For the response variable, students had trouble with the wording future.
of the problem and defining what “best” meant when it came to
taking a selfie. Some of the ideas that students came up with to
collect data for the response variable that they thought defined
“best” included: 3.4. Random Effects and Nested Designs
1. Take 10 selfies and determine which one you would post to This next activity was designed for the module including ran-
social media. dom effects and nested designs. An experiment uses random
2. Take 10 selfies and count how many you would post to social effects when the treatments are selected at random from all
media. possible treatments. For example, we may believe that people’s
3. Take one selfie and rate it on a scale from one to ten as how perception of a drawn object may be altered, if different artists
likely you are to post it on social media. draw that object (artist effect). If we are not interested in which
In the full-class discussion, students found fault in each of artists are chosen to draw the object, just that we have pictures
these methods, so they were not used. For the first two options, from different artists in our sample, then the artist will be a
some students were worried that participants would not find a random effect in our design.
selfie that they liked in 10 photos. They would most likely find A nested design, like factorial designs looks at combinations
one in 100 selfies but then the students were worried about the of factor levels and measures some response variable. The differ-
amount of time it would take to collect the data. Additionally, ence between nested designs and factorial designs is that in the
it was not clearly defined what would be measured for the first nested design, the levels of the second (nested) factor variable
option. For the third option students argued that the first selfie differ across the levels of the first factor variable. Continuing
almost never looks good and most people would rate it as either with our artist example, each artist may find five individuals to
one or two and they wouldn’t get very much variability in the guess what they have drawn. We may number these individuals
data. one through five for each artist. However, guesser one for the
It took a while, but students finally settled on the option first artist is not at all related to guesser one of the second artist,
where the participant would be allowed to take unfiltered selfies despite having the same label.
until they found one they would be willing to post on social Additionally, with the factorial design, several participants
media. The number of photos they took to reach that selfie was are assigned to each combination of levels to use replication and
taken as the measurement of the response variable. Students to be able to see possible variation. With nested designs, multiple
were able to come up with a list of potential factor variables for measurements should be taken from each of the combinations
the explanatory variable as well, but eventually settled on the to achieve the same goal. Continuing with the example we have
angle of the camera (above, straight on, or below face level) and been using, we would want to take multiple measurements from
the lighting of the room (high or low lighting). the combinations of guesser nested within artist.
When designing this activity for students, I found it difficult
3.3.3. Instructor Comments to get them to the desired design with typical leading questions.
This was probably the most interesting problem to watch stu- Instead of the usual in-class process that students had followed
dents plan during the semester. Students seemed very interested during the semester’s activities, I decided it would be best to
232 V. WOODARD

give the students the design for the experiment and just have Analysis and Conclusions
them implement it. Since students didn’t have to design the
6. Please answer the following questions
experiment themselves, part of their write-up was to describe
why this was a nested design and when random effects were used • Why is this a nested design?
in the experiment. • What are the layers of nesting within this design?
• Are we using random effects or fixed effects for each layer
3.4.1. Prompt of our design?
Problem: How does our perception affect our opinion and if we • Conduct the analysis using software. Are any of the effects
take information second hand, can we still trust that information? significant? Interpret the findings.
Supplies: Notecards with commonly used words and phrases
Plan 3.4.2. Experimental Design
Due to the intricacies of the data collection of this experi- To setup for this experiment, group leaders were given eight note
ment, it has already been planned for you. cards. On four of the cards, they had been given a randomly
Data Collection selected word or phrase from a list of common words or phrases
For this activity, you will need to get into 10 groups of five that were found on the internet. Additionally the cards were
people. labeled A-1, A-2, A-3, A-4, B-1, B-2, etc. The letter was used as
1. Select one person at random to be the leader of your group. an artist identification and the number was used to identify the
The leader will need to come to me to get a stack of note cards. word within the group. Each label was put on two note cards, one
Four of these cards will have a word or phrase on them. Four with a word, and one that was blank. The group leaders were
of the cards will be blank. Please give one blank note card to given no more than eight minutes to draw the four words or
each other member of your team and do not let them see the phrases on the back of the card that contained the original word
words on your note cards. or phrase. These are identified as “Draw Word …” in Figure 4.
2. On the back of each the notecards that contains a word or When the team leaders were finished drawing the words they
phrase, your leader needs to draw (to the best of their ability) were assigned, they then gave a picture (concealing the word or
that word or phrase. Please do not let any of your classmates phrase) to one of their group mates. This group mate needed
(including members of your group) see your words. to guess what they thought the original word or phrase was
3. Next, the leader will show the drawing to one of their team- and write their guess on the blank note card they were given.
mates. The teammate should write their guess on the blank These are identified as “Guess Word …” in Figure 4. The group
note card that was given to them. members then needed to go out on campus to find a participant
4. Each teammate will now need to leave the classroom and find to draw the guess of the word or phrase on the reverse side of
a participant who will draw the team member’s guess on the the second note card. These are identified as “Draw Guess …”
back of the note card they have been provided. in Figure 4.
5. Finally, bring the note cards back to the room. As a class, you Once this process was complete, the students came back
will be asked to guess what you think has been drawn on each to the classroom and gave the new drawings (now named the
of the cards. We will determine the proportion of people that products of the experiment) to the instructor. In sequence, the
can correctly identify the original word or phrase. products were placed on the overhead projector and any student

Figure 4. Diagram of the nested experimental design for the artist experiment.
 JOURNAL OF STATISTICS AND DATA SCIENCE EDUCATION 233

not involved with the original word, either as the artist or the 1. What is the whole plot factor? How do you know?
guesser, made a mental guess as to what they thought was being 2. What is the split plot factor? How do you know?
depicted in the picture. 3. What is our response variable? How will we measure it?
Before class a Google Form that allowed students to input 4. Give a detailed description of how we would conduct this
the original artist (A through J), the card number (1 through 4) experiment.
and whether they guessed the original word or phrase correctly 5. How would you design the spreadsheet for data collection?
(using 1 for correct and 0 for incorrect) had been created. As
we cycled through the cards, the students were given just a few Data Collection
seconds to identify what they thought was depicted on the image Analysis and Conclusion
on the screen. The original artist was then asked to reveal the
6. For your write up for this activity
original word. Students completed the Google Form for each
picture and the form data was released to the class after all of • Describe to Astrid how we designed the experiment.
the word and image pairs had been shown. The proportion of Remember that Astrid is a cookie baker, not a statistician.
students who correctly guess the original word was used as the Be detailed in what you did, but keep the language simple.
response variable for each product. • What recommendation would you give Astrid based on
Diagram of the nested experimental design for the artist the data that was collected? Why would you recommend
experiment. this? Note: you may not find significant results with the
3.4.3. Instructor Reflections data, but Astrid still wants an answer. Make a reasonable
This structure worked very well for a class of 50 students (three decision with the data you have available.
students were absent on the day of the experiment). If you are • Give Astrid at least two diagrams or graphs to look at that
working with a smaller class size, you may have your students all may help her understand the design of the experiment and
be artists. They would then be responsible for finding guessers the results that you found.
and redrawers outside of the classroom.
To ensure that groups aren’t picking their best artist for 3.5.2. Students’ Experimental Design
the experiment and potentially confounding the random artist For this experiment, groups of students were randomly assigned
effect, the students were not told the nature of the experiment a bag of pre made cookie dough that had a temperature on it:
before they selected their group leader. Be prepared to handle a either 350 degrees or 375 degrees. The dough had been made
lot of laughter and students getting off track when in the guess- before class in one batch so there wouldn’t be any difference
ing phase. Some of the redraws will be completely ridiculous. in the dough between the groups and so that there would be
enough time to finish the experiment in 75 min. The larger
3.5. Split Plot Design batch was partitioned into six smaller batches of equal weight
so that students would have the same size cookies when they
The final experiment to be discussed here was for the Split Plot divided the dough into 12 cookies on their pan. One possible
Design module. A Split Plot design is used when you have a configuration of the experiment can be seen in Figure 5.
treatment that must be applied to all experimental units in a The students were then to find participants to eat the cookies
large setting, but a subset of the units can be given different and provide ratings for them on a scale from one to ten based
treatment on a second variable. For example, in an oven, all on each of their taste, color and texture. The response variable
cookies being baked on a pan will have to be cooked at the same was the sum of the overall ratings for the cookie (a value out of
temperature. However, some of the cookies can be randomly 30 points).
assigned to receive different times for cooking. This makes the For this experiment, I allowed students to be participants in
oven the whole plot, the temperature the whole plot factor, and the study, in that they were allowed to eat one of the cookies that
the time the split plot factor. was made during the experiment and then rate it. A problem that
3.5.1. Prompt we ran into though was that the students were sampling cookies
Problem: Astrid loves to bake cookies. She has recently started from several of the other groups, causing some independence
a new company that sells the cookies that she bakes. People love issues. This occurred partly because we ran out of time at the
her cookies and they are in high demand! She was looking for a end of class and students did not have time to find enough par-
way to speed up the process of making the cookies so that she ticipants to eat their cookies. Instead, they had their classmates
can make more per hour. However, she doesn’t want to sacrifice eat and rate them.
the quality of her cookies in the process. She has enlisted your
help to test customer satisfaction of cookies baked for different 3.5.3. Instructor Comments
times at different temperatures. Suppose that we have six ovens This experiment clearly requires the use of a kitchen with mul-
available to us to conduct our experiment. tiple stoves. The class needed to be partitioned into very large
Our goal is to design a split-plot design to help Astrid answer groups to allow everyone to be part of a group, which meant
the question: How long and at what temperature should I bake my that several students were not doing anything to help with the
cookies? experiment (there were too many cooks in the kitchen!). This
Supplies: Cookie baking supplies (including an oven); works much better if you can find enough ovens so that you have
Cookie Dough just a few students per group, or if you just have a smaller class
Plan size.
234 V. WOODARD

Figure 5. Diagram of a potential Split Plot design for the cookie experiment.

I recommend making a batch of cookies in one of the school a noodle, etc. I thoroughly applied my knowledge doing these
ovens before you allow students to conduct the experiment hands on activities.”
so you can calibrate the oven temperatures and times in the
experiment. I tried this at home and my oven seemed to be a 4.2. Final Reflections
lot more powerful than the ovens at school, making somewhat
Based on students’ reactions in the classroom and course
undercooked cookies to be served to participants. You should
instructor feedback, it appeared that most students liked the
also look into the guidelines for preparing and serving food to
design of the course and the way that the content that was
students at your school.
presented. The work that students demonstrated on their final
Should an individual wishing to do this project find them-
projects showed that they had not only learned the nuances of
selves without the resources of a school kitchen, the data from
the designs presented that semester, but also developed some
this experiment is provided as an extra resource to the readers.
critical thinking skills about factors that may influence the
After students are given the chance to design the experiment
outcome of their study. Additionally, while some students stuck
in their groups, a narrative of the actual data collection process
with the familiar completely randomized design for their final
could be given to students along with the data so that they can
project, the research questions most students asked necessitated
analyze it and provide a solution for Astrid.
and subsequently were answered with more advanced designs
that we discussed during the semester. Because the students had
4. Concluding Remarks conducted the experiments prior, they likely felt comfortable
using those experimental designs for their independent final
4.1. Student Reflections projects.
Students were given the chance to reflect on the course through However, as a warning to those that may be considering
course evaluations. Students’ comments indicated that they using guided note outlines, with or without a flipped classroom
found the activities of the course to be informative and fun to design, some students didn’t seem to take the course as seriously
implement. Some students found that elements of each of the as others. While students were required to complete the note
experiments we did in the course helped them with designing outlines on their own by watching the video and then complete
and implementing their own experiments for the final project. the canvas quiz to earn credit, some students indicated that
For example, one student commented: they just filled in the blanks without thought, skipped over the
“I really liked that each of the experiments we did in class discussion questions and then took the quiz. Students either
were unique and focused on different elements of designing an need to be motived to complete the note outlines, or more
experiment. My favorite experiment was the one I did for my incentivized, to make sure they are getting the most out of the
final project because I felt like everything learned throughout work done away from the classroom.
the course was used, and it showed me how many different Finally, a note about the material covered. Because this was
aspects of designing and implementing an experiment there the first iteration of the course, I wasn’t sure exactly how much
are.” material the students would feel comfortable covering during
Not only were the activities informative, students found them the semester. After having taught the course once, I realized that
to be fun and something they could take ownership of, especially there was room for more material in some of the weeks. In the
the ones that involved food. In addition to the Split Plot Design next iteration of the course, I plan on including lessons for power
(Astrid’s Cookies) students also got to experiment with bags of and sample size calculations and spending a bit more time on the
M&Ms in the module about contrasts. For example, one student assumptions for inference needed for each type of test.
commented:
“I felt like design of experiments really allowed me the chance Data Availability Statement
to be creative with what we were learning in the classroom. We Data sharing is not applicable to this article as no new data were created or
did so many fun activities with taking selfies, M&Ms, throwing analyzed in this study.
 JOURNAL OF STATISTICS AND DATA SCIENCE EDUCATION 235

ORCID Easterling, R. G. (2004), “Teaching Experimental Design,” The American

Statistician, 58, 244–252. DOI:10.1198/ 000313004X1477.
Victoria Woodard https://orcid.org/0000-0003-0894-0075
Gelman, A., and Nolan, D. (2017), Teaching Statistics: A Bag of Tricks,
Oxford: Oxford University Press.
References Hunter, W. G. (1977), “Some Ideas About Teaching Design of Exper-
iments with 2 (to the 5th) Examples of Experiments Conducted
American Statistical Association. (2016), “Guidelines for Assessment by Students,” The American Statistician, 31, 12–17. doi:10.1080/
and Instruction in Statistics Education College Report,” available 00031305.1977.10479185.
at https://www.amstat.org/docs/default-source/amstat-documents/ Lawrance, A. J. (1996), “A Design of Experiments Workshop as an
gaisecollege_full.pdf Introduction to Statistics,” The American Statistician, 50, 156–158.
American Statistical Association Undergraduate Guidelines Workgroup. doi:10.1080/00031305.1996.10474364.
(2014), “Curriculum Guidelines for Undergraduate Programs in Statisti-
Lee, H. S., Mojica, G. F., Thrasher, E. P., and Baumgartner, P. (2022),
cal Science,” available at https://www.amstat.org/education/curriculum-
“Investigating Data like a Data Scientist: Key Practices and Pro-
guidelines-for-undergraduate-programs-in-statistical-science-
cesses,” Statistics Education Research Journal, 21, 3. DOI:10.52041/serj.
Annis, D. H. (2005), “Rethinking the Paper Helicopter: Combining Statis-
v21i2.41.
tical and Engineering Knowledge,” The American Statistician, 59, 320–
326. DOI:10.1198/000313005X70777. Muehlenstaedt, T., and Lanzerath, M. (2019), “Teaching Split Plot Experi-
Blackwood, R. (2016), “AP Reading Best Practices Night,” available at ments with a Boomerang Tin,” arXiv. DOI:10.48550/arXiv.1907.10684
StatsMonkey: http://apstatsmonkey.com/StatsMonkey/ReadBestPractice. Oehlert, G. W. (2010), A First Course in Design and Analysis of Experiments.
html Gary W. Oehlert.
Blades, N., Schaalje, G., and Christensen, W. (2015), “The Second Course Wickham, H. (2014), “Tidy Data,” Journal of Statistical Software, 59, 1–23.
in Statistics: Design and Analysis of Experiments?” The American Statis- DOI:10.18637/jss.v059.i10.
tician, 69, 326–333. DOI:10.1080/00031305.2015.1086437. Wild, C. J., and Pfannkuch, M. (1999), “Statistical Thinking in
Box, G. (1992), “Teaching Engineers Experimental Design with a Paper Empirical Enquiry,” International Statistical Review, 67, 223–248.
Helicopter,” Quality Engineering, 4, 453–459. doi:10.1111/j.1751-5823.1999.tb00442.x.