In Their Own Words

In Their Own Words
What Scholars and Teachers Want

You to Know About Why and How
to Apply the Science of Learning in
Your Academic Setting
Editors
Catherine E. Overson
Christopher M. Hakala
Lauren L. Kordonowy
Victor A. Benassi
Division 2, American Psychological Association 2023

9 i
Feedback
Feedback regarding the editorial content of this book or any of its chapters should be directed toward
the individual authors. Contact information will be provided at the end of each chapter. Feedback
regarding technical matters of formatting or accessibility of this text via the online environment of the
Internet should be directed to the Internet Editor. If you have any complaints or difficulties in accessing
these materials, be sure to provide a detailed description of the issue(s); you should include information
about the browser you are using and its version number, as well as the type of computer you are using
and its operating system.
Copyright and Other Legal Notices

The individual essays and chapters contained within this collection are Copyright © 2023 by their
respective authors. This collection of essays and chapters as a compendium is Copyright © 2023 Society
for the Teaching of Psychology.
You may print multiple copies of these materials for your own personal use, including use in your classes
and/or sharing with individual colleagues if the author's name and institution, and a notice that the
materials were obtained from the website of the Society for the Teaching of Psychology (STP), appear on
the copied document. For research and archival purposes, public libraries and libraries at schools,
colleges, universities, and similar educational institutions may print and store in their research or
lending collections multiple copies of this compendium as a whole, without seeking further permission
of STP (the editors would appreciate receiving a pro forma notice of any such library use). No other
permission is granted to you to print, copy, reproduce, or distribute additional copies of these materials.
Anyone who wishes to print, copy, reproduce, or distribute copies for other purposes must obtain the
permission of the individual copyright owners. Particular care should be taken to seek permission from
the respective copyright holder(s) for any commercial or "for profit" use of these materials.
ISBN: 978-1-941804-72-8
Suggested Reference Format

For the overall text, reference the book in this fashion:
Overson, C. E., Hakala, C. M., Kordonowy, L. L., & Benassi, V. A. (Eds.). (2023). In their own words: What
scholars and teachers want you to know about why and how to apply the science of learning in
your academic setting. Society for the Teaching of Psychology.
https://teachpsych.org/ebooks/itow
Individual chapters should be referenced in this fashion (an example):
Kapur, M., & Roll, I. (2023). Productive failure. In C. E. Overson, C. M. Hakala, L. L. Kordonowy, & V. A.
Benassi (Eds.), In their own words: What scholars and teachers want you to know about why and
how to apply the science of learning in your academic setting (pp. 196-206). Society for the
Teaching of Psychology. https://teachpsych.org/ebooks/itow
Acknowledgments
Preparation of this book and the work presented in several chapters were supported in part by grants
from the Davis Educational Foundation. These grants were received from the Davis Educational
Foundation established by Stanton and Elisabeth Davis after Mr. Davis's retirement as chairman of
Shaw's Supermarkets, Inc.
We also acknowledge the support provided by the Office of Academic Affairs/Provost’s Office,
University of New Hampshire. Finally, we extend our appreciation and thanks to the authors of the
chapters presented in this book. Their work offers teachers and others much to consider and apply as
they strive to promote student learning.
We thank the Editor-in-Chief (Jessica Cerniak) and Associate Editors (Karla Lassonde and Lisa Rosen) of
the Society for the Teaching of Psychology E-Books series for their guidance and assistance in the
preparation of this book.
i
About the Editors
Catherine Overson earned her Ph.D. in Psychology from the University of New Hampshire (UNH), with a
specialty in social psychology and science of learning. She is Director of the Center for Excellence and
Innovation in Teaching and Learning at UNH and Affiliate Associate Professor of College Teaching
(Graduate School). Her research focuses on the application of science of learning principles to teaching
and learning in college and university courses. Catherine coordinates and teaches courses for the UNH
College Teaching Programs. She is a member of Division 2 (Society for the Teaching of Psychology; STP)
of the American Psychological Association (APA). She is co-editor (with Victor Benassi and Christopher
Hakala) of Applying the Science of Learning in Education: Infusing Psychological Science into the
Curriculum (2014, STP). Catherine has worked on four Davis Educational Foundation (DEF) grants over
the last 12 years.
Christopher Hakala has over 25 years’ experience as a professor and Center Director in higher education
institutions. Over the years, his research has focused on reading comprehension, teaching and learning,
effective faculty development, and assessment. In addition, Chris has been invited to present at many
conferences around the country as well as dozens of colleges and universities on topics ranging from
reading narrative text to how to effectively manage large classes, or how to engage students in ways
that maximize student learning. At Springfield College, Chris directs the Center for Excellence in
Teaching, Learning and Scholarship and teaches courses in the psychology department.
Lauren L. Kordonowy
Lauren Kordonowy received her B.A. in Biology at Kenyon College, and she earned an M.Sc. in Biological
Sciences from Simon Fraser University. She then pursued her Ph.D. at UNH in genetics. Lauren is
the Science of Learning Project Coordinator at CEITL. Her work includes developing and implementing
projects for improving learning outcomes in undergraduate courses. Lauren works closely with faculty
members to address learning goals in their courses, and she has experience collaborating with faculty
across many departments. One of Lauren’s current long-term projects is the creation and deployment of
the Student Cognition Toolbox.
Victor A. Benassi
At the University of New Hampshire, Victor Benassi was a Professor of Psychology (1982 – 2018), Vice
Provost for Undergraduate Studies (2000 – 2003), and Faculty Director of the Center for Excellence and
Innovation in Teaching and Learning (2007 – 2018). He is a fellow of the American Psychological
Association and served as APA’s 2013 Division 2 President (Society for the Teaching of Psychology). He
has been a PI or co-PI on five Davis Educational Foundation grants that focus on the application of
science of learning principles in college and university courses. With Catherine Overson and Christopher
Hakala, he is co-editor of Applying the Science of Learning in Education: Infusing Psychological Science
into the Curriculum (Society for the Teaching of Psychology, 2014). In 2003, he received the American
Psychological Foundation’s Charles L. Brewer Distinguished Teaching of Psychology award.
ii
Table of Contents
Acknowledgments i
About the Editors ii
Table of Contents iii
Introduction
Catherine E. Overson, Christopher M. Hakala, Lauren L. Kordonowy, and Victor A. Benassi 1
Part 1: Past, Present, and Future of Applying the Science of Learning in Education
Constructing a Canon for the Science of Learning

Nora S. Newcombe 8
Introducing Desirable Difficulties Into Practice and Instruction: Obstacles and Opportunities
Elizabeth L. Bjork and Robert A. Bjork 19
The Culture of Teaching We Have Versus the Culture of Teaching We Need
Stephen L. Chew 31
Research Through the Eyes of Teachers
Luke I. Rowe and John Hattie 44
The Scholarship of Teaching and Learning: Scaling New Heights, but It May Not Mean What You
Think It Means
Regan A. R. Gurung 61
Part 2: Science of Learning: Principles and Approaches
The Prequestion Effect: Why it is Useful to Ask Students Questions Before They Learn
Shana K. Carpenter, Quentin King-Shepard, and Timothy Nokes-Malach 74
Successive Relearning: An Introduction and Guide for Educators
John Dunlosky, Maren Greve, Sabrina Badali, Kathryn T. Wissman, and Katherine A. Rawson 83
Applying the ICAP Framework to Improve Classroom Learning
Michelene T. H. Chi and Nicole S. Boucher 94
Spaced and Interleaved Mathematics Practice
Doug Rohrer and Marissa K. Hartwig 111
Task Complexity, Learner Expertise, and Instructional Goals in Managing Instructional Guidance
Slava Kalyuga 122
Interleaved Training and Category Learning
Sean H. K. Kang 132
Research-Based Principles for Designing Multimedia Instruction
Richard E. Mayer 143
Learning by Teaching
Logan Fiorella 158
iii
The Region of Proximal Learning and Curiosity
Janet Metcalfe 171
What Every Teacher Should Know About Cognitive Load Theory and the Importance of Cognitive Load to
Instruction
Greg Ashman and John Sweller 185
Productive Failure
Manu Kapur and Ido Roll 196
Using Worked Examples for Ill-Structured Learning Content
Alexander Renkl 207
Cognitive Strategies for Improving Students’ Conceptual Learning in Physics
Jennifer L. Docktor and José P. Mestre 225
Implementing Cooperative Learning in Large Undergraduate Classes via PLTL
Regina F. Frey and Scott E. Lewis 239
Frequent Quizzing Accelerates Classroom Learning
Chunliang Yang, David R. Shanks, Wenbo Zhao, Tian Fan, and Liang Luo 252
Adapting Peer Instruction for Online Learning Using a Learning Science Framework
Julie Schell and Andrew C. Butler 263
Part 3: Preparing Faculty, Educational Developers, Student Success Professionals, and

others to Apply the Science of Learning
Cultivating and Sustaining a Faculty Culture of Data-Informed Teaching: How Centers for Teaching
and Learning Can Help
Marsha Lovett, Chad Hershock, and Judy Brooks 281
The Learning Scientists: Promoting Communication About the Science of Learning
Megan A. Sumeracki, Althea N. Kaminske, Carolina E. Kuepper-Tetzel, and Cynthia L. Nebel 295
Different Goals Imply Different Methods: A Guide to Adapting Instructional Methods to Your Context
Kenneth R. Koedinger, Martina A. Rau, and Elizabeth A. McLaughlin 303
Are Study Strategies Universal? A Call for More Research With Diverse Non-WEIRD Students
Roberta Ekuni, Sabine Pompeia, and Pooja K. Agarwal 316
The Learning Lab: An Experiment in Student-Centered Teaching and Learning
Marlon Kuzmick, Lauren Davidson, Jordan Koffman, and Tamara J. Brenner 328
Considerations for Academic Support Centers Regarding Learning Strategy Recommendations
Jennifer A. McCabe 341
Applying the Science of Learning: The Cognition Toolbox at 15 Years and Counting
Victor A. Benassi, Catherine E. Overson, Lauren L. Kordonowy, Elizabeth M. Tappin, and
Michael C. Melville 352
Part 4: Preparing Students to Apply the Science of Learning

How to Teach Powerful Strategies so that Students Self-Regulate Their Use: The KBCP Framework
Mark A. McDaniel and Gilles O. Einstein 365
iv
Adaptive Blended Learning to Foster Self-Regulated Learning – A Principle-Based Explanation of a Self-
Regulated Learning Training
Tino Endres 378
Cultivating Greater Spontaneity in Learning Strategy Use
Emmanuel Manalo, Yuri Uesaka, and Clark A. Chinn 395
The Science of Learning Initiative at Colorado State University
Anne M. Cleary and Matthew G. Rhodes 407
Teaching Students to ‘Study Smart’ – A Training Program Based on the Science of Learning
Felicitas Biwer and Anique B. H. de Bruin 419
The Student Cognition Toolbox: Empowering Students to Become Better Learners
Catherine E. Overson, Lauren L. Kordonowy, Jennifer F. Calawa, Elizabeth M. Tappin, and
Victor A. Benassi 434
Part 5: Putting the Science of Learning into Practice

How Students' Decisions to Space Their Practice are Related to Better Learning
Veronica X. Yan, Paulo F. Carvalho, and Faria Sana 445
Sweat So You Don’t Forget
Michelle Ogrodnik, Barbara Fenesi, and Jennifer J. Heisz 450
Learning How to Learn (Better) in Introduction to Psychology
Carolyn R. Brown-Kramer 458
Peer Discussions Improve Student Learning
Jonathan G. Tullis and Robert L. Goldstone 465
Using Online and Clicker Quizzes to Learn Scientific and Technical Jargon
Steven C. Pan, James E. Cooke, Jeri L. Little, and Mark A. McDaniel 473
The Value of Quizzing Students to Support Transfer of Learning
Lou Ann Griswold 481
Implementing Exam Wrappers
John E. Edlund 488
Wrapping up Lessons With Closed-Book and Open-Book Tests
Judith Schweppe and Ralf Rummer 494
Throw Them in the Deep End? Quizzing With Factual Versus Application Items
Danney Rasco 499
The Science of Virtual Teaching and Learning for Ophthalmology Medical Education
Tony Succar and John R. Grigg 505
Infusing Critical Thinking Skills Into Course Content
Laura C. Edwards 511
About the Authors 521
v
In Their Own Words Introduction: In Their Own Words . . .

You to Know About Why and How University of New Hampshire
Editors
Springfield College
Lauren L. Kordonowy
Victor A. Benassi

Lauren L. Kordonowy and Victor A. Benassi
University of New Hampshire
Over the past several decades, there has been a continually growing body of scholarship focusing on
conditions that promote students’ learning, retention, and transfer of academic knowledge. The term
‘science of learning’ is often used to describe this field of specialization. Much of this work has been
undertaken by psychologists, primarily from a cognitive perspective (Dunlosky et al., 2013; Fiorella &
Mayer, 2016; Lang, 2021; Mayer, 2011). Earlier investigations tended to be completed in controlled
laboratory contexts, and later research extended these investigations into authentic academic settings.
This book provides the next-generation of work in this area: applying the concepts from the science of
learning to course-based research. Work on the cognitive load theory of learning, for example, discussed
in depth by Ashman and Sweller (this volume), provides an excellent example of how science of learning
research has led to valuable insights and work on the science of instruction.
Scholars from a variety of specializations are also addressing questions related to how students learn
and what can be done to promote that learning. For example, in this volume, we include chapters by
discipline-based education research (DBER) scholars, whose work involves disciplines in chemistry (Frey
& Lewis, this volume) and physics (Docktor & Mestre, this volume). In addition, there is the well-
established area of the Learning Sciences (Sawyer, 2014). As described in Gurung’s chapter (this
volume), the scholarship of teaching and learning (SoTL) continues to develop, with an international
journal—International Journal of the Scholarship of Teaching and Learning—first published in 2007.
Similar to our previous volume (Benassi et al., 2014), the present book represents a selective but
substantial body of scholarship on the science of learning. As one index of growth over the past decade,
the current volume is nearly twice the page count as the 2014 book. Much, but not all, of the work
represented in the current volume was completed with college student samples. The book chapters
illustrate the interwoven nature of the authors’ research with directly applicable teaching practices for
college-level academic settings.
Readers of this book who work with students at other academic levels should also find much of interest
and relevance to their work. However, when considering applications to other student populations,
teachers should be circumspect when implementing research-based interventions previously
undertaken with college students. Indeed, critical evaluation should also be undertaken as teachers and
others apply interventions with any population of college students.
Most of the work described in this book is based on theory and research in cognitive psychology. We
might have selected other approaches and topics that have their research base in behavior analysis,
computational modeling and computer science, neuroscience, social and developmental psychology,
and other research areas. We agree with Newcombe (this volume) that there is additional relevant
scholarship directly related to science of learning and education which does not appear in this volume.
That work is for another time and perhaps for others to undertake.
1
Organization of Book
Part 1: Past, Present, and Future of Applying the Science of Learning in Education
The five chapters in Part 1 reflect the authors’ perspectives on the past, present, and future of the
application of science of learning in education. In this section, Bjork and Bjork discuss the concept of
desirable difficulties in an historical context. Newcombe “tried to broaden the horizons of what we
include in the science of learning” (p. 14), and she explores several perspectives and interventions not
specifically addressed in the current volume. Gurung describes the origins and evolution of work on SoTL
over the past 30 years, and he relates SoTL to contemporary scholarship on the science of learning.
Chew describes the current state of teaching and presents the Cognitive Challenge Framework to
organize and guide pedagogical research. Rowe and Hattie ask: “How effective are the learning sciences
at equipping teachers to improve student learning?” They answer their own question: “For all the
promising research that has been produced by learning scientists over the past three to four decades,
there is surprisingly little evidence to help us answer this question” (p. 43). These five chapters provide a
broad and deep context for the remaining chapters of the book.
Part 2: Science of Learning—Principles and Approaches
The 16 chapters in Part 2 address important concepts, principles, theories, and research findings related
to the science of learning. Most of the chapters are written by scholars who address their topic from the
perspective of cognitive psychology. In addition, we invited chapters representing discipline-based
education research (DBER), a broad and growing field led by scholars in STEM disciplines (Frey & Lewis,
this volume; Docktor & Mestre, this volume).
The array of topics explored in these chapters complement and, in some cases, extend topics covered in
our earlier book (Benassi et al., 2014). For example, the chapter on the testing effect in the classroom by
Yang et al. (this volume) describes findings from a recent meta-analysis, updating the previous chapter
by Pyc et al. (2014). Similarly, Renkl (this volume) provides a chapter on the use of worked examples in
problem solving in “ill defined” domains, extending the work described in his 2014 chapter on worked
examples in “well-defined” domains. There are also many chapters on topics not included in our 2014
book (e.g., productive failure, region of proximal learning, the ICAP framework, learning by teaching).
Despite the breadth of coverage of cognitively based science of learning research, there remains
additional scholarship in the learning sciences that is not addressed in the current volume (as noted by
Newcombe, this volume; see also Sawyer, 2014). Several additional volumes would be required to
provide a comprehensive view of the landscape.
Part 3: Preparing Faculty, Educational Developers, Student Success Professionals, and
others to Apply the Science of Learning
The chapters in Part 3 describe efforts to inform and prepare colleagues to apply science of learning
principles and practices in their instruction. These chapters describe a wide range of topics related
broadly to working with faculty and staff who want to use science of learning applications in their work
with students. Lovett et al. and Overson et al. (this volume) illustrate initiatives that teaching and
learning centers can undertake to promote the use of evidenced-based instructional methods by faculty.
Ekuni et al. (this volume) provide crucial information on study strategies use by students (WEIRD and
non-WEIRD). McCabe (this volume) details considerations for academic support staff concerning
learning strategy recommendations. Koedinger et al. (this volume) describe the Knowledge-Learning-
Instruction Framework, which is a guide to help teachers adapt instructional methods to their courses.
In addition, Sumeracki et al. (this volume) describe the work of The Learning Scientists.
2
Part 4: Preparing Students to Apply the Science of Learning
The chapters in Parts 2 and 3 are focused on interventions that can be applied by teachers to promote
their students’ learning. For example, a teacher may: create and deploy an assignment that requires
students to answer practice quiz questions after they watch a brief instructional video; create a series of
worked-out example problems which students complete as homework during earlier stages of learning
how to solve certain math problems; design and use a slide presentation during a lecture that
incorporates Mayer’s (this volume) multimedia principles of learning. Each of these interventions
involves a teacher creating an assignment or activity that includes an evidence-based instructional
intervention. Although these types of interventions have been shown to enhance learning for students
who have engaged in assigned activities, the chapters in Part 4 address whether students can be
instructed to use appropriate and effective learning strategies and whether they use these strategies on
their own. From survey research, we have learned that students often report using study strategies that
are associated with poorer learning (e.g., rereading to-be-learned material repeatedly) relative to
strategies that promote better learning (e.g., engaging in retrieval practice) (Ekuni et al., this volume).
The focus of chapters in this part of the book is on describing instructional approaches that directly
teach students about and instruct them in how to effectively use cognitively based study strategies. A
growing body of evidence demonstrates that students can learn to use these strategies. The question is
whether students will continue to use them on their own during study.
Part 5: Putting the Science of Learning into Practice
In Part 5, we provide 11 examples of authors’ perspectives on studies they completed in academic
settings that examined one or more science of learning principles. The topics addressed in these
chapters represent only a small segment of the published work on how the science of learning is applied
in academic settings. Still, they are excellent examples of the science of learning in action.
Suggestions for Reading the Book

• Readers can access chapters in any order; however, we recommend first reading the chapters in
Part 1. Because many of the chapters in the book are either directly or indirectly related to one
another, where relevant, we provided cross-referencing of chapters.
• We asked authors to refrain (when possible) from using jargon and technical details on research
methods and statistical analyses. We asked them to write their chapters in such a way that the
content is accessible to instructors and others from any field or discipline. If readers require or
want further explanation of the material included in chapters, consider the resources listed
below (Other Resources That May Be Useful to Teachers).
• Readers will notice that some of the chapters include descriptive statistics (e.g., Means [Ms] and
Standard Deviations [SDs]) and the results of statistical significance testing. Also, a number of
the chapters report a statistic, d, which represents statistical effect size estimates. More
specifically, d is
a measure of effect size based on the standardized difference between two means: It
indicates the number of standard deviation units by which the means of two data sets
differ. . . . The metric is used to represent effect sizes in meta-analysis as well as in the
determination of power, with values of 0.20, 0.50, and 0.80 representing small,
medium, and large effect sizes, respectively (American Psychological Association,
Dictionary of psychology).
3
Is the Research Described in This Book Ready for Prime Time?
We addressed this question previously (Benassi et al., 2014, pp. 3-4; see also articles by Daniel, Dunlosky
& Rawson, Mayer, and Roediger & Pyc [all in the December 2012 issue of the Journal of Applied
Research in Memory and Cognition]; Dunlosky et al., 2013; Roediger, 2013). In the current volume, the
authors of chapters in Part 1 address this question from several points of view.
Our current view is much the same as it was in 2014 (Benassi et al., 2014):
We agree that much more work needs to be done in a wide variety of educational contexts that
examine the principles and instructional methods presented in this book . . .. At the same time,
we also agree with Roediger and Pyc (2012) that we know a lot about cognitive principles and
that there is every reason to believe that they can be applied to good purpose in the academic
courses. (pp. 3-4)
As Roediger and Pyc (2012) aptly stated:
Do the principles governing learning stop when we switch from a lab to a classroom? All the
evidence we know leads us to suspect that generalizations can be made, even though, yes,
complexities will arise in the process and some pieces of advice will need to be revised as we
learn more. Of course, the data base of research in classroom experiments is not zero, after all,
and so far the returns seem promising. What is the downside of applying what we know now,
even if the knowledge is not perfect? (p. 263).
Despite the fact that the applied research described in this volume has been documented to be
applicable and efficacious in real educational settings, some scholars have considered why there is not
more widespread systematic use of interventions that promote student learning. Rowe and Hattie (this
volume) and Willingham and Daniel (2021) addressed this issue head on. One approach to addressing
this problem, at least at the college/university level, is through the work of teaching and learning
centers. The chapters in this volume by Benassi et al., Lovett et al., and Kuzmick et al. represent three
approaches focused on collaborating with instructors to develop, implement, and assess the impact of
‘evidence-based’ interventions in their academic courses.
Other Resources That May be Useful to Teachers

There is a growing number of books addressing science of learning and education (learning sciences)
that were written with nonexperts in mind (e.g., Ambrose et al., 2010; Fiorella & Mayer, 2015; Lang,
2021; Mayer, 2011; McGuire, 2015; National Academies of Sciences, Engineering, and Medicine, 2018;
Rhodes et al., 2019; Schwartz et al., 2016). Our book complements these and other similar resources by
including many of the same topics from the perspective of the scholars who have done basic and applied
research on the topics.
In addition to the many accessible books, we provide links (below) to illustrative webpages that supply a
wealth of information and recommendations on applying science of learning principles and techniques
in real academic settings:
The Learning Scientists:
https://www.learningscientists.org/
Retrieval Practice: Unleash the Science of Learning:
https://www.retrievalpractice.org/
Visible Learning: Make Learning Visible
4
https://www.visiblelearning.com
Science of Learning (The Institute for Learning and Teaching, Colorado State University)
https://tilt.colostate.edu/sol
Center for Excellence and Innovation in Teaching and Learning, University of New Hampshire:
https://www.unh.edu/professional-success/ceitl/grants/science-learning-grants
https://www.unh.edu/professional-success/ceitl/resources/student-cognition-toolbox-sct
Deans for Impact: The Science of Learning:
https://deansforimpact.org/resources/the-science-of-learning/
ManyClasses:
https://www.manyclasses.org/
LearnLab:
https://learnlab.org/
Johns Hopkins Science of Learning Institute:
http://scienceoflearning.jhu.edu/
Institute of Educational Sciences:
http://ies.ed.gov/
The International Society of the Learning Sciences (ISLS):
https://www.isls.org/about/
Center for Integrative Research on Cognition, Learning, and Education (CIRCLE), Washington
University:
https://circle.wustl.edu/about-circle/
The Human-Computer Interaction Institute at Carnegie Mellon University:
http://www.hcii.cmu.edu/
Author Note
We acknowledge funding by the Davis Educational Foundation to the University of New Hampshire for
its support over the past decade in our efforts to develop, implement, and assess the impact of science
of learning informed instructional interventions on students’ learning, retention, and transfer of
academic material in college courses. The Foundation was established by Stanton and Elisabeth Davis
after Mr. Davis's retirement as chairman of Shaw's Supermarkets, Inc. We also acknowledge the ongoing
support of the Office of the Provost and Vice President for Academic Affairs, University of New
Hampshire.
Please direct correspondence to Catherine Overson at [email protected]
References
Ambrose, S. A., Bridges, M. W., DiPietro, M., Lovett, M. C., & Norman, M. K. (2010). How learning works:
7 research-based principles for smart teaching. Jossey-Bass.
American Psychological Association (n.d.). Dictionary of psychology. https://dictionary.apa.org/cohens-
d).
Agarwal, P. K., & Bain, P. M. (2019). Powerful teaching: Unleash the science of learning. Jossey Bass.
5
Brown, P.C., Roediger, H. L, & McDaniel, M.A. (2014). Make it stick: The science of successful learning.
Harvard University Press.
Daniel, D. B. (2012). Promising principles: Translating the science of learning to educational practice.
Journal of Applied Research in Memory and Cognition, 1(4), 251–253.
https://doi.org/10.1016/j.jarmac.2012.10.004
Dunlosky, J., Rawson, K. A., Marsh, E. J., Nathan, M. J., & Willingham, D. T. (2013). Improving students'
learning with effective learning techniques: Promising directions from cognitive and educational
psychology. Psychological Science in the Public Interest, 14(1), 4-58.
https://doi.org/10.1177/1529100612453266
Fiorella, L., & Mayer, R. E. (2015). Learning as a generative activity. Cambridge University Press.
Lang, J. M. (2021). Small teaching: Everyday lessons from science of learning (2nd ed.). Jossey Bass.
Mayer, R. E. (2011). Applying the science of learning. Pearson.
McGuire, S. Y. (2015). Teach students to learn. Stylus.
National Academies of Sciences, Engineering, and Medicine. (2018). How people learn II: Learners,
contexts, and cultures. The National Academies Press.
Rhodes, M. G., Cleary, A. M., & DeLosh, E. L. (2019). Guide to effective studying and learning: Practical
strategies from the science of learning. Oxford University Press.
Roediger, H. L., & Pyc, M.A. (2012). Applying cognitive psychology to education: Complexities and
prospects. Journal of Applied Research in Memory and Cognition, 1(4), 263-
265. https://doi.org/10.1016/j.jarmac.2012.10.006
Roediger, H. L. (2013). Applying cognitive psychology to education: Translational educational
science. Psychological Science in the Public Interest, 14(1),1-3.
Sawyer, R. K. (Ed.). (2014). Cambridge handbook of the learning sciences (2nd ed.). Cambridge University
Press.
Schwartz, D. L., Tsang, J. M., & Blair, K. P. (2016). The ABCs of how we learn. W.W. Norton & Company.
Willingham, D.T., & Daniel, D.B. (2021). Making education research relevant: How researchers can give
teachers more choices. Education Next, 21(2), 28-33. https://www.educationnext.org/making-
education-research-relevant-how-researchers-can-give-teachers-more-choices/
6
In Their Own Words
Part 1
Past, Present, and Future of
Applying the Science of Learning
in Education
Editors
Lauren L. Kordonowy
Victor A. Benassi

7
In Their Own Words Constructing a Canon for the Science of
Learning
Nora S. Newcombe
Editors
Temple University
Lauren L. Kordonowy
Victor A. Benassi
An irony of college instruction is that most professors learn to teach simply by drawing on memories of
their own experience as students, coupled with trial and error. This lack of formal preparation stands in
clear contrast to the extensive pre-service curriculum common for K-12 instructors. Thus, one answer to
the problem of preparing college instructors would seem to be to offer courses to graduate students
and current professors on teaching better. However, doing so based on evidence is challenging. The pre-
service curriculum for K-12 teachers has come under increasing fire for failing to use the advancing
science of learning and alternatives are appearing (Deans for Impact, 2015). Existing programs at
universities seem similar. They often use modules and mini-courses that rely more on anecdote and
testimonial than on science.
Luckily, guides to remedying this situation are appearing. One is the present book, whose chapters
describe many powerful evidence-based techniques that both instructors and students can adopt, such
as prequestioning, relearning, interleaving, embedded questions and more. Although the techniques
require thought, effort, and careful design to use well, they do work. They have a concrete and
graspable quality that make them attractive. In my view, however, the “strategy approach” represents
only a selective sampling of what we know about how people learn (National Academies of Sciences,
Engineering, and Medicine, 2018). Ideally, this book would be “Volume 1” in a more complete overview.
Science of learning is broader in at least four ways.
• First, our knowledge of how people learn ought to extend to atypically developing learners, to
learners who may not share the cultural assumptions of professors, and to students engaged in
informal learning, as well as to younger and older learners. To what extent do the strategies
explained in this book apply broadly to other populations and other contexts? Do they apply to
all of the students that college professors may encounter, who vary in ability, preparation, and
motivation?
• Second, college teaching typically requires consideration of discipline and domain-specific
issues, which the strategy approach often neglects. This book contains some domain-focused
chapters, such as on teaching physics and chemistry, but most of the chapters are domain-
general. Do learning strategies apply equally well to all of the kinds of courses taught at the
university level? Do we need additional domain-specific strategies?
• Third, there are learning and instructional tools other than strategies; some examples are
analogy, gesture, teaching map, diagram and graph understanding, and using principles of
embodied learning. Are these approaches not also strategies for teaching and learning,
construed broadly?
• Fourth, the contemporary science of learning increasingly involves neuroscience. Although Bruer
(1997) labeled educational neuroscience a “bridge too far”, many current investigators think it
8
has come of age. Can neuroscience provide a methodology for answering questions about
teaching that are difficult to tackle behaviorally, as well as enriching our understanding of why
behavioral principles work, when they do?
Range of Application
We do not yet know as much as we should about whether instructional and study strategies work
equally well for learners with different levels of preparation or motivation or with varying degrees of
general cognitive skills (e.g., variations in how many pieces of information students can hold in mind at
once). One clue comes from investigations of how well strategies work for younger children. There is
both good and bad news on this question.
On the good news side, a meta-analysis of retrieval-based learning by Agarwal et al. (2021) identified a
set of 50 classroom studies, 20 of which involved K-12 students; the data showed substantial effect sizes
in the K-12 setting—actually larger than the effects seen in undergraduate or medical school settings.
However, a limitation was that most of the data came from a single school system in the area of the St.
Louis, Missouri.
In more mixed news, a narrative review by Brod (2020), drawing on meta-analytic results when possible,
examined how well six generative learning strategies worked across various age ranges. The six
techniques were concept mapping, explaining, predicting, questioning, testing, and drawing. Although
there was evidence that all six techniques work for university students, and that five work for high
school students (with mixed evidence for questioning), the data for younger children were less clear,
and even unfavorable for questioning and drawing. (Interestingly, testing seemed effective across the
board, suggesting that Missouri may not be the only setting in which we see the utility of retrieval for
children.) While university instructors may take comfort from this review, it is worth pausing to consider
whether the samples studied so far may have been too homogeneous. Brod points out that variations in
effect sizes with age are likely linked to changes with age in background knowledge, cognitive capacities,
and metacognitive abilities. University students surely vary along all of those dimensions, and college
instructors face considerable individual variation within their classes. Could the overall positive effect
disguise considerable individual variation in effectiveness?
Another piece of mixed news comes from a narrative review of retrieval-based learning in children by
Fazio and Marsh (2019). They argued that, although the general principle of “desirable difficulty” (Bjork
& Bjork, 2011) works across the age spectrum, the trick is titrating the amount of challenge, so it is age-
appropriate—not too much or too little. Younger children may need scaffolding for their retrieval, but
how much is too much (cf. Bjork & Bjork, this volume)? Although spacing is helpful, what amounts of
spacing are desirable at what ages? Feedback can help older children, but it may not help preschoolers.
Agarwal et al. (2021) agree with this point, and they point out that data on variations on these
dimensions is rare across the age range as well as in higher-level classes.
College instructors must make many detailed decisions about exactly how to implement strategies. They
face specific parametric choices regarding how much to do in what format and when, and whether and
how to tailor it to particular students. Students face similar problems in planning their study sessions.
Even though some of these decisions might be based on evidence (e.g., there is a powerful model of
spacing effects based on a large sample, Cepeda et al., 2008), it is difficult to imagine that experimental
work by individual investigators can answer all the questions we need answered. Indeed, the number of
combinations of possible instructional strategies is intractably large (Koedinger et al., 2013), and it is
clear that hitting the zone of proximal learning (see Metcalfe, this volume) can be challenging.
9
One answer is larger projects executed closer to scale. Such work is possible with LearnLab (Koedinger et
al., 2013). Another way to do the necessary research may be via a new effort called Many Classes, in
which instructors at a variety of institutions and across many disciplines pool their data on mutually
agreed upon problems. In the first paper from this effort, Fyfe et al. (2021) explore whether feedback
works better if immediate or if delayed. Despite arguments on both sides, in this pre-registered
experiment, the group found no difference in effectiveness as a function of delay. Arguably, that result is
disappointing, but it ought not to be. Exactly when feedback comes may matter less than we thought it
did, at least in real classrooms.
Discipline-Specific Issues
Instructors in specific disciplines often face distinctive challenges to learning that do not seem easily
addressable using general learning strategies. For example, mathematics educators need to address
issues as basic as understanding the equal sign, even in college teaching (Fyfe et al., 2020). Physics
instructors need to work to alter naïve conceptions of mechanics (Harris et al., 2018; see Docktor &
Mestre, this volume). Chemistry educators need to grapple with whether their students can visualize
molecules and, if not, how to help them do so (Stieff et al., 2014). Geoscience students come to class
with erroneous ideas about how visible two-dimensional surfaces constrain inference about the three-
dimensional composition of rock formations (Gagnier & Shipley, 2016). Many sciences deal with scales
either too large or too small to exist within human experience, and students need considerable support
in grasping these dimensions of time and space (Resnick et al., 2017). An additional issue is that science
classes often include lab work or field trips, which are not often a part of other kinds of study. Should
their work in these contexts be discovery learning, should it be closely guided by lab manuals, or should
it be a blend, or progression, or alternation, or even something else entirely?
The tradition of discipline-based education research (DBER) grew up to address the need to think
through issues of this kind (NRC, 2012). Various disciplines organize resources and conferences to
support instructors in teaching particular kinds of content (e.g., the Science Education Resource Center
at Carleton College). There are degrees, journals, and centers in fields such as physics education,
chemistry education and geoscience education. However, although there is cross-fertilization to some
extent across the science disciplines, it is less than might be desirable, and there is still less interaction
with the mainstream science of learning, as housed largely in departments of psychology. Faculty with
degrees in areas such as physics education often experience tension in terms of whether their natural
homes are in colleges of education or disciplinary departments in the sciences.
Problems arise due to this lack of communication. An important example concerns the uncritical
embrace of the excellent concept of “active learning” (see Chi & Boucher, this volume) in teaching
science, which the 2012 NRC DBER report endorsed, and which really caught on at DBER workshops.
One of the central supports for active learning in STEM is a much-cited meta-analysis of 225 studies of
achievement in college classes in eight STEM subjects. Freeman et al. (2014) reported that failure rates
in courses using traditional lecturing methods were 1.5 times greater than the rates in classes using
active learning, and that active learning would increase grades by half a letter. These results seemed to
hold across disciplines and class sizes. However, Freeman et al.’s analysis simply contrasted traditional
lectures with any other method of teaching, lumping together changes to lecture classes as small as
using clicker systems with more substantial changes, such as asking students to make concept maps, or
even major alterations in traditional teaching, such as flipping the classroom (i.e., asking students to
watch videos of lectures and using class time for questions and discussion). There are separate meta-
analyses of many of these techniques, and their effects are variable, even within a single technique. For
example, a small significant advantage for flipped classrooms varies by subject area (Cheng et al., 2019).
(Some of the techniques, including clickers and concept maps, appear in this book, and separate
10
chapters evaluate these methods.) On the other side of the dichotomy, traditional instruction is not
monolithic. Traditional instruction may well work if it is well-paced and contains necessary explanations
(Hallinen et al., 2021).
Lombardi et al. (2021) organized a major effort to define what active learning in STEM means, to think
through how to conduct research on it, and how to act on what we know now. In an extensive cross-
disciplinary collaboration, they combined their knowledge of research in education and cognitive
science with syntheses of DBER research prepared by teams centered in astronomy, biology, chemistry,
engineering, geography, geoscience, and physics. Collaboratively, they proposed a new conceptual
model of active learning (see Figure 1). There are several points to note in the figure. First, the four
science content areas should engage the student as well as the teacher. Second, interaction between
the teacher and the student should be two-way, not simply one-way instruction. Third, students should
interact with each other. Last, students should engage in reflective practices to solidify and expand their
own understanding. Activities implementing these practices may be very different in nature and
intensity. For example, peer-led tutorial sessions are an intensive way to use the principle of peer
interaction (see Frye & Lewis, this volume), whereas a think-pair-share activity, in which students think
over an idea, discuss it with a partner, and share conclusions with the class, can be a brief interpolation
in a traditional lecture.
Figure 1
Contrasting Traditional and Active Learning
Note. From The curious construct of active learning, by Lombardi et al. (2021), Psychological Science in
the Public Interest, 22(1), p. 18. Reprinted with permission.
11
Although this framework is helpful in thinking about what active learning is, it is still the case that many
specific instructional techniques are combined under this umbrella term. We are left with the dilemma
that it is not possible to evaluate what active learning strategies work when and for whom, given the
diversity of practices that fit the very large category. The solution to this problem is not clear. Perhaps a
LearnLab or Many Classes approach could help. It would also be helpful to delineate specific learning
principles that explain why and when a practice helps student learning. For example, the principle that it
is helpful to make predictions and see them confirmed (or not) helps explain the finding that clicker use
is only helpful when students engage with making predictions (Resnick et al., 2017). A very different use
of classroom clickers is to evaluate the prevalence of students’ misconceptions and even to reveal new
ones (LaDue & Shipley, 2018).
Interestingly, the controversy over “active learning” at the university level is very similar to a debate
regarding “discovery learning” or “inquiry-based learning” at the K-12 level. Brod (2021) pointed out
that several meta-analyses of this literature have shown mixed or small effects. Key issues affecting the
size of the advantages for a nontraditional approach are whether students receive sufficient guidance
and support. In addition, the practices seem to work better with older students, perhaps surprisingly
given the idea that small children are curious and exploratory (Yu et al., 2018). Brod suggested the need
to add students’ prior knowledge, cognitive capacities, and metacognitive skills (the list we saw in the
section above) to the model proposed by Lombardi et al. (2021), as well as students’ beliefs in their
capacity to direct their own learning. However, these student-level variables surely vary within
university classes as well, raising the need to look at within-class differences in responsiveness to active
teaching practices.
The Instructional Toolbox

Many techniques for enhancing teaching and learning fit uneasily under the traditional rubric of
strategies. This section covers four additional tools for improving learning to consider adopting in the
university classroom. The first two tools are likely applicable across most disciplines, while the second
two are likely more useful for STEM instruction. For all four, effective implementation requires thought
because there are more and less effective ways to use the tools.
Analogy
Science instruction often uses analogy, as when textbooks compare the atom to the solar system, or
when a lecturer scales the geologic time scale onto the human life span. Meta-analysis demonstrates
that analogies (also called case comparisons) work to enhance learning (Alfieri et al., 2013). They work
especially well when instructors couple them with active learning (asking learners to find similarities
between cases), but they also work well with some direct teaching (providing principles after the
comparisons). The meta-analysis showed that using images was also helpful, as was testing learners
immediately after presenting material. The technique works for university students (Goldwater &
Gentner, 2015; Jee et al., 2013; Resnick et al., 2017), as well as in the elementary classroom and in
children’s museums (for reviews: Richland & Sims, 2015; Vendetti et al., 2015).
Gesture
Gesture, being both physical and abstract, has many strengths as a communicative tool (Goldin-Meadow
& Wagner, 2005; Novack & Goldin-Meadow, 2015). Learners’ gestures provide a window into what
students know, especially when what they say does not match with what they gesture; in this case,
gesture is often in advance of speech and indicates a readiness to learn. As an example, a gesture may
show the correct grouping of numbers in an algebraic equation even when speech does not. Teachers’
gestures can guide learners, not only by focusing their attention, but also by communicating concepts
12
such as mathematical grouping. Teaching students to perform certain gestures, such as a v with the
fingers to show correct grouping, can enhance their learning and lead to more durable knowledge and
generalization. A particular strength is that gesture can express spatial relations easily and can indicate
relations in an analogue fashion. While many of the gesture studies involve children learning
mathematical concepts, the general principles extend to children engaged in moral reasoning
(Beaudoin‐Ryan & Goldin‐Meadow, 2014) and to university students learning chemistry (Ping et al.,
2021) and geology (Atit et al., 2015).
Reading Diagrams, Graphs, and Maps
Diagrams and graphs play a ubiquitous role in STEM learning, in the social sciences and, increasingly, in
the popular press. Maps are also very useful in showing patterns and relations, and they are key to
communicating some kinds of information, such as patterns of elevation. Many instructors, however,
assume that reading these representations does not require instruction. Yet, findings show that when
students get explicit teaching in reading them, their learning benefits (Bergey et al., 2015). Although
people often assume that dynamic representations (e.g., a video or animation) add value, static
representations are often sufficient and sometimes better, including in teaching about processes such as
lightning and ocean waves (Mayer et al., 2005). We also need to specify better for early educators when
and how to introduce diagrams, graphs, and maps (Uttal & Yuan, 2014); it is clear that their use should
not be delayed, but rather start early and with careful sequencing and support.
Embodied Learning
If active learning is helpful, then one might expect that literal manifestations of active learning, i.e.,
where students act physically in the world, might be yet more helpful. In fact, some theorists suggest
that all human cognition is embodied (Barsalou, 2008). Nathan and Walkington (2017) proposed an
embodied approach to mathematical learning with a focus on pre-college geometry. There may be some
STEM ideas that are particularly suited to learning from embodied experiences, especially for concepts
such as angular momentum that have corresponding physical ways in which they can be felt (Kontra et
al., 2015). In another example, students who feel simulated earthquakes learn better, although they also
needed to put their experiences of vibration onto a map to understand the pattern and principles
(Jaeger et al., 2016). Stull and Hegarty (2016) found that manipulating models, either physically or
virtually, helped students learn organic chemistry. However, hands-on experience is not always helpful.
For example, in a study of learning about what makes objects sink, Castillo et al. (2017) found it could be
harmful. In sum, using hands-on experience or embodied cognition approaches needs careful thought.
Research and practice will benefit from using the lens of thinking about an action-to-abstraction
continuum (Goldin-Meadow, 2015) for thinking about gesture, in which learners grasp some operations
and concepts especially well when embedded in a body-based experience, but they also need to
abstract the ideas to weave them into an overall system of knowledge.
Using Neuroscience
Neuroscience investigation, largely in the form of functional magnetic resonance imaging (fMRI), has
begun to provide important insights for the science of learning. Of course, the idea that experience
changes the brain is in some ways trivially true: if behavior changes, then neural representations must
have changed to support that behavioral change. However, neural studies can now throw light on the
nature of learning. One case in point comes from the Kontra et al. (2015) study of students learning
angular momentum through viewing an embodied experience. Not only did students have higher quiz
scores than a comparison group of students who simply observed the demonstration, their
improvement correlated with activation in sensorimotor regions of the brain, confirming the embodied
13
account against alternatives, e.g., enhanced overall attention. Another example where fMRI confirms
what behavioral work had already suggested are neural investigations showing that both mathematical
reasoning (Amalric & Dehaene, 2016) and transitive reasoning (Alfred et al., 2020) are fundamentally
spatial in nature.
In other cases, working at the neural level can lead to new insights. For instance, Cetron et al. (2020)
investigated the representation of various kinds of real-world building structures for engineering
students who had taken at least two engineering courses with labs, intended for majors, in comparison
to novice peers, with no engineering courses. It turned out that the engineers represented relevant
groupings of the building structures in semantic coding portions of their brain, even without explicit task
demands, as what they learned in their courses becomes part of their categorical knowledge about the
world. Furthermore, fMRI-derived patterns of brain activity correlate with individual differences in
multiple concept knowledge tasks in the physics and engineering domains (Cetron et al., 2019). A
second example builds on behavioral work indicating that naïve theories and intuitive conceptions co-
exist with scientific knowledge acquired from courses (Goldberg & Thompson-Schill, 2009). However,
work at the neural level shows that learners need to inhibit thinking about the earlier ideas (see review
by Mason & Zaccoletti, 2021), showing the cognitive effort required to think scientifically, and adding
substance to theories of cognitive load.
A third use of neural data is to elucidate why some neuromyths are so stubborn. One widely circulated
myth concerns learning styles (e.g., visual and verbal learners), which persists despite lack of supportive
evidence (Pashler et al., 2008). Why are they so appealing? One possibility is that they conform to
people’s introspections that they think one way or the other. Kraemer et al. (2009) presented neural
evidence that such intuitions may be correct. In fact, the subjective styles may correlate with subtle
differences in how people learn to navigate (Kraemer et al., 2017) and may appear as variation in the
classroom, as when students who are high in spatial ability show a flare for STEM learning; observing
such phenomena may confirm teachers’ beliefs in different types of learners (Alfred & Kraemer, 2017).
Viewed this way, the myth is not that the styles exist, but rather the associated inference that different
kinds of learners benefit from different kinds of instruction. It is the latter hypothesis that is
unsupported in analyses such as the review by Pashler et al. (2008).
Conclusion
Science of learning brings together the disciplines of psychology, education, and neuroscience to create
a dynamic field, which is continuously evolving and changing. This dynamic quality poses a challenge for
the writers of textbooks and the designers of curricula. How do they keep abreast of changes? What old
principles need modification or even abandonment? What new principles are solid enough to warrant
addition? Most seriously, how can students come to appreciate that the knowledge we have today is
almost never as complete as the knowledge we will have in a few years, and yet it is better than
intuition and anecdote? In this chapter, I have tried to broaden the horizons of what we include in the
science of learning, but I have not tackled these questions. Determining how to integrate a true science
with an often-static system of accreditation and licensure is a challenge we face, in common with other
professions.
Author Note
Corresponding author email: [email protected]. Work on this chapter was supported by a grant
from the National Science Foundation EHR 1660996.
14
References
Agarwal, P. K., Nunes, L. D., & Blunt, J. R. (2021). Retrieval practice consistently benefits student
learning: A systematic review of applied research in schools and classrooms. Educational
Psychology Review, 33 (1), 1-45. https://doi.org/10.1007/s10648-021-09595-9
Alfieri, L., Nokes-Malach, T. J., & Schunn, C. D. (2013). Learning through case comparisons: A meta-
analytic review. Educational Psychologist, 48(2), 87-113.
https://doi.org/10.1080/00461520.2013.775712
Alfred, K. L., Connolly, A. C., Cetron, J. S., & Kraemer, D. J. (2020). Mental models use common neural
spatial structure for spatial and abstract content. Communications Biology, 3(1), 1-11.
https://doi.org/10.1038/s42003-019-0740-8
Alfred, K. L., & Kraemer, D. J. (2017). Verbal and visual cognition: Individual differences in the lab, in the
brain, and in the classroom. Developmental Neuropsychology, 42(7-8), 507-520.
Amalric, M., & Dehaene, S. (2016). Origins of the brain networks for advanced mathematics in expert
mathematicians. Proceedings of the National Academy of Sciences, 113(18), 4909-4917.
https://doi.org/10.1073/pnas.1603205113
Atit, K., Gagnier, K., & Shipley, T. F. (2015). Student gestures aid penetrative thinking. Journal of
Geoscience Education, 63(1), 66-72. https://doi.org/10.5408/14-008.1
Barsalou, L. W. (2008). Grounded cognition. Annual Review of Psychology, 59, 617-645.
https://doi.org/10.1146/annurev.psych.59.103006.093639
Beaudoin‐Ryan, L., & Goldin‐Meadow, S. (2014). Teaching moral reasoning through gesture.
Developmental science, 17(6), 984-990. https://doi.org/10.1111/desc.12180
Bergey, B. W., Cromley, J. G., & Newcombe, N. S. (2015). Teaching high school biology students to
coordinate text and diagrams: Relations with transfer, effort, and spatial skill. International
Journal of Science Education, 37(15), 2476-2502.
Bjork, E. L. & Bjork, R. A. (2011). Making things hard on yourself, but in a good way: Creating desirable
difficulties to enhance learning. In M. A. Gernsbacher, R. W. Pew, L. M. Hough & J. R. Pomerantz
(Eds.), Psychology and the real world: Essays illustrating fundamental contributions to
society (pp. 56-64). Worth Publishers.
Brod, G. (2020). Generative learning: Which strategies for what age? Educational Psychology Review,
33(4), 1295–1318. https://doi.org/10.1007/s10648-020-09571-9
Brod, G. (2021). How can we make active learning work in K–12 education? Considering Prerequisites for
a Successful Construction of Understanding. Psychological Science in the Public Interest, 22(1), 1-
7. https://doi.org/10.1177/1529100621997376
Bruer, J. T. (1997). Education and the brain: A bridge too far. Educational Researcher, 26(8), 4-16.
https://doi.org/10.3102/0013189X026008004
Castillo, R. D., Waltzer, T., & Kloos, H. (2017). Hands-on experience can lead to systematic mistakes: A
study on adults’ understanding of sinking objects. Cognitive Research: Principles and
Implications, 2(1), 1-12. https://doi.org/10.1186/s41235-017-0061-8
15
Cepeda, N. J., Vul, E., Rohrer, D., Wixted, J. T., & Pashler, H. (2008). Spacing effects in learning: A
temporal ridgeline of optimal retention. Psychological Science, 19(11), 1095-1102.
https://doi.org/10.1111/j.1467-9280.2008.02209.x
Cetron, J. S., Connolly, A. C., Diamond, S. G., May, V. V., Haxby, J. V., & Kraemer, D. J. (2019). Decoding
individual differences in STEM learning from functional MRI data. Nature Communications,
10(1), 1-10. https://doi.org/10.1038/s41467-019-10053-y
Cetron, J. S., Connolly, A. C., Diamond, S. G., May, V. V., Haxby, J. V., & Kraemer, D. J. (2020). Using the
force: STEM knowledge and experience construct shared neural representations of engineering
concepts. NPJ Science of Learning, 5(1), 1-10. https://doi.org/10.1038/s41539-020-0065-x
Cheng, L., Ritzhaupt, A. D., & Antonenko, P. (2019). Effects of the flipped classroom instructional
strategy on students’ learning outcomes: A meta-analysis. Educational Technology Research and
Development, 67(4), 793-824. https://doi.org/10.1007/s11423-018-9633-7
Deans for Impact (2015). The science of learning. Deans for Impact.
https://deansforimpact.org/resources/the-science-of-learning/
Fazio, L. K., & Marsh, E. J. (2019). Retrieval-based learning in children. Current Directions in Psychological
Science, 28(2), 111-116. https://doi.org/10.1177/0963721418806673
Freeman, S., Eddy, S. L., McDonough, M., Smith, M. K., Okoroafor, N., Jordt, H., & Wenderoth, M. P.
(2014). Active learning increases student performance in science, engineering, and
mathematics. Proceedings of the National Academy of Sciences, 111(23), 8410-8415.
Fyfe, E. R., de Leeuw, J. R., Carvalho, P. F., Goldstone, R. L., Sherman, J., Admiraal, D., Alford, L. K.,
Bonner, A., Brassil, C. E., Brooks, C. A., Carbonetto, T., Chang, S. H., Cruz, L., Czymoniewicz-
Klippel, M., Daniel, F., Driessen, M., Habashy, N., Hanson-Bradley, C. L., Hirt, E. R., . . . Motz, B. A.
(2021). ManyClasses 1: Assessing the generalizable effect of immediate feedback versus Delayed
feedback across many college classes. Advances in Methods and Practices in Psychological
Science, 4(3). https://doi.org/10.1177/25152459211027575
Fyfe, E. R., Matthews, P. G., & Amsel, E. (2020). College developmental math students’ knowledge of the
equal sign. Educational Studies in Mathematics, 104, 65-85. https://doi.org/10.1007/s10649-
020-09947-2
Gagnier, K. M., & Shipley, T. F. (2016). Visual completion from 2D cross-sections: Implications for visual
theory and STEM education and practice. Cognitive Research: Principles and Implications, 1(1),
1-18. https://doi.org/10.1186/s41235-016-0010-y
Goldberg, R. F., & Thompson-Schill, S. L. (2009). Developmental “roots” in mature biological knowledge.
Psychological Science, 20(4), 480-487. https://doi.org/10.1111%2Fj.1467-9280.2009.02320.x
Goldin-Meadow, S. (2015). From action to abstraction: Gesture as a mechanism of change.
Developmental Review, 38, 167-184. https://doi.org/10.1016%2Fj.dr.2015.07.007
Goldin-Meadow, S., & Wagner, S. M. (2005). How our hands help us learn. Trends in Cognitive Sciences,
9(5), 234-241. https://doi.org/10.1016/j.tics.2005.03.006
Goldwater, M.B., & Gentner, D. (2015). On the acquisition of abstract knowledge: Structural alignment
and explication in learning causal system categories. Cognition, 137, 137-153.
https://doi.org/10.1016/j.cognition.2014.12.001
16
Hallinen, N. R., Sprague, L. N., Blair, K. P., Adler, R. M., & Newcombe, N. S. (2021). Finding formulas:
Does active search facilitate appropriate generalization? Cognitive Research: Principles and
Implications, 6(1), 1-18. https://doi.org/10.1186/s41235-021-00316-y
Harris, J., George, N. R., Hirsh-Pasek, K., & Newcombe, N. S. (2018). Where will it go? How children and
adults reason about force and motion. Cognitive Development, 45, 113-124.
https://doi.org/10.1016/j.cogdev.2018.01.002
Jaeger, A. J., Wiley, J., & Moher, T. (2016). Leveling the playing field: Grounding learning with embedded
simulations in geoscience. Cognitive Research: Principles and Implications, 1(1), 1-14.
Jee, B. D., Uttal, D. H., Gentner, D., Manduca, C., Shipley, T. F., & Sageman, B. (2013). Finding faults:
Analogical comparison supports spatial concept learning in geoscience. Cognitive Processing,
14(2), 175-187. https://doi.org/10.1186/s41235-016-0026-3
Koedinger, K. R., Booth, J. L., & Klahr, D. (2013). Instructional complexity and the science to constrain it.
Science, 342(6161), 935-937. https://doi.org/10.1126/science.1238056
Kontra, C., Lyons, D. J., Fischer, S. M., & Beilock, S. L. (2015). Physical experience enhances science
learning. Psychological Science, 26(6), 737-749. https://doi.org/10.1177/0956797615569355
Kraemer, D. J., Rosenberg, L. M., & Thompson-Schill, S. L. (2009). The neural correlates of visual and
verbal cognitive styles. Journal of Neuroscience, 29(12), 3792-3798.
https://doi.org/10.1523%2FJNEUROSCI.4635-08.2009
Kraemer, D. J., Schinazi, V. R., Cawkwell, P. B., Tekriwal, A., Epstein, R. A., & Thompson-Schill, S. L.
(2017). Verbalizing, visualizing, and navigating: The effect of strategies on encoding a large-scale
virtual environment. Journal of Experimental Psychology: Learning, Memory, and Cognition,
43(4), 611. https://doi.org/10.1037%2Fxlm0000314
LaDue, N. D., & Shipley, T. F. (2018). Click-on-diagram questions: A new tool to study conceptions using
classroom response systems. Journal of Science Education and Technology, 27(6), 492-507.
Lombardi, D., Shipley, T. F., & Astronomy Team, Biology Team, Chemistry Team, Engineering Team,
Geography Team, Geoscience Team, and Physics Team. (2021). The curious construct of active
learning. Psychological Science in the Public Interest, 22(1), 8-43.
Mayer, R. E., Hegarty, M., Mayer, S., & Campbell, J. (2005). When static media promote active learning:
annotated illustrations versus narrated animations in multimedia instruction. Journal of
Experimental Psychology: Applied, 11(4), 256-265. https://doi.org/10.1037/1076-898x.11.4.256
Mason, L., & Zaccoletti, S. (2021). Inhibition and conceptual learning in science: a review of studies.
Educational Psychology Review, 33(1), 181-212. https://doi.org/10.1007/s10648-020-09529-x
Nathan, M. J., & Walkington, C. (2017). Grounded and embodied mathematical cognition: Promoting
mathematical insight and proof using action and language. Cognitive Research: Principles and
Implications, 2(1), 1-20. https://doi.org/10.1186%2Fs41235-016-0040-5
National Academies of Sciences, Engineering, and Medicine. (2018). How people learn II: Learners,
contexts, and cultures. National Academies Press.
National Research Council. (2012). Discipline-based education research: Understanding and improving
learning in undergraduate science and engineering. National Academies Press.
17
Novack, M., & Goldin-Meadow, S. (2015). Learning from gesture: How our hands change our minds.
Educational Psychology Review, 27(3), 405-412. https://doi.org/10.1007%2Fs10648-015-9325-3
Pashler, H., McDaniel, M., Rohrer, D., & Bjork, R. (2008). Learning styles: Concepts and evidence.
Psychological Science in the Public Interest, 9(3), 105-119. https://doi.org/10.1111/j.1539-
6053.2009.01038.x
Ping, R., Church, R. B., Decatur, M. A., Larson, S. W., Zinchenko, E., & Goldin-Meadow, S. (2021).
Unpacking the gestures of chemistry learners: What the hands tell us about correct and
incorrect conceptions of stereochemistry. Discourse Processes, 58(3), 213-232.
https://doi.org/10.1080%2F0163853X.2020.1839343
Resnick, I., Davatzes, A., Newcombe, N. S., & Shipley, T. F. (2017). Using analogy to learn about
phenomena at scales outside of human perception. Cognitive Research: Principles and
Implications, 2(2)1, 1-17. https://doi.org/10.1186/s41235-017-0054-7
Resnick, I., Newcombe, N. S., & Shipley, T. F. (2017). Dealing with big numbers: Representation and
understanding of magnitudes outside of human experience. Cognitive Science, 41(4), 1020-1041.
https://doi.org/10.1111/cogs.12388
Richland, L. E., Simms, N. (2015), Analogy, higher order thinking, and education, WIREs: Cognitive
Science. Volume 6(2), 177-192. https://doi.org/10.1002/wcs.1336
Stieff, M., Dixon, B. L., Ryu, M., Kumi, B. C., & Hegarty, M. (2014). Strategy training eliminates sex
differences in spatial problem solving in a stem domain. Journal of Educational Psychology,
106(2), 390. https://doi.org/10.1037/a0034823
Stull, A. T., & Hegarty, M. (2016). Model manipulation and learning: Fostering representational
competence with virtual and concrete models. Journal of Educational Psychology, 108(4), 509.
https://doi.org/10.1037/edu0000077
Uttal, D.H., & Yuan, L. (2014). Using symbols: Developmental perspectives. Wiley Interdisciplinary
Reviews: Cognitive Science, 5(3), 295-304. https://doi.org/10.1002/wcs.1280
Vendetti, M., Matlen, B., Richland, L., & Bunge, S. (2015). Analogical reasoning in the classroom: Insights
from cognitive science. Mind, Brain, and Education, 9(2), 100–106.
https://doi.org/10.1111/mbe.12080
Yu, Y., Landrum, A. R., Bonawitz, E., & Shafto, P. (2018). Questioning supports effective transmission of
knowledge and increased exploratory learning in pre‐kindergarten children. Developmental
Science, 21(6), e12696. https://doi.org/10.1111/desc.12696
18
In Their Own Words Introducing Desirable Difficulties Into Practice
and Instruction: Obstacles and Opportunities
Elizabeth L. Bjork and Robert A. Bjork
Editors
University of California, Los Angeles
Lauren L. Kordonowy
Victor A. Benassi
Increasingly, learning is happening outside of classroom settings, and computer-based technologies

offer new options for managing our own, our students’, or our children’s learning. As teachers, parents,
or learners ourselves, we can, however, be susceptible to beliefs and illusions that can lead us to mis-
assess and mis-manage our own and other’s learning. A variety of research findings suggest that the
potential exists for learners and instructors alike to enhance self-regulated and teacher-regulated
learning in innovative ways, but doing so can run counter to both prevailing practices and learners’
intuitions.
The basic problem, as we have emphasized elsewhere (e.g., Bjork & Bjork, 2011, 2014), is that enhancing
self-regulated learning and teaching involves “making things hard on yourself, but in a good way”—and
also making things hard on one’s students or children as well. What we meant by “making things hard”
is introducing a set of difficulties into one’s own or one’s students’ learning, including: varying the
conditions of study and practice, rather than keeping them constant and predictable; interleaving the
study or instruction on separate topics, rather than grouping or blocking such study or instruction by
topic; spacing, instead of massing study sessions on a given topic; and using tests, rather than repeated
presentations, as learning events.
Learning Versus Performance; Research Versus Intuition

We have also tried to provide learners with a few easy-to-remember precepts that can guide their
future study or teaching activities. Namely, that (a) current performance is not a reliable index of
learning, which can only be measured by recall after a delay; that (b) forgetting—rather than being the
enemy of learning or a process that tears down what learning has built up—is a process that can provide
opportunities for enhanced learning, thus serving an important role in the adaptive system of human
learning and memory; that (c) we cannot trust our instincts as to how best to optimize study and
practice; indeed, to optimize learning and instruction often requires going against one’s intuitions, and
that (d) using one’s memory changes one’s memory, both by making retrieved information more
recallable than it would have been otherwise (e.g., Bjork, 1975; Landauer & Bjork, 1978) and by making
information in competition with the retrieved information less recallable in the future (see Anderson et
al., 1994).
Confronting the Challenge That Desirable Difficulties Are Typically Undesired
Having been asked to convey in “our own words” what we most want students and teachers to know
regarding how to apply findings from the science of learning has led us to think back on our efforts to
spread the desirable difficulties gospel, so to speak. It verges on laughable that we thought 25 years or
so ago that we would simply tell people about certain key findings, and they would then immediately
change how they managed their own learning. First of all, desirable difficulties are still difficulties from a
19
learner’s standpoint, and doing anything that might impair one’s performance during the learning
process is not very appealing, especially when doing so may also entail greater effort. Additionally, our
advice runs counter to the advice many learners may have been given by their teachers, explicitly or
implicitly, creating a who-do-I-trust problem. And, finally, there is a win-stay/lose-shift conundrum for
many individuals, especially for those in the college audiences with whom we have talked who may well
think “what I have been doing has gotten me to where I am, so why should I change?”
What is needed, therefore, for many learners and teachers is a more complete rationale for when and
why difficulties can be desirable. In the next section we provide an overview of a theoretical framework
that provides a rationale for why difficulties can be desirable.
Why Might Difficulties Be Desirable: A New Theory of Disuse
We tried to provide a rationale for why certain difficulties can be desirable in a theoretical framework
we titled “A New Theory of Disuse” (Bjork & Bjork, 1992), which we developed as a contribution to a
Festschrift honoring William K. Estes. In our framework we tried to capture what we had come to think
of as some “important peculiarities” (Bjork & Bjork, 1992, p. 39) of the human learning and memory
system. Although at times we have regretted having given our framework the title, “A New Theory of
Disuse (NTD)”—rather than a title that would more clearly evoke its relation to learning and forgetting—
we adopted that title by reference to Thorndike’s (1914) original “law of disuse,” which stated that
learned habits without continued practice fade or decay from memory as time passes. Thorndike’s
decay idea was thoroughly discredited by McGeoch (1932) and others who demonstrated decades ago
that memories become inaccessible with disuse, but remain in one’s memory and can be relearned at an
accelerated rate. We wanted to give Thorndike credit, though, for emphasizing that use is critical for
keeping memories accessible.
In the NTD Framework, we make the assumption that an item in memory can be characterized by two
strengths—a storage strength (e.g., how well learned and interconnected that item is with related items
in our memory) and a retrieval strength (e.g., the current ease of access we have to that item given the
cues presently available). Such a distinction is by no means new with us. It corresponds, for example, to
Estes’ (1955) distinction between response strength and habit strength and to Hull’s (1943) distinction
between momentary reaction potential and habit strength. In a more general way, the distinction also
corresponds to the time-honored distinction between performance, which can be directly measured,
and learning, which can only be inferred and measured at a delay (see Soderstrom & Bjork, 2015, for a
review).
What is new about the NTD are our assumptions about storage strength, retrieval strength, and how
they interact. The theory assumes that storage strength of a given memory, such as the name of a friend
or of an episode we have experienced, is never lost once accumulated, but that the retrieval strength of
a given memory is dependent on recency and the association of that memory to current environmental
and other cues. In general, retrieval strength will decline as time passes and environmental and other
cues change, but we have all had the experience that a name we are unable to recall at one point in
time comes to us without apparent effort at a later time, often when we are in a different context. If
something is well learned and frequently accessed, such as a street address where one has lived for a
number of years, that address has both high storage strength and high retrieval strength, whereas the
street address of a store one patronized for the first time on a given morning might have high retrieval
strength for much of that day, but would have minimal storage strength. The name of our best friend in
elementary school, on the other hand, would be an example of an item in memory that remains at high
storage strength but now has minimal retrieval strength, assuming one has been out of contact with
that individual in recent years. And there are, of course, frequent examples of information in our
20
memories that is low in both storage strength and retrieval strength, such as information one heard in a
lecture, but did not understand well, and then cannot recall later in the day.
Importantly, there is an asymmetry in how storage strength and retrieval strength are assumed to
interact. The NTD theory assumes that the higher the current level of storage strength of some item in
memory, the larger the gain in retrieval strength that will result from a restudy or a successful retrieval
of that item of information, but, in contrast, and somewhat unintuitively, that the higher the current
level of retrieval strength of some item in memory, the smaller the gain in storage strength that will
result from restudying or retrieving that item. Thus, forgetting (loss of retrieval strength) can enhance
learning (the gain in storage strength), which is why, in the theory, manipulations such as spacing and
variation, which reduce retrieval strength, can enhance learning, as measured by performance at a
delay—or, said differently, why forgetting can be the “friend of learning” (e.g., Bjork, 2015; Bjork &
Bjork, 2019). These dynamics between storage strength and retrieval strength are nicely illustrated at
The Learning Scientists, which was posted by Veronica Yan, who is now on the faculty of the University
of Texas (also see Yan et al., this volume).
A final assumption of the theory is that an act of retrieving to-be-learned information or a skill is a more
powerful event in terms of its effects on both retrieval strength and storage strength than is the act of
restudying such information. In real-world instruction, therefore, an important key for teachers is to
create conditions or provide cues that will foster a given student’s being able to generate to-be-learned
knowledge or skills, versus simply presenting that information or demonstrating that skill again to the
student.
Finally, to take advantage of this relationship between storage strength and retrieval strength in either
one’s own self-regulated learning or as a way to aid the learning of others, we need to remember that it
is important to schedule our study sessions or our teaching so that we do not immediately return to
covering the same information again. Revisiting important to-be-learned facts and concepts can be very
valuable—even essential—for comprehension and retention, but we should try to schedule such
revisiting for a time when retrieval strength has been reduced somewhat. We might manage to do so,
for example, by covering some related topic before returning to the topic in question. Such a strategy
would also address one issue that teachers—particularly those teaching in elementary and high-school
classrooms—report, which is not having enough instruction time to insert rest breaks between their
presentations on any one topic, given how many topics they are required to cover in the school year.
Spacing via interleaving of topics offers an efficient way of introducing spacing into one’s teaching
without a loss of classroom time.
Obstacles to Introducing Desirable Difficulties

At this point, it seems a good idea for us to point out some metamemory issues to consider when
attempting to use desirable difficulties, either in one’s own learning or to facilitate others’ learning.
Meta-Memory Considerations
A general characteristic of desirable difficulties (such as the spacing or interleaving of study or practice
trials) is that they present challenges (i.e., difficulties) for the learner, and hence can even appear to be
slowing the rate at which learning is occurring. In contrast, their opposites (such as massing or blocking
of study or practice trials) often make performance improve rapidly and can appear to be enhancing
learning.
Thus, as either learners or teachers, we are vulnerable to being misled as to whether we or our students
are actually learning effectively, and, indeed, we can easily be misled into thinking that these latter
types of conditions, such as massing or blocking, are actually better for learning. Such dynamics
21
probably play a major role in why students often report that their most preferred and frequently used
types of study activity include activities such as rereading chapters (e.g., Bjork et al., 2013), typically
right away after an initial reading. Such activities can provide a sense of familiarity or perceptual fluency
that we can interpret as reflecting understanding or comprehension and, thus, produce in us what we
have sometimes called an “illusion of comprehension” (Bjork, 1999; Jacoby et al., 1994).
Similarly, when information comes readily to mind, which frequently is the case in blocked practice, or
with no contextual variation in a repeated study or practice setting, we can be led to believe that such
immediate access reflects real learning when, in fact, such access is likely to be the product of cues that
continue to be present in the unchanging study situation, but that are unlikely to be present at a later
time, such as on an exam. As both learners and teachers, we need to be suspicious of conditions of
learning, such as massing and blocking, that frequently make performance improve rapidly, but then
typically fail to support long-term retention and transfer. To the extent that we interpret current
performance as a valid measure of learning, we become susceptible both to mis-judging whether
learning has or has not occurred and to preferring poorer conditions of learning over better conditions
of learning.
Desirable Versus Undesirable Difficulties
Similar to our sometimes regretting giving the title “New Theory of Disuse” to our theoretical
framework, we have sometimes regretted adopting the term desirable difficulty (originally coined in
Bjork, 1994a, 1994b) to designate those conditions of study and practice—such as spacing, interleaving,
contextual variation, and retrieval practice—that pose challenges but often enhance learning and
transfer. “Desirable difficulties” has a nice alliteration, but it is a phrase that can be problematic. First,
from a student’s perspective, introducing difficulties is not very desirable. Second, from an instructor’s
standpoint, it may lead to the assumption that simply introducing difficulty into one’s lectures or lessons
is a way to achieve better learning by your students. We have had to emphasize that the word desirable
is important, and that many difficulties are undesirable during instruction and forever after. Desirable
difficulties, versus the array of undesirable difficulties, are desirable because they trigger encoding and
retrieval processes that support learning, comprehension, and transfer. If, however, the learner does
not have the background knowledge or skills to respond to them successfully, they become undesirable
difficulties.
For this reason, it is necessary to consider what level of difficulty is appropriate in order for that level to
enhance a given student’s learning, and the appropriate level that is optimal may vary considerably
based on a student’s background and prior level of knowledge. To illustrate, while it is typically desirable
to have learners generate a skill or some knowledge from memory, rather than simply showing them
that skill or presenting that knowledge to them, a given learner needs to be equipped via prior learning
to succeed at the generation task—or at least succeed in activating relevant aspects of the necessary
skill or knowledge—for the act of generating to then potentiate their subsequent practice or study (e.g.,
Little & Bjork, 2016; Richland, Kornell, & Kao, 2009).
Opportunities, Amid the Obstacles, to Introduce Desirable Difficulties

In a recent forum of papers published in the Journal of Applied Research in Memory and Cognition
(JARMAC), the Editor, Paula Hertel, asked a series of researchers to comment on the “everyday
challenges to the practice of desirable difficulties” (Hertel, 2020, p. 425) and asked the two of us to
submit a commentary on those papers (Bjork & Bjork, 2020). These papers are both very informative
regarding the benefits that can be gained by introducing desirable difficulties, but also in their
descriptions of challenges that often need to be overcome, including resistance from the learners—and
22
sometimes from teachers or trainers themselves—to having such challenges introduced into instruction
and/or training. In what follows we report on some of the opportunities and obstacles the contributors
to this forum identified.
Law School Instruction
In his contribution, Schulze (2020) describes both a very successful effort to upgrade Florida
International University’s (FIU) law-school instruction by incorporating findings from the science of
learning and, in particular, desirable difficulties into the school’s instructional activities, but it also
highlights a commonly reported challenge encountered in such enterprises. Most important, Shulze and
his FIU colleagues were able to achieve amazing and inspiring results from their efforts: By revamping
FIU’s law-school instruction, they were able to increase the rate of FIU students passing the bar exam
from being about fifth among Florida law schools to being first in “seven of the last ten exams.” That
achievement is truly encouraging because it also exceeds any prediction that would be made based on
the entering credentials of FIU students, as measured by their LSAT scores and other metrics, as
compared to other Florida law schools. It may be an unwarranted and overly optimistic generalization
on our part, but this finding suggests to us that across education programs more broadly, optimizing
instructional practices by introducing desirable difficulties may act as a kind of leveler—that is, allowing
students to succeed regardless of what they have been able to accomplish in their prior academic
careers.
Second, regarding the challenges described by Shulze (2020) in this endeavor, we found his comments
on the students’ attitude with respect to using testing or retrieval practice as a helpful learning tool to
be especially intriguing. For the two of us and many other learning science researchers, testing (or
retrieval practice) is seen as having multiple pedagogical advantages from both a memory and a meta-
memory standpoint: namely, the retrieval processes triggered by testing can enhance later retrieval;
reduce the likelihood of recalling competing incorrect information; and provide feedback to learners as
to what has and has not been learned (See Roediger & Karpicke, 2006, for a review).
According to Shulze (2020), however, such learning advantages of testing have not previously been
appreciated in the milieu of law-school instruction. Rather, he reveals that “our obsession with
summative assessment leads students to believe that testing, or retrieval practice, is meant in all cases
only to assess the student’s ability, knowledge, and aptitude,” and, as a consequence, “students cannot
fathom the idea of self-testing unless they are fully prepared for the real exam; and no law student has
ever felt fully prepared for an exam” (Schilze, 2020, p. 428).
Mathematics Instruction
In another enlightening paper, Rohrer and Hartwig (2020; see also Rohrer & Hartwig, this volume)
reflect upon some of the challenges to introducing manipulations that have been shown to enhance
learning in the laboratory setting into actual classroom teaching—even one as well established as the
finding that long-term retention of skills and knowledge benefit from the spacing of instruction or
practice. Rohrer and Hartwig focus on the extremely important project conducted by Rohrer (2012) in
which he examined whether mathematics education in the real world of schools would profit from using
an interleaved schedule for the teaching of when and how to use certain techniques (such as the
Pythagorean Theorem) for solving problems in mathematics. He found strong support for the benefits of
using interleaving or spacing in such instruction, which seems especially interesting given that most
workbooks, as well as classroom exercises, typically involve blocked, not interleaved, practice. As Rohrer
and Hartwig (2020) point out, however, even with such strong examples that learning in the classroom
could be more effective with the introduction of such desirable difficulties into instruction, it is by no
means a given that research-based changes will get into classroom instruction. Indeed, they lament
23
that—given the prior beliefs that students and teachers may hold—"too often, the classroom is where
promising interventions go to die” (Rohrer & Hartwig, 2020, p. 433; cf. Rowe & Hattie, this volume).
Rohrer and Hartwig (2020) suggested that one factor in this resistance to change by both students and
instructors may stem from the combination of lower performance and greater experienced difficulty
when spacing is introduced. Another contributing factor may be that learners tend not to believe spaced
or interleaved practice is effective because, as we described earlier, massing and blocking can provide
an “illusion of mastery” that is difficult to overcome. Indeed, in our own laboratory, we have discovered
that participants—even after having performed significantly better on a category learning task under
conditions of spaced and/or interleaved practice than under blocked practice—still reported believing
that blocking (versus interleaving) had been better for their learning, and that they would not choose to
engage in interleaving practice in their own study efforts (Yan et al., 2016). Even when, for example, in
the asking of such questions they are warned about the likelihood of being fooled by the apparent ease
of blocked practice and that, in fact, 90% of people actually learn better under conditions of interleaved
practice than under conditions of blocked practice, most participants still held to their belief that they
had learned better with blocking. That is, they believe themselves to be one of the 10% of individuals
who learn better under blocking than under interleaving.
Introducing a “Study Smart” Program to Make Students More Effective Learners
Analogous to the observations of Rohrer and Hartwig (2020), Biwer et al. (2020) discussed the
challenges they have confronted in their admirable effort to implement a “study smart” program to help
students become more effective learners (see also Biwer & de Bruin, this volume). Among the obstacles
they encountered is that many students have already formed “naïve theories” about learning strategies
that need to be “debunked.” Additionally, for students already succeeding in their academic careers,
they are hesitant to change to new study strategies, especially when such strategies seem inconsistent
with the way in which they have previously been taught to study.
We, too, have encountered this type of hesitancy from highly accomplished undergraduates at UCLA.
Each term, we speak to a large group of undergraduates who serve as Learning Assistants (LAs) in many
of the lower-division courses required for majors in the Division of Life Sciences. They are juniors and
seniors who did well in the broad range of these types of required classes and are now working with the
freshman and sophomore students currently enrolled in such classes. As part of holding such positions,
they attend a weekly pedagogy seminar, and it is in that context that we speak to them about desirable
difficulties. They have also been asked to read the paper we wrote for undergraduates entitled Making
Things Hard on Yourself, but in a Good Way: Creating Desirable Difficulties to Enhance Learning (Bjork &
Bjork 2011), and they always seem very receptive to the message in that paper, as well as to the reasons
we espouse to them during our presentation about why using desirable difficulties can enhance
learning. However, even many of these accomplished students occasionally convey to us that, while they
believe using desirable difficulties in their studies would be beneficial, they are fearful of trying to do so
at this point in their undergraduate careers. That is, because they have been successful so far in
obtaining good or excellent grades, they are afraid to make changes with respect to how they currently
study, which for the most part amounts to spending much of their time engaged in massed or blocked
study, including cramming immediately before taking exams.
These LAs, however, seem more than willing to urge their students to try to use desirable difficulties in
their own study practices, and they often report to us that they believe some of their students actually
do so. We thus wonder if more students could be introduced to the value of using desirable difficulties
as part of their study strategies earlier in their academic career, we would have fewer students at higher
levels of education struggling to succeed and often dropping out of the demanding majors.
24
Expectations and Motivations
In her contribution, Finn (2020) makes a similar important observation in terms of the need to address
motivational factors in any attempt to bring desirable difficulties into real-world educational settings.
She points out that students’ memories of their past achievements in academics—or perhaps the lack
thereof—can shape their expectations and goals which, in turn, can heavily influence both students’
effort to learn and their selection of learning procedures. In truth, as we mentioned earlier, the two of
us have been guilty of ignoring such factors by thinking that simply telling students and teachers about
relevant research findings will be enough for them to incorporate desirable difficulties into their study
and instructional activities. Finn (2020) cogently argues that if our goal is to have students replace less
effective study activities—particularly ones that have probably become habitual and been encouraged
by previous teachers or trainers—with more effective activities, then issues of motivation must be
addressed.
Special Considerations in Children’s Learning
Knabe and Vlach (2020) discuss why it is critical—when trying to introduce desirable difficulties, such as
spacing, into children’s educational programs—to take into account the potential effects of the many
individual differences among children in the early years of their living and learning. Such developmental
differences clearly present K-12 instructors with a very difficult task of deciding when the learning of a
given child or age group of children would or would not profit from spaced study schedules. These
differences indicate a critical need for careful research on the limitations and/or boundary conditions of
the spacing effect in early education, and, as Knabe and Vlach (2020) point out, that research is mostly
missing.
Desirable Difficulties in Learning Motor Skills
For anyone considering introducing desirable difficulties into the teaching of motor skills or even into
their own learning of a motor skill, the contribution by Hodges and Lohse (2020) offers a thought-
provoking analysis of the challenges in doing so. The authors also provide an insightful discussion of the
need to identify desirable difficulties prospectively in this domain, something we have not always been
careful to do. They argue that such difficulties must be task relevant; novel (that is, not something the
learner is already doing); and potentially solvable.
This final criterion, which we have pointed out above as being critical, is one that we often find
ourselves needing to emphasize when speaking with educators and creators of instructional materials:
specifically, that for difficulties to be desirable—that is, to promote learning—they must present
challenges to the learner but not ones so challenging that they cannot be overcome by a given
individual, given that individual’s prior relevant knowledge or lack thereof. Game makers are experts at
this task, making each subsequent level of a game more difficult—but not by too much—in, say, a
computer game. As instructors, we need to become more skilled at this endeavor ourselves. We
consider the use of “adaptive” learning schedules where levels of difficulty are tailored to an individual’s
past successes to represent good instantiations of this critical feature of desirable difficulties.
Also, of value in Hodge and Lohse’s (2020) contribution is their discussion of the role of making errors,
often considered something to be avoided, in optimizing motor skill learning. Indeed, as part of
incorporating the desirable difficulty framework, we consider its advice to learners to look upon their
errors as opportunities for enhanced learning to be a critical feature.
25
What Can Be Done to Get Learners to Embrace Desirable Difficulties?
Finally, Zepeda et al.’s (2020) contribution focuses on a challenge that we discussed in our introductory
remarks and that also appears explicitly or implicitly in most of the contributions to this JARMAC forum:
namely, how can we get learners to embrace and employ desirable difficulties in their own study
strategies? As we have mentioned several times, we thought initially that simply showing the benefits of
incorporating desirable difficulties would be enough for students and others to introduce such strategies
into their own learning endeavors. Instead, convincing learners to adopt desirable difficulties can be a
major challenge (cf. Manalo et al., this volume). Zepeda et al. (2020) review relevant research examining
the role of motivation in learning and provide brief descriptions of five approaches that may provide
insights into how learners can be motivated to introduce desirable difficulties into the management of
their own real-world learning. Their essay makes explicit a challenge that comes through in the other
contributions: We need to understand the motivational factors that make learners willing or unwilling to
change habits and embrace difficulties (cf. Rowe & Hattie, this volume).
The Important Role of Public-Audience Books and Websites

Perhaps the most important development in upgrading instruction, via incorporating desirable
difficulties and by other means, is the existence of teachers who are not only introducing research-
based innovations in their own classrooms, but who also are writing books and creating websites to
report their efforts and findings, often in a more interesting and understandable way than do articles in
academic journals. William Emeny, for example, who teaches mathematics at Wyvern College, a
secondary school enrolling 11-16-year-old students in Hampshire, England, and who was awarded a
“secondary teacher of the year” prize in a ceremony at England’s parliament, is a good example. Among
his contributions, he has created a “numeracy ninjas” program for students that draws on the power of
principles such as retrieval practice and interleaving to enhance students’ long-term retention of
mathematics procedures and concepts, which he reports on in his web site,
Greatmathsteachingideas.com. Craig Barton, also a mathematics teacher in England, has written an
important book, How I wish I’d Taught Maths: Lessons learned from research, conversations with
experts, and 12 years of mistakes (Barton, 2018), and he hosts a popular podcast for teachers, including
a session with the two of us. Other examples of teachers drawing on desirable difficulties, especially
retrieval practice, include Kate Jones, a history teacher in Abu Dhabi who has written two well-received
books (Jones, 2019; Jones, 2021) on retrieval practice. She, too, hosted a podcast with the two of us. In
Alabama, Blake Harvard, who teaches the social sciences in Madison, has created an impressive website
titled The Effortful Educator with a goal to applying cognitive psychology in the classroom. Under his
supervision, the Advanced Placement Psychology enrollment at his school has grown from 24 students
in 2012 to over 250 students in 2020.
A book by Benedict Cary (2014), a New York Times reporter, on “How We Learn” and one by Brown et al.
(2014) entitled “Make it Stick: The Science of Successful Learning,” which is a best seller in its category
on Amazon.com, are bringing research results from the science of learning to the broader public in an
accessible way. Similarly, Pooja Agarwal, who maintains a website, Retrieval Practice, focuses on the
applications of retrieval practice and has co-authored Powerful Teaching: Unleash the Science of
Learning (Agarwal & Bain, 2019). Sumeracki et al. (this volume) created and continually develop a
website, The Learning Scientists, that provides a broad array of resources on applications of the science
of learning in educational settings.
26
Concluding Comments
Happily, the books and websites we have mentioned are only a sample, and by the time this book is
published there will be other attempts to make the science of learning more accessible to students,
teachers, and parents. What is perhaps most important is to bring research on learning to students
earlier in their schooling. As we mentioned, even highly motivated students at the university level are
hesitant to incorporate desirable difficulties into their everyday learning activities because what they
have been doing has got them to where they are—and, in general, habits, good or bad, are hard to
break.
Having said that, it is important to emphasize that learning how to learn needs to be a lifetime pursuit.
As we (Bjork & Bjork, 2014) have emphasized elsewhere, “In a world that is ever more complex and
rapidly changing, and in which learning on one’s own is becoming ever more important, learning how to
learn is the ultimate survival tool.” (p. 11)
References
Agarwal, P. K., & Bain, P. M. (2019). Powerful teaching: Unleash the science of learning.
Jossey-Bass.
Anderson, M.C., Bjork, R.A., & Bjork, E. L. (1994). Remembering can cause forgetting: Retrieval dynamics
in long-term memory. Journal of Experimental Psychology: Learning, Memory, and Cognition,
20(5), 1063-1087. https://doi.org/10.1037//0278-7393.20.5.1063
Barton, C. (2018). How I wish I’d taught maths: Lessons learned from research, conversations with
experts, and 12 Years of mistakes. John Catt Educational Limited.
Biwer, F., de Bruin, A. B. H., Schreurs, S., & oude Egbrink, M. G. A. (2020). Future steps in teaching
desirably difficult learning strategies: Reflections from the Study Smart program. Journal of
Applied Research on Learning and Memory, 9(4), 439-46.
Bjork, R. A. (1975). Retrieval as a memory modifier. In R. Solso (Ed.), Information processing and
cognition: The Loyola Symposium (pp. 123-144). Lawrence Erlbaum Associates.
Bjork, R.A. (1994a). Memory and metamemory considerations in the training of human beings. In J.
Metcalfe & A. Shimamura (Eds.), Metacognition: Knowing about knowing (pp.185-205). MIT
Press.
Bjork, R.A. (1994b). Institutional impediments to effective training. In D. Druckman & R.A.Bjork (Eds.),
Learning, remembering, believing: Enhancing human performance (pp. 295-306). National
Academy Press.
Bjork, R. A. (1999). Assessing our own competence: Heuristics and illusions. In D. Gopher and A. Koriat
(Eds.), Attention and performance XVII. Cognitive regulation of performance: Interaction of
theory and application (pp. 435-459). MIT Press.
Bjork, R. A. (2015). Forgetting as a friend of learning. In D. S. Lindsay, C. M. Kelley, A. P. Yonelinas, & H. L.
Roediger, III (Eds.), Remembering: Attributions, processes, and control in human memory: Papers
in honour of Larry L. Jacoby (pp. 15-28). Psychology Press.
Bjork, E. L., & Bjork, R. A. (2011). Making things hard on yourself, but in a good way: Creating desirable
difficulties to enhance learning. In M. A. Gernsbacher, R. W. Pew, & J. R. Pomerantz (Eds.),
27
Psychology and the real world: Essays illustrating fundamental contributions to society (pp. 56-
64). Worth Publishers.
difficulties to enhance learning. In M. A. Gernsbacher & J. Pomerantz (Eds.), Psychology and the
real world: Essays illustrating fundamental contributions to society (2nd edition) (pp. 59-68).
Worth Publishers.
Bjork, R. A., & Bjork, E. L. (1992). A new theory of disuse and an old theory of stimulus fluctuation. In A.
Healy, S. Kosslyn, & R. Shiffrin (Eds.), From learning processes to cognitive processes: Essays
in honor of William K. Estes (Vol. 2, pp. 35-67). Erlbaum.
Bjork, R. A., & Bjork, E. L. (2019). Forgetting as a friend of learning: Implications for teaching and self-
regulated learning. Advances in Physiology Education, 43(2), 164-167.
https://doi.org/10.1152/advan.00001.2019
Bjork, R. A., & Bjork, E. L. (2020). Desirable difficulties in theory and practice. Journal of Applied Research
in Memory and Cognition, 9(4), 475-479. https://doi.org/10.1016/j.jarmac.2020.09.003
Bjork, R. A., Dunlosky, J., & Kornell, N. (2013). Self-regulated learning: Beliefs, techniques, and illusions.
Annual Review of Psychology, 64, 417-444. https://doi.org/10.1146/annurev-psych-113011-
143823
Brown, P. C., Roediger, H. L., & McDaniel, M. A. (2014). Make it stick: The science of successful learning.
Belknap Press.
Carey, B. (2014). How we learn: The surprising truth about when, where, and why it happens. Random
House.
Estes, W. K. (1955). Statistical theory of distributional phenomena in learning. Psychological Review,
62(5), 369-377. https://doi.org/10.1037/h0046888
Finn, B. (2020). Exploring interactions between motivation and cognition to better shape self-regulated
learning. Journal of Applied Research on Learning and Memory, 9(4), 461-467.
Hertel, P. (2020). Everyday challenges to the practice of desirable difficulties: Introduction to the forum.
Journal of Applied Research in Memory and Cognition, 9(4), 425-427.
Hull, C. L. (1943). The principles of behavior. Appleton-Century-Crofts.
Hodges, N., & Lohse, K. R. (2020). Difficulty is a real challenge: A perspective on the role of cognitive
effort in motor skill learning. Journal of Applied Research on Learning and Memory, 9(4), 445-
Jacoby, L. L., Bjork, R. A., & Kelley, C. M. (1994). Illusions of comprehension, competence, and
remembering. In D. Druckman & R. A. Bjork (Eds.), Learning, remembering, believing: Enhancing
human performance (pp.57-80). National Academy Press.
Jones, K. (2019). Retrieval practice: Research & resources for every classroom. John Catt International
Ltd.
Jones, K. (2021). Retrieval practice 2: Implementing, embedding, & reflecting. John Catt Educational Ltd.
28
Knabe, M. L., & Vlach, H. A. (2020). When are difficulties desirable for children? First steps toward a
development and individual differences account of the spacing effect. Journal of Applied
Research on Learning and Memory, 9(4), 447-454. https://doi.org/10.1016/j.jarmac.2020.07.007
Koriat, A., & Bjork, R. A. (2005). Illusions of competence in monitoring one’s knowledge during study.
Journal of Experimental Psychology: Learning, Memory, and Cognition, 31(2), 187-194.
https://doi.org/10.1037/0278-7393.31.2.187
Kornell, N., & Bjork, R. A. (2008). Learning concepts and categories: Is spacing the “enemy of induction”?
Psychological Science, 19(6), 585-592. https://doi.org/10.1111/j.1467-9280.2008.02127.x
Landauer, T. K., & Bjork, R. A. (1978). Optimal rehearsal patterns and name learning. In M. M.
Gruneberg, P. E. Morris, & R. N. Sykes (Eds.), Practical aspects of memory (pp. 625-632).
Academic Press.
Little, J. L. & Bjork, E. L. (2016). Multiple-choice pretesting potentiates learning of related information.
Memory & Cognition, 44(7), 1085-1101. https://doi.org/10.3758/s13421-016-0621-z
McGeoch, J. A. (1932). Forgetting and the law of disuse. Psychological Review, 39, 352-370.
https://doi.org/10.1037/h0069819
Richland, L. E., Kornell, N., & Kao, L. S. (2009). The pretesting effect: Do unsuccessful retrieval attempts
enhance learning? Journal of Experimental Psychology: Applied, 15(3), 243-257.
https://doi.org/10.1037/a0016496
Roediger, H. L, III, & Karpicke, J. F. (2006) The power of testing memory: Basic research and implications
for educational practice. Perspectives in Psychological Science, 1(3), 181-210.
Rohrer, D. (2012). Interleaving helps students distinguish among similar concepts. Educational
Psychology Review, 24, 355-367. https://doi.org/10.1007/s10648-012-9201-3
Rohrer, D., & Hartwig, M. K. (2020). Unanswered questions about spaced and interleaved mathematics
practice. Journal of Applied Research on Learning and Memory, 9(4), 433-438.
Schmidt, R. A., & Bjork, R. A. (1992). New conceptualizations of practice: Common principles in three
paradigms suggest new concepts for training. Psychological Science, 3(4), 207-217.
https://doi.org/10.1111/j.1467-9280.1992.tb00029.x
Schulze, L. N. (2020). Legal education’s difficulty with “desirable difficulties” and its impact on student
success and bar passage rates. Journal of Applied Research on Learning and Memory, 9(4), 428-
432. https://doi.org/10.1037/h0101859
Soderstrom, N. C., & Bjork, R. A. (2015). Learning versus performance: An integrative review.
Perspectives on Psychological Science, 10(2), 176-199.
Thorndike, E. L. (1914). The Psychology of Learning. Teachers College.
Yan, V. X., Bjork, E. L., & Bjork, R. A. (2016). On the difficulty of mending metacognitive illusions: A priori
theories, fluency effects, and misattributions of the interleaving benefit. Journal of Experimental
Psychology: General, 145(7), 918-933. https://doi.org/10.1037/xge0000177
29
Zepeda, C. D., Martin, R. S., & Butler, A. C. (2020). Motivational strategies to engage learners in desirable
difficulties. Journal of Applied Research on Learning and Memory, 9(4), 464-470.
30
In Their Own Words The Culture of Teaching We Have Versus the
Culture of Teaching We Need
Stephen L. Chew
Editors
Samford University
Lauren L. Kordonowy
Victor A. Benassi
We teach in strange times. Although there is much talk about “active learning,” the dominant mode of
teaching is still lecture (Cerbin, 2018; Stains et al., 2018). Some researchers have proposed a dichotomy
between active learning and lecture (Freeman et al., 2014), but Cerbin (2018) pointed out that the two
are compatible. Lecture can promote active learning by incorporating research-based teaching
strategies. Even so, Stains et al. (2018) conducted extensive observations of STEM faculty and found that
most used didactic lecture where students passively listened, even when technology that could facilitate
active learning was available. Thus, the current state of teaching is complex, contradictory, and
disheartening. We have the research-based knowledge to teach better, but we often fail to use it, and
progress can be stymied by false dichotomies and buzzwords. In this chapter, I describe the current
state of teaching, or the culture of teaching we have, and then outline the culture of teaching that I
believe we need.
The Culture of Teaching We Have

College teaching is mired in a state of fad-driven mediocrity despite there being well-established
cognitive principles that can improve student learning (Chew & Cerbin, 2017; Chew et al., 2018). On the
positive side, two different movements have made substantial progress in our understanding of
teaching and learning in the last 20 years. Unfortunately, neither has had much impact on mainstream
teaching (Willingham & Daniel, 2021; Rowe & Hattie, this volume). We have the knowledge to improve
teaching but have yet to change existing norms regarding what constitutes acceptable, competent
teaching.
Teaching currently is ‘lost in a buzzword wasteland.’ That is, it is driven by fads (Chew & Cerbin, 2017).
Teaching fads promise to transform education (e.g., programmed instruction, MOOC’s, and clicker
questions). They follow a similar pattern: initial excitement with reports of strikingly positive findings,
followed by the mixed or negative results, and then descent into obscurity. Then, the field moves on to
the next fad. Teaching changes, but it does not progress. There is little or no accumulation of knowledge
within teaching; we simply move from one fad or buzzword to the next.
This sad state of affairs should cause alarm because, according to virtually any college or university
mission statement, teaching and learning are a top priority. On the contrary, many faculty and
administrators seem pretty content with the way things are (Chew et al., 2018). They provide little
professional development in teaching for their faculty, nor do they incentivize effective teaching in
terms of promotion and advancement. Graduate programs often do not train their students in how to
teach. Many campuses have teaching and learning centers, but these centers often have limited campus
influence. Teaching has advanced, but those advancements have yet to manifest a comprehensive
impact on teaching practice.
31
Information-Driven and Learner-Centered Approaches to Teaching
Understanding the state of teaching requires understanding the distinction between information-driven
and learner-centered teaching (Chew, 2015). In the more traditional information-driven approach, the
goal of teaching is the transmission of information. The role of the teacher is to curate, translate, and
communicate information. In other words, the teacher selects the information to be covered, ensures
that the information is accurate and up-to-date, and then conveys the information in a clear, organized,
and understandable way. Teachers design activities and use technology to help students understand
concepts. The responsibility for learning, however, resides primarily with the student. Nuthall (2007)
noted that the information-driven approach, while intuitively appealing, often fails to yield student
learning.
The information-driven approach underestimates the influence that teachers can have on what and how
well students learn, and it ignores important student variables that also influence learning (Hattie,
2008). For example, the greatest predictor of learning is the prior knowledge of the student (Ambrose &
Lovett, 2014). Greater prior knowledge makes learning new information easier. Thus, students with
varying levels of prior knowledge process the same teacher presentation in different ways according to
what they already know. The Information-driven view is not wrong, but it is incomplete.
The alternative is a learner-centered approach. Weimer (2002) described this perspective this way:
Being learner-centered focuses attention squarely on learning: what the student is
learning, how the student is learning, the conditions under which the student is
learning, whether the student is retaining and applying the learning, and how current
learning positions the student for future learning (p. xvi).
In the learner-centered view, the goal of teaching is creating a schematic understanding of concepts
among students. The success of teaching is measured in student learning, not the knowledge of the
teacher nor the clarity of the learning activities employed. If students do not learn, then the teaching
has not been successful. In learner-centered teaching, teachers and students share responsibility for
student learning. The teacher designs and creates an environment which promotes learning. Students
then engage fully in the class to create a long-term, schematic understanding of the material.
In the information-driven view, effective teaching requires disciplinary knowledge and employing a
suitable pedagogy. In the learner-centered view, successful teaching requires expertise beyond
knowledge of one’s discipline (Shulman, 1986), such as knowledge of common student misconceptions,
the concepts students often have difficulty understanding, and the principles of how people learn.
Teaching is a process of constant assessment and adaptation according to the needs of students
(Nuthall, 2007).
Others have made a similar distinction between teaching approaches with different terms (e.g., Barr &
Tagg, 1995). Weimer (2002), for example, used the term teacher-centered instead of information-driven,
to highlight the emphasis on teacher knowledge and actions. I prefer the term information-driven to be
clear about the focus on information transmission and to remove any implication that a focus on
teacher knowledge is misguided. The terms student-centered and learning-centered are often used
instead of learner-centered. Weimer (2002) argued that the term student-centered put the emphasis on
the desires of the student and led to a consumer-oriented approach to teaching that prioritizes student
satisfaction over learning. Learning-centered eliminates the student altogether, which is inconsistent
with the student sharing responsibility for learning.
32
Learning Science and the Scholarship of Teaching and Learning
The two research approaches that have made progress on teaching and learning are learning science
and the scholarship of teaching and learning (SoTL) They both adopt a learner-centered approach
(Daniel & Chew, 2013), but diverge from there (cf. Gurung, this volume).
Learning science is an interdisciplinary field that applies the principles and findings from cognitive
science, among other fields, to education. Learning science research often consists of highly controlled,
relatively short-term studies done either in the lab or classroom. For example, students might be
randomly assigned to different groups. Each group studies a brief passage using different study
strategies and their learning is assessed both immediately and after a week. Learning science excels at
isolating specific principles related to teaching and learning, such as spaced practice and interleaving
(e.g., Dunlosky et al., 2013; Weinstein et al., 2019).
The weakness of learning science is that it can fail to consider the complexity of the teaching context
(Daniel & Chew, 2013; Rowe & Hattie, this volume). Controlled research methods are fundamentally at
odds with the context in which teachers work. Researchers control for individual differences, focusing
on average increases in learning and replicable findings. Teachers are equally focused on both group
behavior and individual differences. They figure out ways to help the full range of students. The factors
that influence teaching vary from class to class and day to day. A well-established learning principle from
the laboratory may not fully translate to an uncontrolled classroom context (Daniel & Poole, 2009; Perry
et al., 2021). For example, retrieval practice and spacing are all well established as robust learning
strategies in laboratory and classroom-based studies, but a recent meta-analysis (Yang et al., 2021; Yang
et al., this volume) found that the testing effect is modulated by several key variables when used in
classroom contexts, such as giving corrective feedback and material overlap. Carpenter et al. (2016)
found that the effectiveness of retrieval practice varied according to individual differences among
students. Bartoszewski and Gurung (2015) found no correlation between distributed practice and exam
score and only a small correlation between retrieval practice and exam score in an actual course context
where students likely used combinations of learning strategies.
Learning science often focuses on one or two learning outcomes, such as retaining information for later
recall and application. Teachers design pedagogy to address multiple aspects of student learning, such
as improving metacognitive awareness, correcting tenacious misconceptions, and improving mental
mindset (Chew & Cerbin, 2021; Koedinger et al., this volume).
The second research approach is the scholarship of teaching and learning (SoTL), in which pedagogical
practice becomes the subject of scholarly inquiry (cf. Gurung, this volume). The idea of SoTL can be
traced to the Boyer model of scholarship, operationalized by Glassick et al. (1997). The emphasis is less
on control than study of actual teaching practice in real educational settings (e.g., Hattie, 2008). A
researcher might have two sections of general psychology and try one way of teaching correlations to
one section and a different method to the second section. Both sections are then given a common exam
and the results compared. STEM fields have developed their own approach, called Discipline-Based
Educational Research (DBER) (e.g., Frey & Lewis, this volume; Docktor & Mestre, this volume).
SoTL examines learning in authentic contexts, which means the results are more generalizable than
highly controlled learning science research, but it also means that SoTL has all the problems of
conducting research in naturalistic settings. A certain intervention may improve learning but can be
unclear as to why and how the intervention worked. For example, Berry and Chew (2008) found that
students who created concept maps scored higher on exams. The improvement may be due to the
concept maps, but it might also be due to other factors, such as the students spending more time
33
studying because they had to construct the map or providing students with a concrete learning strategy
to pursue when they did not have one before.
A second, more fundamental problem with SoTL is that any change that the teacher believes will
promote learning can, because of that belief, lead to a significant improvement in learning. For example,
a teacher who uses clickers for the first time will pay more attention to their teaching than usual,
perhaps revamping lectures for the first time in years. Students notice the teacher’s enthusiasm and
respond to it. The result is improved learning because of the change, but not necessarily because of the
clickers. Hattie (2009) described this placebo problem: “One only needs a pulse and a belief that an
intervention will work, and it likely will” (p. 16).
Learning Science and SoTL are complementary. The strengths of one can compensate for the
weaknesses of the other. Working together, they could make meaningful progress on teaching and
learning. The problem is that there is little dialogue or collaboration between proponents of each
approach (Daniel & Chew, 2013). Each sees the flaws in the other approach. Learning science faults
educators for relying on intuition instead of research and falling victim to fads and myths. Weinstein et
al. (2019) put it this way: “Unfortunately, educational practice does not, for the most part, rely on
research findings. Instead, we tend to rely on our intuitions about how to teach and learn with
detrimental consequences” (p. 3). Educational practice, in the SoTL tradition, faults learning science for
lack of ecological validity and difficulty in translating cognitive principles to educational practice. In a
review of learning science by educational researchers, Perry et al. (2021) expressed this view:
The evidence for the application of cognitive science principles in everyday classroom
conditions (applied cognitive science) is limited, with uncertainties and gaps about the
applicability of specific principles across subjects and age ranges (p. 7).
Principles do not determine specific teaching and learning strategies or approaches to
implementation. Considering how cognitive science principles are implemented in the
classroom is critical (p. 8).
Neither movement, on its own, is sufficient to understand student learning. Learning science has the
methodological and theoretical precision that SoTL lacks, but SoTL has an understanding of the
complexity of the teaching environment that learning science lacks. Sadly, both often show an in-group
bias that undermines fruitful collaboration. Furthermore, neither approach is making real inroads into
the dominant norm of academic, information-driven teaching.
The Appeal of the Information-Driven Approach

If both learning science and SoTL have shown the limitations of the information-driven approach, then
why is it still the dominant norm for competent teaching? There are several possible reasons. In the
information-driven view, little knowledge beyond up-to-date disciplinary expertise is needed to teach.
Colleges and universities need not worry about ongoing professional development in teaching and
graduate programs need not give students much preparation for teaching. In the information-driven
approach, teachers bear no responsibility for student learning. Instead, quality of teaching is judged by
the accuracy of course content and the quality of delivery. If students are doing poorly, teachers can
blame the admissions office, or the poor high school preparation of the students, or the poor work ethic
of the students. With this approach, teachers can rationalize bad student evaluations and complaints to
holding unprepared students to rigorous personal standards. Finally, this view of teaching is consistent
with the idea that research is more complex intellectual work than teaching (Chew et al., 2018). The
appeal of the information-driven approach is that it makes teaching appear easy, absolves faculty from
the responsibility for student learning, and saves institutions from having to provide ongoing training on
34
teaching for faculty and graduate students. To be clear, I know that many teachers who take an
information-driven approach care deeply about their students and learning. I’m asserting that
innovations in teaching that deviate from the accepted norm are often not encouraged, supported, or
rewarded.
In the learner-centered approach, teaching is a complex and challenging skill to master. Whole bodies of
knowledge other than disciplinary expertise must be mastered, such as pedagogical content knowledge
and knowledge of how people learn (Shulman, 1986). Teaching skill is developed over the span of one’s
career, beginning, ideally, in graduate school.
The culture of teaching in which many off us find ourselves, specifically an information-driven approach,
fits the current zeitgeist that prioritizes research over teaching. Teaching is judged by using best
practices and the latest technology rather than how well that pedagogy and technology has been
implemented to address challenges in student learning. The information-driven approach perpetuates
an unnecessarily low standard of teaching performance that is considered acceptable for faculty. This
culture of teaching must change.
The current culture of teaching is at an impasse. Instead of determining how to improve teaching,
teachers seek a magic bullet, the teaching strategy that is best for all subjects and students. Efforts to
improve teaching focus on “best practices” or “high impact practices” (e.g., Schwartz & Miller, 2020) or
lists of principles that teachers should follow (e.g., Rosenshine, 2021). Although useful, this approach
will not advance teaching because it focuses on procedures rather than on what teachers need to know
to make those procedures work.
The Culture of Teaching We Need

Despite my dismal portrait of the current state of teaching, it is important to remember that we have
made substantial progress in discovering how people learn. The question is how to establish
collaboration between learning science and SoTL and to have their findings become more influential in
mainstream teaching. What is needed, I believe, is nothing less than a cultural shift in the assumptions,
values, and norms of teaching.
This shift should be centered on three core beliefs about learning, all of which are supported by
pedagogical research. First, teaching should be learner-centered and based on empirically supported
principles. The foundation of this cultural shift is a transition from an information-driven to a learner-
centered view of teaching. Second, the framework should recognize the contextual nature of learning.
Third, the emphasis should be on what teachers know, that is, their mental model of how students
learn, more than the practices they employ.
Learning is Contextual
Learning is contextual; no single factor determines learning (Jenkins, 1979, McDaniel & Butler, 2011).
Whether a person learns and what they learn depends on the interaction of multiple factors. The
contextual nature of learning means that teaching involves managing a dynamic interaction of multiple
variables (Chew et al., 2009). For example, the way students study for a class depends on their
perceptions of the teacher’s pedagogical approach (Coertjens et al., 2016; Richardson, 2005; Trigwell et
al., 2005), and students tend to use deeper, more effective learning strategies for courses that they see
as more valuable to them. Teachers understand that student learning is a contextual outcome of
multiple factors (Perry et al., 2021), and teachers adapt pedagogy to classes and students (Nuthall,
2007). Researchers, however, typically conduct controlled experiments with one or two independent
35
variables without worrying about how findings will generalize to more complex, natural situations. For
learning research to be useful, it must be translated into a meaningful academic context.
There is no single best teaching method for all topics, teachers, and students. The teaching approach
that works for one teacher or class may not work for another—and no single pedagogy works every
time. Every teaching method will have boundary conditions that define when it will work and when it
will not. What determines whether pedagogies work is how well a teacher can adapt them to their
classroom context. Any teaching method can be implemented inappropriately and be ineffective. Still,
some teaching methods are likely to be more successful than others. We judge the effectiveness of
pedagogy based on its robustness—its effectiveness across many situations. We learn what pedagogy is
most likely to work in our given teaching contexts.
What the Best Teachers Know

The contextual viewpoint shifts the emphasis from the technology and procedures to what the teachers
know about learning. Teachers are not just conveying information; they are creating an educational
context for students in which learning can occur. That context varies from class to class and even from
student to student. Teachers must create multiple learning opportunities to account for individual
differences among students.
All teachers have a mental model of how their students learn, and this model informs the selection,
design, and implementation of pedagogy (Chew & Cerbin, 2021; Willingham, 2017). If they believe that
students learn best if an expert explains concepts clearly, then they will favor more lecture-oriented
teaching styles. If they believe students learn best when they learn from each other, they will favor
more collaborative, group-oriented pedagogies. When a problem arises while teaching, they rely on
their mental model to diagnose the issue and implement a solution. To the extent that a mental model
is complete and correct, the teacher will be effective, but if the model is incomplete or contains
misconceptions, teaching effectiveness will be undermined.
“Best practices” and “high impact” practices assume teaching is theory-free. They focus on procedure
with no concern about the theory of why a pedagogy is supposed to facilitate learning. The assumption
is that pedagogies should work uniformly and automatically no matter how they are implemented, but
any teaching method can be implemented badly. What teachers know about learning is more important
than any teaching method or technology.
The Cognitive Challenges of Effective Teaching

A first step toward a cultural change is to develop a research-based, theoretical framework on how
students learn that bridges the expanse between learning science, SoTL, and teaching practice (Chew &
Cerbin, 2017). A common theoretical framework would guide translation of pedagogical research into
useful forms for teaching practice. Likewise, issues in teaching could be translated into research
questions.
William Cerbin and I (Chew & Cerbin, 2021) created such a theoretical framework that describes nine
cognitive challenges that teachers must solve for student learning to occur. Failing to address any one of
the challenges can undermine learning. Consistent with the contextual nature of learning, these
challenges interact with each other. Addressing one challenge might create a problem in another
challenge or it might ameliorate other challenges. Teachers must successfully address all challenges to
be effective.
36
Here are the nine challenges of The Cognitive Challenge Framework.
1. Mental Mindset: Mental mindset refers to students’ attitudes, beliefs, and expectations about
learning and the academic context, which influences their learning behavior (e.g., Yeager &
Dweck, 2012).
2. Metacognition and Self-Regulation: Students are often poor judges of their own level of
learning, nor are they effective in managing their study behavior to improve their understanding
(e.g., Pintrich, 2004).
3. Fear and Mistrust: Students who believe their teachers want them to succeed will work harder
and persevere longer than students who see their teachers as indifferent or adversarial (e.g.,
Cox, 2009; Yeager et al., 2013).
4. Prior Knowledge: The greater amount of accurate information that students already possess
about a subject, the easier it is for them to learn more about that subject (Ambrose & Lovett,
2014).
5. Misconceptions: Every field has common student misconceptions that are remarkably resistant
to correction (Taylor & Kowalski, 2014).
6. Ineffective Learning Strategies: Students often prefer the least effective study strategies for
long-term learning (e.g., Dunlosky et al., 2013; Ekuni et al., this volume).
7. Transfer of Learning: Students often fail to generalize learning beyond the immediate classroom
context (e.g., Day & Goldstone, 2012).
8. Selective Attention: Students overestimate their ability to learn while multitasking or in the
presence of distractions (e.g., Ravizza et al., 2012).
9. Constraints of Mental Effort and Working Memory: Students can concentrate on and consider
only a limited amount of information at a given time (van Merienboer & Sweller, 2005; Ashman
& Sweller, this volume).
True to the contextual nature of learning, these factors interact with each other. For example, effective
learning strategies often involve desirable difficulty which helps learning, but that also contributes to
cognitive load, which hurts learning. In this model, effective teaching involves managing a nine-way
interaction of factors to optimize student learning (cf. Koedinger et al., this volume). Challenges that are
particularly salient will vary from school to school, class to class, and even day to day. Note that these
are cognitive challenges, and do not include the social, emotional, or developmental factors in learning.
Teachers can use the framework diagnostically to identify and address learning problems that students
might be having. For example, a problem in learning may be due to ineffective learning strategies, lack
of accurate prior knowledge, or inappropriate mental mindset. Teachers can assess which of these
challenges are the actual problem, and then design pedagogy to address them.
Teaching is a constant process of context-specific assessment and adaptation. Solutions for a given
challenge that are effective for a large, public university will likely differ from effective solutions for a
small, selective liberal arts college or a community college. Different teachers within the same
department also might resolve the same challenge in different ways.
An Expanded View of Student Learning

In research, the simplest measure of learning is the ability to recall information after a delay. In
teaching, successful learning goes far beyond simple recall to include conceptual understanding, schema
formation, problem solving, and transfer (Chi & Boucher, this volume; Koedinger et al., this volume).
Conceptual understanding is typically measured by the ability to reason and solve problems using the
37
learned information. Transfer is usually defined as the ability to apply concepts appropriately in novel
situations (Schwartz et al., 2005). The Cognitive Challenge Framework, expanded on above, supports a
broader more integrated conception of successful student learning.
Pedagogical researchers should embrace an integrated, multidimensional view of successful learning.
For example, an accurate sense of metacognitive awareness and the dominance of accurate information
over popular misconceptions are both important components of successful learning. Students should be
aware of what they know and do not know and of what gaps and weaknesses exist in their
understanding. They should be prepared to revise their beliefs with new and better evidence. Successful
learning is not simply a collection of unrelated facts. It is schematic. Knowledge is connected and
organized into a framework that, once activated, allows the student to comprehend a novel situation or
to analyze and reason through a problem (Ahn et al., 1992). A schematic understanding lays the
groundwork for further learning. Successful student learning involves improving students’ attitudes
toward the subject matter. Students may now appreciate the value of knowing certain information
when they saw no value before. They may have increased academic self-efficacy, believing that they
possess the skill to master a topic through their own effort when they did not feel that way before. They
may be less fearful and even enjoy a class on a topic that they had never enjoyed learning before.
The Cognitive Challenge Framework considers all these aspects of learning as integrated components of
successful learning. For example, a researcher may compare two teaching methods. There may be no
difference in the level of recall between the two methods, but one may be better suited for students
who lack accurate prior knowledge while the other increases positive student beliefs about the value of
learning the subject. Even though recall is the same, each method addresses different aspects of
successful student learning.
An Expanded View of Successful Teaching

Successful teachers find ways to address each of the challenges by creating a course environment that
supports successful student learning. Students come into a class with a wide variety of attitudes toward
and beliefs about the class. They differ in their prior knowledge, their motivation, and their judgments
about how valuable and challenging the class will be and how hard they expect to work. It is the
teacher’s task to create an environment that turns as many students as possible into engaged, effective
learners who see the value in what they are learning. Teachers can design examples, formative
assessments, feedback, or learning activities specifically to address certain challenges. Teachers should
have a variety of teaching techniques at their disposal that they can use depending on particular
challenges to learning that arise (Willingham & Daniel, 2021).
The point at which the teacher successfully addresses all cognitive challenges such that effective
learning becomes possible is a “teachable moment” (Havighurst, 1953). Using the Cognitive Challenge
Framework, teachable moments occur when the conditions below are met (the cognitive challenges for
each condition are in parentheses). Students:
• become aware of gap or error in their knowledge (Metacognition)
• see the value of correcting it (Mindset)
• have a trusted source of accurate information (Student Fear and Mistrust)
• believe they can master new understanding given sufficient effort (Mindset)
• have sufficient resources to attend to that source (Selective Attention, Mental Effort)
• have sufficient prior knowledge to comprehend information (Prior Knowledge)
• recognize when they have mastered the new understanding (Metacognition)
38
• process new information for long-term recall, application, transfer, etc. (Learning Strategies)
• prime new information to be recalled appropriately and be preferred over prior knowledge
(Misconceptions, Transfer)
Teaching skill involves knowing how to bring about teachable moments for as many students as
possible, how to identify them when they occur, and how to make the most of them.
An Expanded Role on Teacher Preparation and Professional

Development
The Cognitive Challenge Framework shows that teaching is a complex, consequential skill. Thus, teacher
training should start early for college faculty, ideally as part of graduate training, and professional
development should occur throughout the academic career (Chew et al., 2018). Teacher training and
professional development should prioritize developing the kinds of knowledge needed to be an effective
teacher (Shulman, 1986). The emphasis on developing a mental model of how people learn does not
negate the usefulness of learning new teaching practices and educational technology. Teachers should
possess a mental model that indicates how these new practices and technology can be used to address
different cognitive challenges. Teachers need to have multiple, robust teaching methods at their
disposal that they can adapt to address different obstacles in student learning.
A Contextual Approach for Conducting, Organizing, and Translating

Research
The Cognitive Challenge Framework can organize and guide pedagogical research according to which
challenges the research addresses. It highlights the importance of considering interactions among
factors and not factors in isolation. Researchers should focus on the contexts in which teaching methods
are most effective, how best to implement the methods, the kinds of learning that results, and the
boundary conditions that limit their impact. The contextual framework argues against meta-analyses
that aggregate studies across diverse contexts and samples (Bryan et al., 2021). Instead, literature
reviews and meta-analyses should be sensitive to contextual interactions that influence the impact of a
pedagogical practice. Yeager et al. (2019), for example, found that a growth mindset intervention is
most beneficial for low-achieving math students in supportive school environments (see also Walton &
Yeager, 2020). Creating a set of teaching methods to address each challenge, knowing the robustness of
certain pedagogies to address different cognitive challenges, and knowing the conditions in which they
work best, and the kinds of learning that results, would benefit teachers greatly (Willingham & Daniel,
2021).
Concluding Comments
In this chapter, I described the current state of teaching and then discussed what, in my view, needs to
be done to move teaching forward. Learning is a complex process, and research has identified a
multitude of factors that can either facilitate or hinder it. We need to apply theoretical perspectives and
empirical findings about learning to our teaching. As a first step toward this new culture, I have
described a theoretical framework that accounts for the contextual nature of learning.
In order for teaching to progress, we need a culture that recognizes the complexity of student learning,
promotes the translation between research and practice, and emphasizes the development of teaching
knowledge and skill. Pedagogy is more than just “best practices,” it is theory-driven practices that are
designed to overcome cognitive challenges and achieve specific learning goals. Many teachers and
39
researchers have begun to embrace this cultural perspective, and we must find ways to encourage this
progress.
Author Note
I thank the editors for their helpful comments on earlier drafts of this essay. Feedback and comments
are welcome and can be sent to [email protected].
References
Ahn, W., Brewer, W. F., & Mooney, R. J. (1992). Schema acquisition from a single example. Journal of
Experimental Psychology: Learning, Memory, and Cognition, 18(2), 391–412.
Ambrose, S. A., & Lovett, M. C. (2014). Prior knowledge is more important than content: Skills and
beliefs also impact learning. In V. A. Benassi, C. E. Overson, & C. M. Hakala (Eds.), Applying
science of learning in education: Infusing psychological science into the curriculum (pp. 7-19).
Society for the Teaching of Psychology, http://teachpsych.org/ebooks/asle2014/index.php
Barr, R. B., & Tagg, J. (1995, November/December). From teaching to learning: A new paradigm for
undergraduate education. Change Magazine, 27, 12–25
Bartoszewski, B. L., & Gurung, R. A. R. (2015). Comparing the relationship of learning techniques and
exam score. Scholarship of Teaching and Learning in Psychology, 1(3), 219–
228. https://doi.org/10.1037/stl0000036
Berry, J. W., & Chew, S. L., (2008). Improving learning through interventions of student-generated
questions and concept maps. Teaching of Psychology, 35(4), 305-312.
Bryan, C. J., Tipton, E., & Yeager, D. S. (2021). Behavioural science is unlikely to change the world
without a heterogeneity revolution. Nature Human Behaviour, 5(8), 980–989.
Cerbin, W. (2018). Improving student learning from lectures. Scholarship of Teaching and Learning in
Psychology, 4(3), 151–163. https://doi.org/10.1037/stl0000113
Chew, S. L. (2015) Do we know how to judge teaching? Inside Higher Education, July 27, 2015.
https://www.insidehighered.com/views/2015/07/27/essay-whether-academe-knows-how-
judge-teaching
Chew, S. L., Bartlett, R. M., Dobbins, J. E., Hammer, E. Y., Kite, M. E., Loop, T. F., Mclntyre, J. G., & Rose,
K. C. (2010). A contextual approach to teaching: Bridging methods, goals, and outcomes. In D. F.
Halpern (Ed.), Undergraduate education in psychology: A blueprint for the future of the
discipline (pp. 95–112). American Psychological Association. https://doi.org/10.1037/12063-006
Chew, S. L., & Cerbin, W. J. (2017). Teaching and learning: Lost in a buzzword wasteland, Inside Higher
Education. https://www.insidehighered.com/views/2017/12/05/need-theory-learning-opinion
Chew, S. L., & Cerbin, W. J. (2021). The cognitive challenges of effective teaching. The Journal of
Economic Education, 52(1), 17-40. https://doi.org/10.1080/00220485.2020.1845266
40
Chew, S. L., Halonen, J. S., McCarthy, M. A., Gurung, R. A., Beers, M. J., McEntarffer, R., & Landrum, R. E.
(2018). Practice what we teach: Improving teaching and learning in Psychology. Teaching of
Psychology, 45(3), 239-245. https://doi.org/10.1177/0098628318779264
Clark, C. M., & Bjork, R. A. (2014). When and why introducing difficulties and errors can enhance
instruction. In V. A. Benassi, C. E. Overson, & C. M. Hakala (Eds.), Applying science of learning in
education: Infusing psychological science into the curriculum (pp. 20-30). Society for the
Teaching of Psychology, http://teachpsych.org/ebooks/asle2014/index.php
Coertjens, L., Vanthournout, G., Lindblom-Ylänne, S., & Postareff, L. (2016). Understanding individual
differences in approaches to learning across courses: A mixed method approach. Learning and
Individual Differences, 51, 69-80. https://doi.org/10.1016/j.lindif.2016.07.003
Cox R. D. (2009). The college fear factor: How students and professors misunderstand one another.
Harvard University Press. https://www.jstor.org/stable/j.ctt13x0g4d
Daniel D. B., Chew S. L (2013). The tribalism of teaching and learning. Teaching of Psychology, 40(4), 363-
Daniel, D. B., & Poole, D. (2009). Learning for life: An ecological approach to pedagogical research.
Perspectives on Psychological Science, 4(1), 91-96. https://doi.org/10.1111/j.1745-
6924.2009.01095.x
Day, S. B. & Goldstone, R. L. (2012). The import of knowledge export: Connecting findings and theories
of transfer of learning. Educational Psychologist, 47(3), 153-176.
Dunlosky, J., Rawson, K. A., Marsh, E. J., Nathan, M. J., & Willingham, D. T. (2013). Improving students’
psychology. Psychological Science in the Public Interest, 14(1), 4–58.
mathematics. Proceedings of the national academy of sciences, 111(23), 8410-8415.
Glassick, C. E. Huber, M.T. & Maeroff, G. I. (1997). Scholarship assessed: Evaluation of the professoriate.
Jossey-Bass.
Hattie, J. (2008). Visible learning: A synthesis of over 800 meta-analyses relating to achievement.
Routledge.
Havighurst, R. J. (1953). Human development and education. Longmans, Green and Co.
Jenkins, J. J. (1979). Four points to remember: A tetrahedral model of memory experiments. In L.S.
Cermak and F. I. M. Craik (Eds.) Levels of processing in human memory. Erlbaum Associates,
pp. 429–446.
McDaniel, M. A., & Butler, A. C. (2011). A contextual framework for understanding when difficulties are
desirable. In A. S. Benjamin (Ed.), Successful remembering and successful forgetting: A festschrift
in honor of Robert A. Bjork (pp. 175–198). Psychology Press.
Nuthall, G. (2007). The hidden lives of learners. New Zealand Council for Education Research Press.
41
Perry, T., Lea, R., Jørgensen, C. R., Cordingley, P., Shapiro, K., Youdell, D. (July, 2021). Cognitive science
approaches in the classroom: A review of the evidence. Education Endowment Foundation.
https://educationendowmentfoundation.org.uk/public/files/Publications/Cognitive_science_ap
proaches_in_the_classroom_-_A_review_of_the_evidence.pdf
Pintrich, P. R. (2004). A conceptual framework for assessing motivation and self-regulated learning in
college students. Educational Psychology Review, 16(4), 385-407.
https://doi.org/10.1007/s10648-004-0006-x
Ravizza, S. M., Uitvlugt, M. G., & Fenn, K. M. (2012). Logged in and zoned out: How laptop internet use
relates to classroom learning. Psychological Science, 28(2), 171-180.
Richardson, J. T. E. (2005). Students’ approaches to learning and teachers’ approaches to teaching in
higher education. Educational Psychology, 25(6), 673-680.
Rosenshine, B. 2012. Principles of instruction: Research-based strategies that all teachers should know.
American Educator, 36(1), 12–19. https://files.eric.ed.gov/fulltext/EJ971753.pdf
Schwartz, D.L., Bransford, J. and Sears, D. (2005). Efficiency and innovation in transfer. In J. Mestre (Ed.),
Transfer of learning from a modern multidisciplinary perspective (pp.1–51), Information Age.
https://aaalab.stanford.edu/assets/papers/2005/Efficiency_and_Innovation_in_Transfer.pdf
Schwartz, A. M., & Miller, R. L. (2020). High impact educational practices: A review of best practices with
illustrative examples. In A. M. Schwartz & R. L. Miller (Eds.), High impact educational practices: A
review of best practices with illustrative examples (pp. 5–27). Society for the Teaching of
Psychology. https://teachpsych.org/ebooks/highimpacted
Shulman L. S. (1986). Those who understand: Knowledge growth in teaching. Educational Researcher,
15(2), 4-14. https://doi.org/10.3102/0013189X015002004
Stains, M., Harshman, J., Barker, M. K., Chasteen, S. V., Cole, R., DeChenne-Peters, S. E., Eagan, M. K., Jr,
Esson, J. M., Knight, J. K., Laski, F. A., Levis-Fitzgerald, M., Lee, C. J., Lo, S. M., McDonnell, L. M.,
McKay, T. A., Michelotti, N., Musgrove, A., Palmer, M. S., Plank, K. M., Rodela, T. M., . . . Young,
A. M. (2018). Anatomy of STEM teaching in North American universities. Science, 359(6383),
1468–1470. https://doi.org/10.1126/science.aap8892
Taylor, A. K., & Kowalski, P. (2014). Student misconceptions: Where do they come from and what can we
do? In V. A. Benassi, C. E. Overson, & C. M. Hakala (Eds.), Applying science of learning in
education: Infusing psychological science into the curriculum (pp. 259-273). Society for the
Teaching of Psychology. http://teachpsych.org/ebooks/asle2014/index.php
Trigwell, K., Prosser, M., & Waterhouse, F. (1999). Relations between teachers' approaches to teaching
and students' approaches to learning. Higher Education, 37(1), 57-70.
https://www.doi.org/10.1023/A:1003548313194
Van Merriënboer, J. J. G., & Sweller, J. (2005). Cognitive load theory and complex learning: Recent
developments and future directions. Educational Psychology Review, 17(2), 147–177.
Walton, G. M., & Yeager, D. S. (2020). Seed and soil: Psychological affordances in contexts help to
explain where wise interventions succeed or fail. Current Directions in Psychological Science,
29(3), 219–226. https://doi.org/10.1177/0963721420904453
42
Weimer, M. (2002). Learner-centered teaching: Five key changes to practice. John Wiley & Sons.
Weinstein, Y., Sumeracki, M., & Caviglioli, O. (2019). Understanding how we learn: A visual guide.
Routledge.
Willingham, D. T. (2017). A mental model of the learner: Teaching the basic science of educational
psychology to future teachers. Mind, Brain, and Education, 11(4), 166-175.
https://doi.org/10.1111/mbe.12155
Willingham, D. T., & Daniel, D. B. (2021). Making education research relevant: How researchers can give
teachers more choices. Education Next, 21(2), 10-15.
Yang, C., Luo, L., Vadillo, M. A., Yu, R., & Shanks, D. R. (2021). Testing (quizzing) boosts classroom
learning: A systematic and meta-analytic review. Psychological Bulletin, 147(4), 399–
435. https://doi.org/10.1037/bul0000309
Yeager, D. S. & Dweck, C. S. (2012). Mindsets that promote resilience: When students believe that
personal characteristics can be developed. Educational Psychologist, 47(4), 302–314.
Yeager, D. S., Hanselman, P., Walton, G. M., Murray, J. S., Crosnoe, R., Muller, C., Tipton, E., Schneider,
B., Hulleman, C. S., Hinojosa, C. P., Paunesku, D., Romero, C., Flint, K., Roberts, A., Trott, J.,
Iachan, R., Buontempo, J., Yang, S. M., Carvalho, C. M., . . . Dweck, C. S. (2019). A national
experiment reveals where a growth mindset improves achievement. Nature, 573(7774), 364–
369. https://doi.org/10.1038/s41586-019-1466-y
Yeager D. S., Purdie-Vaughns V., Garcia J., Apfel N., Brzustoski P., Master A., Hessert W. T., Williams
M.E., & Cohen G. L. (2014). Breaking the cycle of mistrust: Wise interventions to provide critical
feedback across the racial divide. Journal of Experimental Psychology. General, 143(2), 804–824.
43
In Their Own Words Research Through the Eyes of Teachers

You to Know About Why and How Luke I. Rowe
Your Academic Setting Australian Catholic University
Editors
John Hattie
Lauren L. Kordonowy
Victor A. Benassi The University of Melbourne
How effective are the learning sciences at equipping teachers to improve student learning? For all the
promising research that has been produced by learning scientists over the past three to four decades,
there is surprisingly little evidence to help us answer this question. This is a sobering message to open a
chapter in a book that focuses on what, why, and how to integrate findings from the learning sciences
into teaching practices. Yet, we must first ask ourselves why findings from a body of evidence that holds
so much promise could remain mostly misunderstood and largely neglected by the teaching profession,
a profession whose very role is predicated on the supposition that if teaching is happening, learning
must be happening. If learning is one of the core tenants of teaching (and we argue that it is), and the
learning sciences provide the most promising body of knowledge to draw upon to inform effective
teaching and learning (and we argue that it is), then why do teachers seldom concern themselves with
this increasingly sophisticated body of evidence to inform their profession? Research from the learning
sciences continues to be published rapidly, but the uptake of the evidentiary value of this research
remains slow. Clearly, there has been a misunderstanding. But too much blame has been put on
teachers for this predicament, with little attention given to the responsibility of learning scientists. In
this chapter, we argue that as a community of learning scientists we have often missed the mark when it
comes to educating educators. We suggest much of this is because the science of learning research
community has failed to develop a self-referential body of knowledge.
Learning scientists have too frequently failed to apply the very principles derived from the science of
learning unto themselves. Meanwhile, we expect teachers to do otherwise. We expect them to
effectively learn about this potentially potent body of knowledge despite it being communicated in a
manner that is often misaligned with the very principles that derive from this body of knowledge.
Writing a 6000-word book chapter nested within an academic text on the science of learning, with the
hope of helping others learn about such findings, could hardly be more ironic!
Our hypocrisy notwithstanding, we hope this chapter provides an opportunity to examine some key
trends emerging from the learning sciences over recent decades and share examples of what has and
has not worked. We explore potential reasons why the science of learning has so far failed to become a
central component of teacher education programs and we provide some examples of how this can be
remedied in the future. We also use this section to point to some encouraging research that suggests
lessons from the learning sciences can be successfully integrated at scale. Finally, we close our chapter
by proposing a set of challenges to the learning sciences community and all those responsible for
educating educators, including lead-teachers, academics, and tertiary institutions. We situate these
challenges in the context of an overarching principle that calls upon learning scientists to use the science
of learning to effectively teach the science of learning. We hope this section reminds readers that unless
44
and until this body of evidence becomes self-referential in how it is communicated with teachers, it will
remain on the fringes of the wide variety of professions that could otherwise benefit from such lessons.
Research Is Fast, but Uptake by Teachers Is Slow – Why?

The science of learning is a multidisciplinary approach that systematically applies scientific methods to
investigate the causes, correlates, and consequences of learning. It is made up of a community of
researchers both past, present, and future who, through their varied interests and backgrounds, help
build a scientific body of knowledge that reflects a growing understanding of how learning works.
Although it is nearly impossible to summarize the countless contributions to the learning sciences, there
are three critical periods that have punctuated its modern history and made it into the interdisciplinary
field we see today: the first period was associated with radical behaviorism (e.g., Pavlov’s Classical
Conditioning, Skinner’s Operant Conditioning, Watson’s learned phobias); the second with the cognitive
revolution (e.g., Piaget’s theory of cognitive development, Vygotsky’s sociocultural theory of learning,
Neisser’s work on memory, Bandura’s socio-cognitive learning theory); and the third period, which has
left the learning sciences abuzz with much talk of ‘learning in the brain’, has largely emerged from
research conducted in the fields of neurobiology and cognitive neuroscience (stemming back to
Lashley’s law of mass action and equipotentiality, Hebb’s cell assembly theory, Bliss’ work on long-term
potentiation, Seidenberg and McClelland’s distributed and hierarchical memory, and more recently
Maguire’s research on ‘plastic’ changes in London taxi drivers, to name a few).
This triadic approach to the learning sciences – sometimes referred to as mind, brain, education
(emerging from the disciplines of psychology, neuroscience, and education), is arguably among the most
productive when it comes to interdisciplinary collaborations responsible for advancing the learning
sciences among educators. Some believe the field of education to be on the precipice of its own
revolution. Tom Bennett (as cited in Kirschner & Hendrick, 2020), founder of researchED, an
organization which aims to bridge the gaps between research and practice in education, makes the
following argument:
Future historians of education will look back on this period as a Renaissance; a time when
dogma and orthodoxy were being challenged, and gatekeepers, priesthoods, and shamans felt
the ground shift beneath their feet. The sleep of reason has bred monsters of pedagogy, and
they have been fattened and nurtured by the relative ignorance of the teaching profession. Not
a general ignorance, but a specific one: ignorance of the evidence bases behind the claims made
in education. This Renaissance has been accompanied by an evolution, as teachers and
academics reach out to one another and seek sincere, authentic dialogue (p. ii).
It is here that we must take stock and ask ourselves if this is such a crucial moment in the history of
education, where to next? Why is it that the impressive gains in our understanding of learning have not
been met by an equally remarkable uptake of evidence by teachers? Roediger (2013) argued that
research from the learning sciences has indeed influenced educational practice, but only “in fits and
starts” (p. 1). Some have been less optimistic in their appraisal, arguing that teaching has remained
essentially unchanged for over 150 years (Hattie, 2005; Tyack & Cuban, 1995).
Most graduate-level learning sciences programs sit outside of education faculties, where one might
otherwise expect interest in this topic to be high. The findings of one recent survey of 75 graduate
learning sciences programs by Sommerhoff et al. (2018) revealed that the five most common disciplines
involved in the delivery of these programs were computer science (48%), psychology (35%), science and
science education (35%), engineering (25%), and education (21%). According to the programs’ public
websites, the ‘conceptual core’ of the syllabus comprised educationally relevant topics such as designing
learning environments and scaffolding, technology-supported learning (pre-COVID-19), cognitive and
45
metacognition, formal/informal learning contexts, and disciplinary learning (Sommerhoff et al., 2018, p.
342). Why only 21% of programs were delivered in part by education-related faculties is particularly
puzzling for a discipline closely aligned with these topics and one that would likely lose its relevance if
not for its capacity to promote learning.
Furthermore, there is growing evidence that the learning sciences have had a minimal influence on
Teacher Education Programs (TEP). For example, a report by Pomerance et al. (2016) used six strategies
from the learning sciences (e.g., pairing graphics with words, distributing practice, per Pashler et al.,
2007) as the basis of a textbook analysis. They found most tertiary-level teacher-training programs in the
United States failed to adequately cover these in any sufficient detail, while many teacher-texts ignored
them altogether. Surma et al. (2018) encountered a similar issue when they analyzed the topic coverage
of two highly effective learning strategies, distributed and retrieval practice, in materials represented in
83% of Flemish and Dutch TEPs. Of the 136 textbooks and syllabi documents used to substantiate these
programs, only 21% provided full coverage of deliberate practice, two-thirds had no mention of
deliberate practice, and 84% failed to refer to retrieval practice. Only 3 of 24 programs provided an
evidence-based account of deliberate practice, while 6 of 24 did so for the notion of retrieval practice.
Similar trends are also observed among practicing educators who often prefer to draw upon their
experience, swap ideas with colleagues, or defer to their intuitions than consult evidence to inform their
practice (Lysenko et al., 2014).
Alas, there remains a stubborn gap in which research seems to be ‘lost in translation.’ It has become
increasingly apparent that merely disseminating information about teaching and learning strategies,
however effective they may be, is not enough to prevent teachers from resorting back to what they
have always done (Levin, 2013; Slavin, 2017, 2020). The translation gap between the learning sciences
and its application, between researchers and practitioners, remains one of the least systematically
studied areas of the science of learning. While we acknowledge a critical need for our field to reflect
upon and study these translation gaps collectively, we offer a series of conceptual arguments and
evidence-informed examples that may help us traverse this gap. We believe that insofar as the goal of
the learning sciences is “application,” and specifically mobilization and utilization, researchers must go
to greater lengths to examine why it is that such gaps continue to exist. The relevance of the learning
sciences in the future will depend on it.
Barriers and Bottlenecks in the Use of Evidence From the Learning Sciences
Why do most educators and teacher educators continue to overlook attempts from learning scientists to
advance their field when there is an abundance of scientifically credible evidence pointing toward more
effective teaching and learning practices? There are likely many varied and complex answers to this
question. Table 1 provides a non-exhaustive list of examples of barriers to the uptake of evidence
focusing on four major themes: objections based on personal beliefs; objections based on gaps in
knowledge and skills; objections based on systematic barriers; and objections based on concerns about
the quality of evidence in educational research.
Table 1
Common Obstacles to Evidence-Informed Practice Among Educators
Themes Obstacles to evidence
Objections based on • Believes research is irrelevant to education
personal beliefs • Believes research has the potential to be damaging or harmful to
learning (e.g., ignores local context)
46
• Believes research conflicts with the broader goals of education
• Believes research is poor- or low-quality (e.g., using averages to hide
important trends among subgroups)
• Believes science of learning evidence is blindly privileged above other
forms of evidence
• Believes this form of evidence promotes measurement at the expense
of equity and equality
• Believes practical knowledge, experience, and intuition should be
privileged above scientific forms of evidence
Objections based on • Research is difficult to understand or too technical
gaps in knowledge and • Lack of expertise in translating evidence to practice
skills
• Low research literacy—e.g., lacks expertise in discerning poor from
high-quality research, unsure of how to interpret statistics
• Awareness of evidence is low
• Unsure of where to source credible evidence
• Unsure of how to discern experts and non-experts
• Unsure where to begin or take first steps
Objections based on • Poor buy-in/low commitment from school or district leaders
systematic barriers • Discouraged by peers and colleagues to use evidence in practice
• Incentive and reward systems are not aligned with the use of best
practice based on evidence
• Lacks the time to explore, translate, implement, and review evidence-
informed practices
• Lacks support in implementing evidence into practice
• Contradicts local experience
• Overreliance on expensive resources to successfully implement
programs
• Research is inaccessible—e.g., sits behind journal paywalls
• Received little or no training in how to conduct, interpret, or apply
research
• Evidence may be accessible only in a foreign language
Objections based on • Believes transient effects that dissipate with time
concerns about research • Major concerns about the replicability of educational research
quality
• Major concerns about the study designs and main methods used to
collect data in educational research (e.g., correlational studies)
• Major concerns about the statistical inferences made in educational
research (e.g., making causal inferences based on correlational
results)
47
• Believes research has low generalizability to diverse populations (e.g.,
research from Western, Educated, Industrialized, Rich, and
Democratic (W.E.I.R.D) samples will not generalize to non-WEIRD
populations) (cf. chapter by Ekuni et al., this volume).
Note. The notion of research refers to research derived from the learning sciences, particularly from the
triad of psychological science (especially cognitive psychology), neuroscience, and educational science.
Some of these objections were inspired by Slavin (2020, pp. 26–29).
An empirical attempt at understanding why teachers do/do not use educational research to inform their
practice was conducted by Lysenko et al. (2014) and involved administering forty-three items in the
Questionnaire about the Use of Research-Based Information (QURBI) to teachers (n = 1,979), school
administrators (n = 125), and professionals (n = 321) from 66 public secondary schools in Canada.
Overall, uptake was extremely low with participants reporting having used evidence to inform
educational practice only once or twice in the previous year. Websites were the most used source of
evidence, and the most common reason for use was “To improve your professional practice.” Teachers
rated their awareness of research slightly higher if research results were “accompanied by clear and
explicit recommendations” or based on “discussions of research-based information with colleagues.”
Statistical analysis suggested the most important predictor of use of evidence to inform practice was the
practitioners’ (e.g., teachers’) attitudes and opinions about research (7% to 9% of variance explained),
expertise in finding, reading, and then translating research into practice (5%), and organizational factors
(1%). The authors concluded that evidence-based practice among educators tends to coincide with
developing positive attitudes toward evidence, building expertise in the use of evidence, increasing
awareness and accessibility of evidence, and creating an organizational culture in support of evidence-
based practice (Lysenko et al., 2014, p. 21). Yet this conclusion was based on educators who shared a
generally weak disposition toward evidence. It may therefore make sense to not only consider what
promotes the use of evidence among educators but also what stops them from using it in the first place.
Some arguments suggest that barriers to using evidence may be less about practical gaps in knowledge,
skills, and resources, and more aligned with principled objections and personal beliefs. Biesta (2009,
2012) illustrated these sentiments among educators when he argued that education has broader
interests than mere learning:
“Education is about education, not about learning. . . . This requires that we keep working on
the question as to what is distinctively educational about education and that we resist the
learnification of educational discourse and practice wherever and whenever it arises” (pp. 583–
584).
His notion of the ‘learnification’ of education speaks to the potential pitfalls of seeing everything in
education through the lens of learning—students as ‘learners,’ teachers as ‘facilitators of learning,’ and
education is, in his view, reduced to ‘teaching and learning.’ He argues that learning is nested within the
context of education and inexorably bound to content; education, on the other hand, has a purpose that
goes beyond learning. It is our view, however, that learning is an essential aspect of human nature that
helps meet fundamental needs and, in this sense, is as much of a teleological practice as is education.
Education can be, in sharp contrast to Biesta’s views, an extremely important vehicle in helping students
nurture and meet this fundamental human need. Learning can, therefore, be much broader than
education, and yet paradoxically, education can also be much broader than learning. They are not
48
mutually exclusive fields and provide crucial means to each other’s ends, particularly when areas of
mutual benefit are identified (see Figure 1).
Figure 1
Learning and Education as Overlapping but Distinct Fields
Nevertheless, Biesta’s (2012) main criticism stands in that educational researchers and learning
scientists too often make the mistake that evidence-based reforms in education be predicated on
narrow definitions of learning, teaching, and education. To the extent we continue to make this mistake,
the learning sciences community will remain on the edges of reform movements; abundant in policy but
missing in practice. The notion of the ‘boomerang effect’ seems befitting of this. Hovland et al. (1957)
coined this term in their research on persuasive communication when they noticed that “attempts to
change attitudes in the direction advocated by communication on a social issue at times produced shifts
in the direction opposite to that intended” (p. 244). To avoid this effect, we must create win-win
relationships in which the goals of these two fields overlap, and there is a strong alignment of mutual
benefit, power, and agency. Learning scientists, who wish to push their agenda upon educators, who see
translation as a one-way street, who think themselves more knowledgeable than educators, will be
rightfully excluded from this zone of mutual benefit. Learning sciences can and should aim to achieve a
privileged role in the hearts and minds of educators, but we must earn this influence by way of a deeper
understanding of the teaching profession’s beliefs, attitudes, understandings, and use of evidence.
Improving Receptivity Toward the Use of Evidence From the Learning

Sciences
Slavin (2017) argued that evidence-based reform could not prevail unless three conditions are met: a
variety of programs based on high-quality evidence; trusted and user-friendly evidence reviews; and
policy incentives to engage in evidence-informed programs. We add an additional condition of getting
smarter about how to utilize and mobilize research evidence and not just ‘dump’ it on teachers. It is not
difficult to see major strides in each of these areas. For example, innovative approaches to sharing
lessons from the learning sciences are giving cause for optimism. This is reflected in a growing number
of peer-reviewed articles in which the main goal is to promote the appropriate application of lessons
from the learning sciences into classroom practice (e.g., Dunlosky et al., 2013; Weinstein, Madan, et al.,
2018). These articles are often distinguishable from conventional academic publications because they
have been written in non-expert language, are relatively brief, are accompanied by excellent visual
diagrams, flow-charts, or worked examples, and are made freely accessible to the teaching community
rather than being hidden behind paywalls that are reserved exclusively for the academic community.
49
There has also been a recent trend over the past two decades to create books and associated resources
that seek to “summarize” the best available evidence from the learning sciences – a challenge that may
otherwise deter teachers who often lack the time, expertise, and access to what is too often an
unwieldy quagmire of peer-reviewed papers hidden behind expensive paywalls that may cost as much
as an entire book for single-use access. This democratizing approach to educational evidence was
pioneered in the early to mid-2000s by Marzano (e.g., Marzano, 2007; Marzano et al., 2001) and Hattie
(e.g., Hattie, 2008, 2012; Hattie & Yates, 2014) and has since evolved in increasingly creative ways.
For example, the predecessor to the current volume by Benassi et al. (2014) is a freely available e-book
that has been viewed more than any other in the history of the Society for the Teaching of Psychology e-
book series and has been integrated into the syllabus of several science of learning courses. Kirschner
and Hendrick’s (2020) How Learning Happens summarizes key findings from 28 seminal publications
across six areas of education and learning (e.g., how the brain learns, teacher activities, learning in
context), with an abundance of dot-points, pictures, and flow-charts to aid in understanding and
retaining key messages. Weinstein et al.’s (2018) Understanding How We Learn provides a visual guide
to six key strategies from the learning sciences by making the most of worked examples, flow-charts,
highly relevant and engaging pictures, and plain-language explanations throughout (also, Sumeracki et
al., this volume). Similarly, Watson and Busch (2021), despite having no formal background in the
learning sciences, have recently published their second edition of The Science of Learning: 99 Studies
That Every Teacher Needs to Know, which aims to support teachers by summarizing key studies in the
science of learning, again using relevant pictures, dot-points, and one- or two-page summaries that
focus on classroom implications that can be easily printed and displayed in class or staffrooms in schools
and tertiary institutions. Other books, such as Make it Stick (Brown et al., 2014), have become popular
among educators and learning enthusiasts1 not so much for their expository brevity as their ability to
string together many key principles from the learning sciences into a compelling narrative – which itself
speaks to a powerful principle from the learning sciences resulting in superior comprehension and recall
(Fiorella & Mayer, 2015; Mar et al., 2021). Each of these innovative examples seeks to capitalize on the
‘translation gap’ left behind by learning scientists.
Similarly, Table 2 describes several examples of the growing number of organizations that use websites
to offer innovative solutions to many translation problems faced by researchers and educators.
Specifically, they provide teachers with a variety of up-to-date, freely accessible, high-quality (or at least
quality indexed), and readily translatable evidence summaries from the learning and educational
sciences.
Table 2
Organizations Specializing in Mobilizing Research from the Learning Sciences to Educational Practice
Organization Website Description
Best-Evidence https://besteviden Provides critical summaries and reports of educational
Encyclopedia ce.org/ evidence (mostly meta-analyses) focusing on improving
learning in reading, mathematics, writing, science, early
childhood education, and other topics relating to students in
the K-12 setting
1
This book has one of the highest review scores in its genre, with over 13,000 reviews and an average score of
4.14 out of 5 stars on https://www.goodreads.com/book/show/18770267-make-it-stick at the time of writing
this chapter (September, 2021).
50
Evidence for https://www.evide A database of educational evidence summaries, estimates of
ESSA (Every nceforessa.org/ effectiveness, and implementation requirements to help
Student educational leaders, teachers, and parents improve student
Succeeds Act) learning
Visible https://www.visibl Statistically summarizes results from over 1,600 meta-
Learning Metax elearningmetax.co analyses on educational influences, including estimates of
m/ effectiveness and credibility of the research
Learning https://www.learni Resources (e.g., podcasts, posters) and evidence-summaries
Scientists ngscientists.org/ for teachers, students, and educators based on cognitive
science
PEN Principles https://www.slrc.o Videos and posters about 12 principles for learning based on
rg.au/resources/pe evidence from Psychology, Education, and Neuroscience
n-principles/
Retrieval https://www.retrie Resources (e.g., practice guides) and strategies for teachers
Practice valpractice.org/ to improve the use of retrieval practice in classrooms
Institute of https://www2.ed.g Reviews, summarizes, and implements research to help
Education ov/about/offices/li guide evidence-informed policy and practice in education.
Sciences st/ies/index.html Includes the What Works Clearinghouse (WWC), National
Center for Education Evaluation and Regional Assistance
(NCEE), and the National Center for Educational Research
(NCER)
Evidence for https://evidencefor Summarizes research aimed at learning in Australian
Learning learning.org.au/ classrooms. Includes evidence-informed estimates of average
months of growth, cost of implementation, and security of
various influences on learning
Education https://educatione Hundreds of studies are summarized on the website, dozens
Endowment ndowmentfoundati of large-scale classroom-based evaluations have been
Foundation on.org.uk/ conducted, and research partnerships have been built with
over half of England’s 20,217 state-funded schools and 40
‘Research Schools’ across the country, in which the EEF
supports the use of evidence among educators to improve
teaching and learning (Edovald & Nevill, 2021). The website
also provides resources for universities and research
communities to bring their own evidence-based teaching
programs to scale
Deans for https://deansforim Build partnerships with ‘Deans’ and other educational
Impact pact.org/ leaders in the USA to integrate an evidence-informed
framework into educator-preparation programs with the aim
of improving teaching practice. A recent study by Deans for
Impact (2021) involving 1,036 teacher candidates showed
significant improvements in knowledge of the learning
sciences following targeted training in 6 key dimensions
(Connecting the Dots; Managing the Learning Load;
51
Deepening Meaning and Learning; Practicing with Purpose;
Building Feedback Loops; and Creating a Motivating
Environment). Trends also showed superior outcomes for
those who received more learning-science instruction (none
vs. low, medium, high).
Note. Information, services, and resources in the descriptions above are non-exhaustive representations
of what these organizations and their respective websites offer and were necessarily constrained to that
which was most relevant to the purview of this chapter.
Excellence at Scale
Bringing evidence to scale is not the same thing as bringing excellence in learning to scale. Evidence can
be ubiquitously available, even widely understood, but poorly implemented. Evidence for evidence’s
sake is just another example of what has come to be known as Campbell’s law: “The more any
quantitative social indicator is used for social decision-making, the more subject it will be to corruption
pressures and the more apt it will be to distort and corrupt the social processes it is intended to
monitor” (Campbell, 1979, p. 34). Major past reform movements, such as the high-stakes testing era
associated with the No Child Left Behind Act 2001, arguably succumbed to these pressures (Hursh,
2005). Such corruptions not only damage lives and lead to lost opportunities for authentic learning
among students, but also create resistance among educators. It could be argued that we are surrounded
by too much evidence and not enough interpretation and implementation of evidence. For this reason,
evidence-based reform must be vigilant at anchoring evaluations of excellence not on the proliferation
of evidence or measures of teachers’ knowledge of evidence but instead on the impact evidence-based
claims have on student learning. This is, after all, the claim that lies at the heartland of this educational
reform movement: That credible evidence from the learning sciences will lead to improved learning
among students, and ultimately improve the educational system and the lives of those within it. If this is
not happening, then something is wrong with the evidence, the implementation, or both.
Policies and incentives should be geared toward the adoption of high-impact, evidence-based programs
in schools and universities, with the condition that the surrounding educational architecture,
encompassing things like professional development, online learning systems, resources, and
publications - will be organized in support of those ends (Slavin, 2020). But this will not be enough. We
must also celebrate and make visible those who successfully implement evidence at all levels of scale. In
other words, we need to have excellent evidence of evidence-based practice so that students, teachers,
educational administrators, parents, and educational policy makers have visible examples of what these
practices look like and how they can be related to their circumstances. In Australia, the Australian
Institute for Teaching and School Leadership (AITSL) is striving to achieve this in a variety of ways,
including with brief videos about evidence-informed teaching practices in action – by teachers for
teachers. This has the added benefit of promoting a research-rich culture among educators, where they
can use artifacts and exemplars as the basis for collaborative inquiry cycles that focus on a particular
aspect of evidence, putting them in the driver’s seat of change rather than having change imposed on
them (see Furlong et al., 2014, p. 39; Rickinson & Edwards, 2021).
Others have suggested schools and universities develop stronger partnerships as a means of overcoming
the barriers to evidence-informed practice. These may take the form of teacher-research projects,
evaluations, professional learning and development, or be formalized through upskilling qualifications. A
52
review of such partnerships by Cain (2019) found school-university partnerships often worked best
when formed around longer-term rather than short-term programs, broad networks rather than
individuals, and consistent value-adding activities rather than programs in which benefits rapidly
dissipate. An impressive body of research by Nutley (e.g., Davies et al., 2008; Nutley et al., 2007, 2019)
shows that schools can indeed be successfully transformed into research-rich, evidence-informed
institutions as long as they consistently engage in long-term, interactive programs and partnerships that
encourage open communication among all stakeholders, are sufficiently resourced (e.g., technologically,
financially, and emotionally), and build strategic social systems in which evidence is frequently
championed, incentivized, integrated, evaluated, and celebrated.
Another recent study that provides a reason for optimism about the uptake of evidence comes from a
survey of 419 Australian and 135 British teachers who were asked about their engagement with
educational research (Mills et al., 2021). In contrast to much of the previous trends in the literature,
results revealed a relatively strong orientation toward evidence. For example, over 80% of the Australian
teachers surveyed agreed that they were “good at evaluating the quality of published research,” and
around 70% agreed or strongly agreed that they could “translate research findings into classroom
practice” (p. 86). Similar trends were observed among teachers from England for both of these items,
with 75% indicating that ‘I could probably do this’ or ‘I could certainly do this’ when asked whether they
were capable of “evaluating the quality of published research while 80% indicated that ‘I could probably
do this’ or ‘I could certainly do this’ when asked whether they were capable of “translating research
findings into classroom practice.” These findings must nevertheless be interpreted with caution as they
may be more indicative of overconfidence among educators rather than a true reflection of their
objective abilities, which also tends to happen across a wide variety of areas in which people are asked
to estimate and predict their own skills and abilities (e.g., Dunlosky & Rawson, 2012; Dunning et al.,
1990).
Two Major Challenges, Seven Minor Challenges

Major Challenge 1. A Framework for Evaluating, Organizing, and Communicating the
Learning Sciences
Some disciplines have progressed by building a body of knowledge that relies upon ‘first principles’, such
as those seen in physics (e.g., the ‘standard model’), or adding to a ‘unifying theory’, as in evolutionary
biology, where behavior is interpreted through the prism of solving adaptive problems that promote
survival and reproduction. At present, the learning sciences do not have the luxury of ‘first principles’ or
a ‘unifying theory’ of learning. This sometimes makes for a confusing and chaotic field that fails to offer
researchers and practitioners a coherent framework to organize and reference their scientific findings.
The Encyclopedia for the Sciences of Learning (Seel, 2012), last updated in 2012, has over 2,000 entries
that range from theories about memory, such as ‘Depth of Processing’ theory, to comparative
musicology and ethnomusicology with a focus on ‘World music education’. Notably, these entries are
not organized thematically or conceptually, but instead alphabetically. In our view, this speaks to a
distinct absence of any compelling framework that might help organize this multitude of entries in a
superior way compared to the alphabet. Furthermore, there is no specific profession, standalone
discipline, research methodology, or programmatic syllabus underpinning the science of learning
(Sommerhoff et al., 2018). Therefore, a major challenge for the learning sciences, particularly as the
body of research grows, is to develop an organizing framework, such as a typology or taxonomy, that
situates otherwise disparate theories into key categories in the learning sciences. Roediger (2013)
recognized this point nearly a decade ago and called upon the community of learning scientists to build
a “well-developed translational educational science” (p.1).
53
Though there have been some promising attempts at achieving this outcome (e.g., Darling-Hammond et
al., 2020), more needs to be done to address the potential for continued fragmentation of the field into
disciplinary siloes that continue to evade educators. Whatever framework, taxonomy, or typology
emerges, it will need to readily lend itself to the application of its own set of best-practice principles to
be used effectively and redefine the use of evidence as an act of learning (Honig et al., 2017). That is, we
must frame evidence about learning in a way that can be more easily learned by those whose livelihoods
could greatly benefit from it. This means reinterpreting the gap between evidence and practice as a
scientific problem for the learning sciences community to address rather than simply putting the burden
of translation directly onto educators. It means focusing not just on building more stockpiles of data, but
on building frameworks that address the need for clarity around how educators are interpreting,
applying, monitoring, and updating their practices in light of evidence (Hattie, 2003, 2005).
One important but grossly unexplored area in the learning sciences that could benefit from such a
framework is the selection, sequencing, and combining of evidence-informed practices among teachers.
For example, we know that the value of retrieval practice on student learning is relatively high when
conducted as a standalone strategy (Dunlosky et al., 2013; Sotola & Crede, 2020), but what happens
when retrieval practice is combined with one, two, or three other strategies, such as interleaving, the
picture superiority effect, and elaborative interrogation? How much instructional time can realistically
revolve around isolated approaches before students begin to dread the next lesson? Do such strategies
waiver in their potency as they are repeatedly applied? And what emerges when these typically
individualistic strategies are combined with social strategies, such as cooperative learning or jigsaw
methods (e.g., Johnson & Johnson, 2014)? What comes of these effects longitudinally – do we find
incremental benefits to teaching students effective learning strategies in the early years? If so, then
could university students be the least likely to benefit from such interventions, having already proven
themselves effective at learning and (potentially) insensitive to gains or set in their ways? Would this not
have enormous implications for the efficacy of these strategies, having had their effect suppressed by
ceiling effects inherent in these samples?
Learning scientists need to do more to synthesize accessible answers to such questions, which may, in
part, explain why such an obvious disconnect exists between the proverbial laboratory and the
classroom, between the evidence and the practice. Unfortunately, many of the methods founded upon
the science of learning fade into the multitude of matters before teachers: building rapport and trust,
student safety, wellbeing, setting expectations, planning, printing, using technology, COVID-proofing
lessons, registration requirements, legal requirements, performance reviews, and maybe, with a bit of
luck, student learning. For tertiary educators, there is the added challenge of teaching in a manner that
improves Student Evaluations of Teaching (SET) even when such measures are grossly misaligned with
effective teaching and learning. For example, a meta-analysis by Uttl et al. (2017) revealed only a weak
relationship exists between students’ evaluations of teaching effectiveness and student learning. Often
SETs are based on a narrow transmission model of teaching, eschewing questions about the depth of
learning, failing to ask students to engage in non-structured questioning and ‘wicked’ problems, and
avoiding the need to engage students in dialogue. A recent review by Heffernan (2021) also found SETs
are potentially open to prejudices (e.g., sexism, racism) that have little to do with teaching effectiveness,
and that is at odds with efforts to promote gender equality, inclusion of marginalized groups, and
diversity in the tertiary sector. Yet there remains a widespread misconception that professors with high
SET scores are also highly effective teachers, despite persistent issues in how ‘effectiveness’ is being
measured.
A familiar framework for selecting, sequencing, and combining different approaches to learning could go
a long way in helping educators integrate evidence into their busy lives while providing an alternative
54
benchmark against which quality of practice could be evaluated without risk of reprisal or prejudice.
Learning scientists could likewise use such a framework to evaluate the impact they are having on the
educational community and, in turn, have a clearer direction on how to improve their science. Science is
only self-correcting if it is self-correcting (Ioannidis, 2012). The future of the learning sciences could be
much brighter if it spends even a quarter as much time thinking about how it could improve its own
practice as it does thinking about informing the practices of teachers and students; both will benefit.
Major Challenge 2. Seeing Research Through the Eyes of Teachers
One of the dictums of the Visible Learning approach is for teachers to see learning through the eyes of
students (e.g., Hattie, 2008; Hattie & Yates, 2014). This anchors teachers’ perspectives in the mind frame
of those whom they most desire to have the greatest impact—students. The same could be said of
learning and educational scientists who could vastly improve the uptake of evidence if they would see
research through the eyes of teachers. For learning scientists to have a prominent voice in the lives of
learners, they must do more to understand teachers at all levels of the education sector (Farley-Ripple
et al., 2018). Learning scientists must do more to build relationships of trust, mutual understanding, and
mutual benefit if they are to become teachers of teachers. They must approach research not only with
intentions of mutual benefit but with the very lessons embedded in this body of knowledge (Honig et al.,
2017).
These models of learning should not be considered analogous to outdated teacher-to-student models
where knowledge and skills are transmitted down a one-way street, from researcher-to-teacher.
Researchers may fall into patterns that seem to belittle, shame, or highlight the ignorance of teachers
rather than cultivating mutual understanding and shared learning. Much of the literature on so-called
‘neuromyths’ reinforces just how confused teachers are about a subject-matter in which they do not
proclaim expertise: neuroscience. These articles, sometimes well intended, imply teachers will improve
their craft if they simply learn more about the brain (e.g., Goswami, 2006). While teachers are not
experts in the brain (nor do they need to be), they remain more informed than the general public about
it (see Macdonald et al., 2017). Yet we continue to problematize the profession of teaching by
highlighting their relative ignorance (Dekker et al., 2012). This kind of ‘problematizing’ may have
unforeseen consequences that undermine efforts to promote the uptake of evidence among teachers.
First, highlighting teachers’ ignorance creates resentment toward researchers, reducing the likelihood of
building trusting partnerships. Second, it assumes efforts to know more about the brain will improve
one’s teaching, when this is far from guaranteed (Bowers, 2016; Bruer, 1997; Horvath et al., 2018).
Finally, such approaches do more to underscore the ignorance of researchers than that of teachers
because they show just how eager some researchers are to build a body of knowledge that is unhinged
from—and thus risks lacking relevance to—problems of educational practice. Less discussion on
neuromyths, more mobilization of proactive learning science implications for classrooms.
Learning scientists must therefore become students of teachers. We must study them, learn from them,
listen to them, and find out what evidence they want rather than telling them what evidence they need.
Translation must be ‘to’, ‘with’, and ‘from’ teachers. We must discover what questions are important to
teachers and approach these with the same resolve as any other field of scientific interest. We must
learn more about what factors promote or inhibit teachers’ accurate interpretation and use of evidence.
If a small but powerful minority of teachers and policy makers are prone to mindlessly (mis)apply
research, as some have observed (see Hattie, 2012), then we must understand why this lapse has
occurred and how it can be remedied.
Levin (2013) suggested we reject traditional notions of ‘translation’ because they too often assume that
evidence will provide a solution if only it were correctly translated. Instead, he prefers to use the term
55
‘mobilization’ because it emphasizes the interactive relationship between research and practice, rather
than assuming knowledge is transmitted down a one-way street. We would go a step further and argue
that it is teachers, not researchers, who possess the primary knowledge about why the barriers and
bottlenecks exist to the use of evidence, about problems of practice, and about the constraints of
integrating the learning sciences into their profession. Until learning scientists are prepared to build a
bridge that enables them to become students of teachers, opportunities for mutual benefit will be
missed. With this in mind, we close this chapter by asking seven key questions that the learning sciences
community must address in the future. These are not only questions to be answered, but surmountable
challenges to be overcome in the immediate future:
• Do teachers who apply principles from the learning sciences to their teaching help their students
learn more efficiently and effectively than teachers who know nothing of these principles?
• Does the level of depth of teachers’ knowledge about the learning sciences correspond to a
similar level of depth of commitment to changes in teaching practice and, more importantly,
improved educational outcomes for their students?
• What downstream measurable value is the learning sciences community adding to and beyond
the educational sector?
• How can we replicate the success of some learning sciences programs (e.g., Deans for Impact,
Education Endowment Foundation) and bring it scale, starting with those who need it most?
• What are the most pressing questions, problems, and needs of teachers that can be addressed
by the learning sciences community but are currently being neglected?
• How can the science of learning community better address contemporary challenges in
education such as building more cohesive and inclusive societies, learning how to live
sustainably, and breaking cycles of disadvantage?
• How can the learning sciences work at improvements on the education system, while remaining
relevant to those who must work within the education system?
If we are to see changes to education that mirror those seen in fields such as medicine, agriculture, or
technology, as they transitioned from pseudoscience to evidence-informed fields, we would be wise to
heed the advice of Kuhn (1970) who, in his ‘Structure of Scientific Revolutions’ emphasized the “need to
study the community structure of science . . . there is no area in which more work is so badly needed”
(pp. 209-10). He goes on to pay homage to Wittgenstein by arguing that “Scientific knowledge, like
language, is intrinsically the common property of a group or else nothing at all” (p. 210). The learning
sciences community must do a better job of including and elevating educators in this conversation. If
teachers remain on the fringes of the learning sciences community, learning scientists will remain on the
fringes of the educational community. Forsaking this opportunity not only risks stifling what could
become an educational revolution, but also runs the risk that the learning sciences will become “nothing
at all” to those they so desire to help. What a shame that would be.
Author Note
Corresponding author: [email protected]
56
References
Benassi, V. A., Overson, C. E., & Hakala, C. M. (Eds.). (2014). Applying science of learning in education:
Infusing psychological science into the curriculum. Society for the Teaching of Psychology.
http://teachpsych.org/ebooks/asle2014/index.php
Biesta, G. (2009). Good education in an age of measurement: On the need to reconnect with the
question of purpose in education. Educational Assessment, Evaluation and Accountability, 21(1),
33–46. https://doi.org/10.1007/s11092-008-9064-9
Biesta, G. (2012). Philosophy of education for the public good: Five challenges and an agenda.
Educational Philosophy and Theory, 44(6), 581–593. https://doi.org/10.1111/j.1469-
5812.2011.00783.x
Bowers J. S. (2016). The practical and principled problems with educational neuroscience. Psychological
Review, 123(5), 600–612. https://doi.org/10.1037/rev0000025
The Belknap Press of Harvard University.
Bruer, J. T. (1997). Education and the brain: A bridge too far. Educational Researcher, 26(8), 4–16.
Cain, T. (2019). School-university links for evidence-informed practice. Education Sciences, 9(2), 97.
https://doi.org/10.3390/educsci9020097
Campbell, D. T. (1979). Assessing the impact of planned social change. Evaluation and Program Planning,
2(1), 67–90. https://doi.org/10.1016/0149-7189(79)90048-X
Darling-Hammond, L., Flook, L., Cook-Harvey, C., Barron, B., & Osher, D. (2020). Implications for
educational practice of the science of learning and development. Applied Developmental
Science, 24(2), 97–140. https://doi.org/10.1080/10888691.2018.1537791
Davies, H., Nutley, S., & Walter, I. (2008). Why ‘knowledge transfer’ is misconceived for applied social
research. Journal of Health Services Research & Policy, 13(3), 188–190.
https://doi.org/10.1258/jhsrp.2008.008055
Deans for Impact. (2021). Deepening Meaning and Learning: How future teachers are putting learning
science into action to support all students. Deans for Impact. https://deansforimpact.org/wp-
content/uploads/2021/06/DFI_Deepening-Meaning-and-Learning.pdf
Dekker, S., Lee, N. C., Howard-Jones, P., & Jolles, J. (2012). Neuromyths in education: Prevalence and
predictors of misconceptions among teachers. Frontiers in Psychology, 3.
https://doi.org/10.3389/fpsyg.2012.00429
Dunlosky, J. (2013). Strengthening the student toolbox: Study strategies to boost learning. American
Educator, 37(3), 12–21.
Dunlosky, J., & Rawson, K. A. (2012). Overconfidence produces underachievement: Inaccurate self
evaluations undermine students’ learning and retention. Learning and Instruction, 22(4), 271–
280. https://doi.org/10.1016/j.learninstruc.2011.08.003
57
Dunning, D., Griffin, D. W., Milojkovic, J. D., & Ross, L. (1990). The overconfidence effect in social
prediction. Journal of Personality and Social Psychology, 58(4), 568.
Edovald, T., & Nevill, C. (2021). Working out what works: The case of the education endowment
foundation in England. ECNU Review of Education, 4(1), 46–64.
Farley-Ripple, E., May, H., Karpyn, A., Tilley, K., & McDonough, K. (2018). Rethinking connections
between research and practice in education: A conceptual framework. Educational Researcher,
47(4), 235–245. https://doi.org/10.3102/0013189X18761042
Furlong, J. (2014). Research and the teaching profession: Building the capacity for a self-improving
education system: Final report of the BERA-RSA inquiry into the role of research in teacher
education. British Educational Research Association, & Royal Society of Arts (Great Britain).
https://www.thersa.org/globalassets/pdfs/bera-rsa-research-teaching-profession-full-report-
for-web-2.pdf
Goswami, U. (2006). Neuroscience and education: From research to practice? Nature Reviews
Neuroscience, 7(5), 406–413. https://doi.org/10.1038/nrn1907
Hattie, J. (2003, October 19-21). Teachers make a difference: What is the research evidence? [Paper
presentation]. Building Teacher Quality: What Does the Research Tell Us, ACER Research
Conference, Melbourne, Australia. http://research.acer.edu.au/research_conference_2003/4/
Hattie, J. (2005, August 7-9). What is the nature of evidence that makes a difference to learning? [Paper
presentation]. Using Data to Support Learning, ACER Research Conference 2005, Melbourne,
Australia. https://research.acer.edu.au/research_conference_2005/7
Hattie, J. (2008). Visible learning: A synthesis of over 800 meta-analyses relating to achievement (1st ed.).
Routledge.
Hattie, J. (2012). Visible learning for teachers: Maximizing impact on learning (1st ed.). Routledge.
Hattie, J., & Yates, G. (2014). Visible learning and the science of how we learn. Milton Park, Abingdon,
Oxon: Routledge.
Heffernan, T. (2021). Sexism, racism, prejudice, and bias: A literature review and synthesis of research
surrounding student evaluations of courses and teaching. Assessment & Evaluation in Higher
Education, 1–11. https://doi.org/10.1080/02602938.2021.1888075
Honig, M. I., Venkateswaran, N., & McNeil, P. (2017). Research use as learning: The case of fundamental
change in school district central offices. American Educational Research Journal, 54(5), 938–971.
Horvath, J. C., Donoghue, G. M., Horton, A. J., Lodge, J. M., & Hattie, J. A. C. (2018). On the irrelevance of
neuromyths to teacher effectiveness: Comparing neuro-literacy levels amongst award-winning
and non-award winning teachers. Frontiers in Psychology, 9, 1666.
Hovland, C. I., Harvey, O. J., & Sherif, M. (1957). Assimilation and contrast effects in reactions to
communication and attitude change. Journal of Abnormal Psychology, 55(2), 244–252.
58
Hursh, D. (2005). The growth of high‐stakes testing in the USA: Accountability, markets and the decline
in educational equality. British Educational Research Journal, 31(5), 605–622.
Ioannidis, J. P. A. (2012). Why science Is not necessarily self-correcting. Perspectives on Psychological
Science, 7(6), 645–654. https://doi.org/10.1177/1745691612464056
Johnson, D. W., & Johnson, F. P. (2014). Joining together: Group theory and group skills (11th ed.).
Pearson Education.
Kirschner, P. A., & Hendrick, C. (2020). How learning happens: Seminal works in educational psychology
and what they mean in practice (1st ed.). Routledge. https://doi.org/10.4324/9780429061523
Kuhn, T. S. (1970). The structure of scientific revolutions (2nd ed., Enl). University of Chicago Press.
Levin, B. (2013). To know is not enough: Research knowledge and its use. Review of Education, 1(1), 2–
31. https://doi.org/10.1002/rev3.3001
Lysenko, L. V., Abrami, P. C., Bernard, R. M., Dagenais, C., & Janosz, M. (2014). Educational research in
educational practice: Predictors of use. Canadian Society for the Study of Education, 37(2), 27.
https://www.jstor.org/stable/canajeducrevucan.37.2.06
Macdonald, K., Germine, L., Anderson, A., Christodoulou, J., & McGrath, L. M. (2017). Dispelling the
myth: Training in education or neuroscience decreases but does not eliminate beliefs in
neuromyths. Frontiers in Psychology, 8, 1314. https://doi.org/10.3389/fpsyg.2017.01314
Mar, R. A., Li, J., Nguyen, A. T. P., & Ta, C. P. (2021). Memory and comprehension of narrative versus
expository texts: A meta-analysis. Psychonomic Bulletin & Review, 28(3), 732–749.
Marzano, R. J. (2007). The Art and Science of Teaching: A Comprehensive Framework for Effective
Instruction (1st ed.). Association for Supervision & Curriculum Development.
Marzano, R. J., Pickering, D. J., & Pollock, J. E. (2001). Classroom instruction that works: Research-based
strategies for increasing student achievement (1st ed.). Association for Supervision and
Curriculum Development.
Mills, M., Mockler, N., Stacey, M., & Taylor, B. (2021). Teachers’ orientations to educational research
and data in England and Australia: Implications for teacher professionalism. Teaching Education,
32(1), 77–98. https://doi.org/10.1080/10476210.2020.1843617
Nutley, S., Boaz, A., Davies, H., & Fraser, A. (2019). New development: What works now? Continuity and
change in the use of evidence to improve public policy and service delivery. Public Money &
Management, 39(4), 310–316. https://doi.org/10.1080/09540962.2019.1598202
Nutley, S., Walter, I., & Davies, H. T. (2007). Using evidence: How research can inform public (1st ed.).
Policy Press.
Pashler, H., Bain, P. M., Bottge, B. A., Graesser, A., Koedinger, K., McDaniel, M., & Metcalfe, J. (2007).
Organizing Instruction and Study to Improve Student Learning. American Psychological
Association. https://doi.org/10.1037/e607972011-001
Pomerance, L., Greenberg, J., & Walsh, K. (2016). Learning about Learning: What every new teacher
needs to know. National Council on Teacher Quality.
https://www.nctq.org/publications/Learning-About-Learning:-What-Every-New-Teacher-Needs-
to-Know
59
Rickinson, M., & Edwards, A. (2021). The relational features of evidence use. Cambridge Journal of
Education, 1–18. https://doi.org/10.1080/0305764X.2020.1865877
Roediger, H. L. (2013). Applying cognitive psychology to education: Translational educational science.
Psychological Science in the Public Interest, 14(1), 1–3.
Seel, N. M. (Ed.). (2012). Encyclopedia of the sciences of learning. Springer US.
https://doi.org/10.1007/978-1-4419-1428-6
Slavin, R. E. (2017). Evidence-based reform in education. Journal of Education for Students Placed at Risk
(JESPAR), 22(3), 178–184. https://doi.org/10.1080/10824669.2017.1334560
Slavin, R. E. (2020). How evidence-based reform will transform research and practice in education.
Educational Psychologist, 55(1), 21–31. https://doi.org/10.1080/00461520.2019.1611432
Sommerhoff, D., Szameitat, A., Vogel, F., Chernikova, O., Loderer, K., & Fischer, F. (2018). What do we
teach when we teach the learning sciences? A document analysis of 75 graduate programs.
Journal of the Learning Sciences, 27(2), 319–351.
Sotola, L. K., & Crede, M. (2020). Regarding class quizzes: A meta-analytic synthesis of studies on the
relationship between frequent low-stakes testing and class performance. Educational
Psychology Review, 33, 407-426. https://doi.org/10.1007/s10648-020-09563-9
Surma, T., Vanhoyweghen, K., Camp, G., & Kirschner, P. A. (2018). The coverage of distributed practice
and retrieval practice in Flemish and Dutch teacher education textbooks. Teaching and Teacher
Education, 74, 229–237. https://doi.org/10.1016/j.tate.2018.05.007
Tyack, D. B., & Cuban, Larry. (1995). Tinkering toward utopia: A century of public school reform. Harvard
University Press.
Uttl, B., White, C. A., & Gonzalez, D. W. (2017). Meta-analysis of faculty’s teaching effectiveness: Student
evaluation of teaching ratings and student learning are not related. Studies in Educational
Evaluation, 54, 22–42. https://doi.org/10.1016/j.stueduc.2016.08.007
Watson, E., & Busch, B. (2021). The science of learning: 99 studies that every teacher needs to know.
Routledge.
Weinstein, Y., Madan, C. R., & Sumeracki, M. A. (2018). Teaching the science of learning. Cognitive
Research: Principles and Implications, 3(2). https://doi.org/10.1186/s41235-017-0087-y
Weinstein, Y., Sumeracki, M., & Caviglioli, O. (2018). Understanding how we learn: A visual guide (1st
ed.). Routledge. https://doi.org/10.4324/9780203710463
60
In Their Own Words The Scholarship of Teaching and Learning:
Scaling New Heights, but It May Not Mean
What You Think It Means
Editors
Regan A. R. Gurung
Lauren L. Kordonowy
Victor A. Benassi Oregon State University
One of the most quoted lines from the 1990’s movie, The Princess Bride, is uttered by an ambidextrous
sword fighting Spaniard, Inigo Montoya. After hearing his comrade frequently exclaim “Inconceivable” in
response to a multitude of events, Montoya shakes his curly locks, looks bemused, and quips, “You keep
using that word. I do not think it means what you think it means.” Sans locks and rapier skills, I find
myself thinking this same thought every time someone says “Scholarship of Teaching and Learning”
(SoTL).
A look at SoTL today shows an area of research spanning multiple disciplines (Hake, 2015; McKinney et
al., 2017) and practiced in different types of academic institutions (Gurung et al., 2019). Similar to the
science of learning, faculty from different areas practice SoTL, although the work is not always referred
to similarly. Some scholars may practice Disciplinary Based Educational Research (DBER), which is
essentially SoTL within a discipline. Other scholars see themselves as part of the area described as the
learning sciences (Feldman, 2020). There are also many definitions of SoTL (Pan, 2009) and even
researchers doing work in higher education may not always recognize they are part of the learning
sciences movement or practicing SoTL (Joosten, 2020). To best reflect what the term is now commonly
understood to mean, and to include its multiple facets, my working definition of SoTL views it as
encompassing the theoretical underpinnings of how people learn, the intentional, systematic,
modifications of pedagogy, and assessments of resulting changes in learning, collectively defined
as SoTL (Gurung & Landrum, 2014).
It has been over 30 years since the phrase first entered the lexicon of higher education and it is now
acknowledged as a valid, effective practice benefiting faculty, students, and institutions (Hutchings et al.,
2011). Over the years many faculty have conducted SoTL, an activity often initiated by an instructor
asking questions about their students’ learning and aiming to improve their teaching. Today SoTL is an
active area of engagement with three journals catering to research on it just in psychology, and many
more in other disciplines. SoTL promises to shed more light on the nature of learning in the years ahead.
In this chapter I shall briefly overview the origins of the term and the work that comprises it, with an
emphasis on how it has taken on different meanings. I then consider how conceptualizations of SoTL
have changed over time, and I itemize key areas for how it needs to evolve in the years ahead.
The Origins and Evolution of SoTL

We are now in what can be called the third wave of SoTL (Gurung & Schwartz, 2010). Although the art
and science of teaching have been examined for centuries and even scrutinized by psychologists such as
James (1899), the linking of the terms “scholarship” and “teaching” did not take place until Boyer (1990)
kicked off the first wave of activity in this regard.
61
Boyer (1990) changed the narrative on how teaching is viewed and aimed to elevate the hard work of
teaching to how research is valued in higher education. To do this, he dismantled the more traditional
and restrictive view of scholarship and strove to “define the work of faculty in ways that reflect more
realistically the full range of academic and civic mandates” (Boyer, 1990, p. 16). He called traditional
scholarship the scholarship of discovery. The act of making connections across disciplines and linking
isolated facts he termed the scholarship of integration. These two forms of scholarship “reflect the
investigative and synthesizing traditions of academic life” (Boyer, 1990, p. 21). The work of making
research practical, applying and contributing to human knowledge, was termed the scholarship of
application.
Teaching “is often viewed as a routine function” (Boyer, 1990, p. 23), downplaying the careful planning,
the hard work of delivery, and the need for teachers to know their content well. Recognizing this, Boyer
introduced the moniker “scholarship of teaching” (SoT) to capture the hard work that is involved in
teaching. What is important to note is that there is no mention here of actually conducting research on
one’s teaching, but more to capturing what Richlin (2001) later termed scholarly teaching, the
thoughtful, intentional, motivational changes teachers make to their pedagogy after reflection and
reading. Many early reports of SoT involved narrative essays about teaching capturing a wisdom of
experience: “This is what I did.”
Later scholars developed six criteria by which the four types of scholarship should be judged: the work
must be characterized by clear goals, adequate preparation, appropriate methods, significant results,
effective presentation, and reflective critique (Glassick et al., 1997, p. 36). Soon, faculty from different
disciplines with training in various methodologies were applying the skills of their own disciplines to
scrutinize their own teaching. Consequently, the majority of early SoT was conducted by instructors on
their own classes, a form of design still dominating SoTL today.
Capping the first wave, the Carnegie Association added ‘Learning’ to Boyer’s “Scholarship of Teaching”
to give us the acronym we are now familiar with, SoTL. By all accounts, this morphing of SoT to SoTL was
consciously engineered to make teachers explicitly accountable for learning (Hutchings & Shulman,
1999), although it may have had the unintended consequence of also moving the onus away from
reporting of teaching activities to actively doing research on one’s teaching. Motivated by Boyer’s work,
large national studies of colleges and universities examined faculty priorities and reward structures in
different academic domains (Diamond & Adam, 2000; Glassick et al., 1997). Soon, institutional policies
changed to better factor in faculty doing research on teaching (O’Meara & Rice, 2005) and more
departments moved towards counting research on teaching as not just important, but as scholarship in
its own right. While good news for expanding knowledge on teaching and learning, it quickly took on a
life of its own, de-emphasizing descriptions of teaching efforts per se, and instead instigating research
on teaching and learning by faculty, many of whom had no relevant training or background in
educational research.
The second wave of SoTL activity saw both an increase in resources to catalyze SoTL as well as increases
in interdisciplinary SoTL. Scholars such as Donald (2000) took into account the different ways learning
occurs in various academic disciplines, which led to more examinations of one’s discipline in the light of
other disciplines and refining one’s pedagogies based on how one’s discipline is unique. Shulman (2005)
further advanced this activity coining the phrase ‘signature pedagogies’, addressing the way each
discipline teaches students to think like the professionals in that discipline (Gurung et al., 2009). This
focus on a discipline’s signatures nicely led to examinations of key ‘bottlenecks’, ‘threshold
concepts’ problems, and impediments to learning (whether concepts or processes) in disciplines (Meyer
& Land, 2005; Pace & Middendorf, 2004).
62
If the first twenty years since Boyer (1990) saw a spreading of SoTL across disciplines and various types
of institutions and a validation of its growing worth (Hutchings et al., 2011), the third wave saw an
increase in the sophistication of designs and methodologies. Conversations in the area turned towards
establishing methodological rigor (Felten, 2013) and gold standards for SoTL (Wilson-Doenges & Gurung,
2013), and there was a proliferation of outlets for SoTL. Of note, the American Psychological Association
launched a new journal in 2015, The Scholarship of Teaching and Learning in Psychology, illustrating the
growing appetite for research on teaching and learning.
There has also been a flurry of activity aimed at fine tuning and parsing out the work done under the
umbrella of SoTL. A lot of early SoTL was research conducted on individuals’ own classes and featured a
single sample. As SoTL entered the third wave of activity, not only did we see an increase in multi-site
samples (e.g., Gurung et al., 2012), making results more generalizable, we also saw moves to translate
lab research to the classroom and to design and implement classroom innovations (Daniel & Chew,
2013; Soicher, 2020). Best characterized as a rise in implementation science, a term borrowed from the
medical sciences, this evolution of SoTL is well described in a framework proposed by Soicher et al.
(2020).
In a useful visualization of different related efforts working to marry research and application, as it
applies to teaching and learning, Soicher et al.’s (2020) framework assumes that the goal of science in
this context is evidence-based practice (see Figure 1). The framework consists of three major sections -
including scholarly teaching (Richlin, 2001), science of learning, and practice-based research - and
explicitly acknowledges the connections between these areas and the recursive nature of the enterprise.
Not only does it locate the role and position of basic cognitive science under the Science of Learning, it
also separates out Scholarly Teaching, what many teachers do without publishing findings and an
activity closer to the original formulation of the Scholarship of Teaching (Boyer, 1990). This framework
also makes an important distinction between efficacy research, proof of concept lab-based research
(housed under the science of learning), and effectiveness research, which includes classroom tests of
education interventions. On one hand, efficacy research is used to evaluate the impact of an
intervention under ideal—usually laboratory—conditions. On the other, effectiveness research is
concerned with whether a pedagogical practice found to be efficacious in a laboratory setting does more
good than harm in a real-world context (Soicher et al., 2020).
Figure 1
Note. From “Adapting implementation science for higher education research: The systematic study of
implementing evidence-based practices in college classrooms” by R. N. Soicher et al., 2020, Cognitive
63
Research: Principles and Implications, 5 (1), pp. 1-15 (https://doi.org/10.1186/s41235-020-00255-0).
Copyright CC BY 4.0.
Changes in Perceptions of SoTL

Moving beyond the big picture changes in how SoTL has been conducted over the last 30 years, it is
important to also consider how perceptions of this activity and barriers to doing the work have also
changed. In addition to a small-scale study completed by Carnegie Foundation scholars over 15 years
ago (Huber & Hutchings, 2005) and another cross-disciplinary survey again by the Carnegie group
(Hutchings et al., 2011), there have been two large studies using similar survey items that provide a
good view of the changing nature of SoTL.
A task force of the Society for the Teaching of Psychology (Division 2 of the American Psychological
Association) conducted a survey to ascertain the degree to which psychology departments and the
institutions of higher education that house them have enacted SoTL (Gurung et al., 2008). Gurung and
colleagues assessed the level of support faculty perceived in their departments and at their institutions,
as well as the role of SoTL in personal decisions and obstacles to conducting SoTL. In general, “survey
respondents failed to report a prevailing sentiment of support” (Gurung et al, 2008, p. 257). Findings
regarding departmental and institutional support for SoTL presented a mixed picture. The field of
psychology seems to recognize SoTL better than higher education as a whole (i.e., when compared to
the results seen in a survey of higher education, Huber & Hutchings, 2005). For example, Gurung et al.
found that 60% of the survey respondents reported having colleagues involved in SoTL, and 78%
reported that departmental policies encourage SoTL. Both the psychology study (Gurung et al., 2008)
and the general survey (Huber & Hutchings, 2005) did find that SoTL was being considered in
department retention, tenure, and promotion, although there is still a long way to go in getting more
uniform acceptance of SoTL.
Since the first psychology SoTL survey study, other studies have focused on discipline specific
perceptions (e.g., family studies, DiGregorio et al., 2016) across academic disciplines with specific
populations (e.g., early career faculty; Matthews et al. 2014) and within one institution (Secret et al.,
2011). In general, these researchers found that individual faculty valued SoTL but that at the
institutional level, there was mixed support.
The most recent study determined how SoTL activities are evaluated within diverse departments across
diverse academic disciplines (Gurung et al., 2019). Gurung and colleagues assessed SoTL practices and
perceptions among faculty nationwide across diverse academic disciplines, institutional type, teaching
experience, and academic rank, directly comparing the results with those collected 10 years previously
(Gurung et al., 2008). The sample consisted of faculty from 21 different disciplines with half being from
psychology. Questions assessed faculty perceptions of: (1) departmental support for SoTL; (2)
institutional support for SoTL; (3) the role SoTL may play in personnel decisions; and (4) perceived
obstacles of engaging in SoTL.
The results from this national study shine a positive light on the contemporary nature of
SoTL, suggesting the practice has evolved into an accepted form of scholarship. Faculty in the study
viewed SoTL as a part of their daily teaching routines more than faculty did in the survey conducted 10
years previously (Gurung et al. 2019). Furthermore, faculty in the sample reported more departmental
support for SoTL work in 2018 than was the case in 2008. Responses clearly showed SoTL has become
more accepted and valued among faculty. Comparisons between disciplines showed that psychology
faculty agreed more than non-psychology faculty that departmental policies valued teaching reflection,
departmental norms supported SoTL activity, and the department chair encouraged SoTL involvement
(Gurung et al., 2019). Psychology faculty, relative to non-psychology faculty, also believed that their
64
departmental colleagues were supportive of SoTL efforts and that their departments supported such
work. Perhaps not surprisingly, given the emphasis of research on teaching over the mere
documentation of teaching efforts championed by Boyer (1990), faculty saw peer-reviewed publications
as the most prized SoTL product.
A Caveat
Before addressing the next steps for what SoTL has and can become, it is important to not forget about
the practice of teaching itself. Higher education first needs to recognize the importance of teaching and
put the spotlight back on this enterprise. By allowing research on teaching to creep into and then near-
completely encompass what it means to do SoTL, we sacrifice the gains made by Boyer’s (1990) coining
of the phrase “Scholarship of Teaching.” In contrast, SoTL, while shedding light on the factors that
influence learning and contributing to documenting key limiting factors and individual differences in
learning, has become something much more than capturing the hard work of teaching (e.g., course
design, assessment).
University administrators, department chairs, and those involved with retention, promotion, tenure, and
contract extensions, all need to acknowledge the hard work that is teaching. The documentation of this
work (e.g., teaching portfolios) needs to be facilitated and rewarded (Bernstein et al., 2009). Centers for
teaching and learning staff can provide faculty with efficient ways to document their teaching beyond
student evaluations of teaching. They can provide retention, promotion, and tenure committees, deans,
provosts, and those who assess teaching with best practices to review the resulting volume of
information. Although it is exciting to see the activity of SoTL today, the crux of the matter is that what is
currently encompassed by SoTL is more often what Boyer meant by the scholarship of discovery,
integration, and application. Most contemporary SoTL publications do all three of these activities.
Ironically, SoTL for many is more research-related and less teaching-related.
To not lose focus on what the Scholarship of Teaching was meant to refer, it may be prudent for us to
use a different term for what SoTL has become. One alternative is provided by Halpern et al.’s (1998)
“paradigm for the twenty-first century” (p. 1292), which extended Boyer’s four categories into five, and
it defined ‘scholarship’ to include both ‘scholarship of pedagogy (SoP)’ and ‘scholarship of teaching”. For
most who practice the science of learning, SoTL is really the SoP. This term did not gain traction in the 20
plus years since it was coined, but it may be more useful now than ever to help us differentiate between
SoT and SoTL, an important distinction that is critical to ensuring that teaching effectiveness is measured
in colleges and universities and not obfuscated with SoTL.
The Future of SoTL

Beyond the definitional issue and not shortchanging the work of teaching, there are a few major ways
research on teaching and learning can and should progress. This involves the major categories of: Doing
robust work, better assessment of learning, and focusing on individual differences.
First outlined nearly 10 years ago, gold standards of SoTL (Wilson-Doenges & Gurung, 2013) provide a
template for conducting robust research on teaching and learning. Similar to and building on Glassick et
al.’s (1997) criteria for scholarship, one can extend many of the benchmarks for what counts as good
social science research to good SoTL. Unfortunately, most of the research published in major journals
catering to SoTL do not hit many of the standards. There are also distractions such as turf battles over
which discipline’s methodology should take center stage (Chick, 2014). The simple answer is that there is
a place for both qualitative and quantitative work (Gurung, 2014). Perhaps the feature most noticeable
for its absence includes designs that extend beyond a single sample or are longitudinal in nature
(Wilson-Doenges et al., 2016). The current state of research on teaching shows few exemplars of this
65
standard. Whereas a handful of studies have included more than one institution (Gurung, 2017; Gurung
& Burns, 2019; Gurung et al., 2012), the majority do not, and few studies are longitudinal in nature.
It is possible the reason for this lack of robustness is the difficulty inherent in conducting multi-site work.
There are now exemplars of procedures to rectify the problem. In one of the most exciting
developments for SoTL, and catalyzed by the replication crisis in psychology, the ManyClasses project
(Fyfe et al., 2021) sets the stage for giant leaps forward. According to this model, many researchers all
tackle the same research question across institutions, teachers, student groups, and topics. Noting the
need for independent samples from different classes across sites consistent with calls for the same
(Gurung & Hackathorn, 2018; Wilson-Doenges & Gurung, 2013), a ManyClasses study examines
educational practices across class contexts. The robustness and rigor of a randomized experiment is
maintained while allowing teachers to prepare materials that are unique to and tailored to their colleges
and universities. Such studies are embedded experiments where a theoretically-derived variable is
manipulated, participating teachers create assignments normed for their disciplines, and then present
the material to the students.
The first published ManyClasses study (Fyfe et al., 2021) examines how the timing of feedback on class
assignments influences subsequent performance on class assessments. The large group of authors used
data from 38 different classes and found no effect of immediate versus delayed feedback on student
learning that generalizes across classes. What is impressive about this form of research is that it reduces
the likelihood of “p-hacking” with a reliance on preregistration. In the study of feedback, 40 pre-
registered moderators related to class-level and student-level characteristics were analyzed. There were
no credible non-zero effects, although the team found promise of modest advantages for feedback in
some classes. By having multiple classes in the study, the limiting factors of a phenomenon were easier
to identify and contextual factors more likely to lead to increases in learning, were similarly easier to
find. In stark contrast to the absolute majority of SoTL research that uses a single class, the ManyClasses
approach stands to change how SoTL is done.
Although significantly less complex than the ManyClasses approach, another method to facilitate multi-
site research is The Hub for Introductory Psychology and Pedagogical Research facilitates collaboration
between pedagogical researchers (Gurung & Hackathorn, 2019), an effort similar to Distance Education
and Technological Advancements work (Joosten, 2020). Researchers can find instructors willing to open
their classes to interventions and testing and connect with other faculty interested in examining
questions related to teaching and learning.
Another issue is that few studies in SoTL are longitudinal in nature, such that they extend beyond the
end of the term (for notable exceptions, see Lawrence, 2013; McCarthy & Frantz, 2016). Furthermore,
the main dependent variables are limited in scope, often being recall of studied terms in lab studies or
scores on final exams in classroom studies. How well do students remember what they have learned
after the end of a course? In one study, students retook a cumulative final exam 2 years after the end of
the course (Landrum & Gurung, 2013). Retention of material dropped from 80.6% correct to 56.0%. In
perhaps the longest-term study of learning, Hard et. al. (2019) examined retention in introductory
psychology students four years after taking the course. The researchers invited 27 senior
undergraduates who had previously taken the course either the fall or winter quarter of their first year
to participate. During their senior year, students took a test on questions pulled from their first midterm
exam and showed high performance, getting 70% correct (compared to a mean of 89% when these
items were completed during the first year). The students who accepted the invitation (66%) did
originally score higher than those that did not participate, suggesting a partial explanation for the
finding. Although the four-year follow-up only used 16 items, this is one of a small number of studies
that provide information on retention of material after a long period of time. If science of learning
66
researchers want to better understand learning as a relatively long-term change in behavior or gain in
information, more studies of this nature are urgently called for.
Ironically, for a field or area with ‘learning’ in its name, the actual measurement of learning in SoTL
studies has not received the attention it deserves, or at least not in the field of psychology. Unlike fields
such as physics, psychology lacks a well-established universal assessment of students’ knowledge of
course content and concepts that can be used across colleges and universities (Hake, 2015). How can
our discipline compare student learning if there is no consistent means of assessing knowledge gained in
major courses? Although multiple taskforces have deemed a national standardized assessment
untenable (APA, 2017; Halonen et al., 2022), the benefits of a common measure of knowledge are
difficult to ignore. Aside from sections within the very comprehensive Educational Testing
Service's Major Field Test (Educational Testing Service, 2023a), the entirety of the GRE Psychology
Subject Test (Educational Testing Service, 2023b), or a newly published measure of basic knowledge in
psychology (Solomon et al., 2021), a national assessment of psychology in the USA is lacking. Project
Assessment, featuring exemplar assessments from the 2016 Summit of National Assessment of
Psychology, contains some examples of potential tests.
Another factor related to the measurement of learning relates to the choice of predictors of learning.
Learning is complex. While the science of learning has done a commendable job of exploring the
different predictors of learning, more work is needed to look for moderators (Gurung & Hackathorn,
2019). Studies often test whether an intervention (e.g., study training) or classroom feature (e.g.,
providing slides) is associated with more learning, but not enough studies test how specific student
characteristics, especially differences in personality, may influence the findings. As demonstrated by
research in educational psychology, factors such as effort, ability, study skills, habits, self-efficacy,
motivation, academic goals, contextual influences, and perceived social support all have a powerful
influence on learning (Komarraju & Nadler, 2013). Similarly, many cognitive science designs include
individual difference measures, but these are often constrained to differences in memory abilities.
Specifically, although there are lots of such findings of individual differences predicting learning
outcomes in correlational studies, more work using individual difference variables as moderators in
studies with manipulated independent variables (where interaction effects are examined) is needed.
Although, in relative terms, there are few studies that examine the interaction between individual
difference variables and manipulated instructional variables, the number of such studies has been
increasing in recent years. The utility of doing so is already clear. In a study of learning during the
pandemic, students’ self-efficacy was a significant predictor of learning (Gurung & Stone, 2021). It is
likely the inclusion of such individual differences in a broader range of studies of learning will allow the
prediction of a greater share of variance.
One major area for the science of learning and pedagogical research to move on to is the testing of
different models to teach a course. Traditionally, this type of comparison has been limited to course
modality (e.g., face to face, hybrid, or online) or the effectiveness of blended models (e.g., Hudson, et
al., 2015). There are new and different models to covering content and developing skills (Jhangiani, &
Hardin, 2015). One way is to focus on major themes in a course and then modify content accordingly
(Hard et al. 2021). Another way may be to design a course around debunking myths of the field
(Bernstein, 2018). Both of these ways involve cutting down on how many topics are covered. But is one
way better than the other? While at a more macro level and involving many more moving parts, there
may be course models and designs that foster great learning. Currently, the science of learning focuses
on a more micro level, an assignment, assessment, of class session; moving the camera back, as it were
may, be a prudent future direction, and an important one.
67
Concluding Thoughts
SoTL has budded into many different avenues of research over the last 30 years, and the term has taken
on different meanings by different groups. On the upside, more and more faculty, educators, and
researchers are taking a close look at what is associated with better learning. There are significant
methodological challenges to doing SoTL but the general enterprise of aiming to understand how we
learn, and to find the factors that can increase learning (whether personal, situational, institutional, or
interactions among them), is alive and well. Administrators and faculty developers, in particular, should
note that SoTL is not exactly what Boyer (1990) originally intended when he hoped to celebrate the hard
work that is teaching. Even if SoTL does not mean what everybody thinks it means, it still means a lot.
The endeavors of the related areas of the science of learning, implementation science, and disciplinary
based education research (to name but the most commonly used terms), and both efficacy and
effectiveness research, all promise a higher quality of education if we all stay the course.
Author Note
Portions of this chapter were inspired and driven by the preparation of the author’s 2021 APS-David
Myers Distinguished lecture on the science and craft of teaching psychological science. Address
correspondence to [email protected] and follow the author on Twitter
@ReganARGurung.
References
Bernstein, D. A. (2017). Bye-bye intro: A proposal for transforming introductory psychology. Scholarship
of Teaching and Learning in Psychology, 3(3), 191–197. https://doi.org/10.1037/stl0000093
Bernstein, D., Addison, W., Altman, C., Hollister, D., Meera K., Prieto, L. R., Rocheleau, C. A., & Shore, C.
(2009). Toward a scientist-educator model of teaching psychology. In D. F. Halpern (Ed.), The
NCUEP: A blueprint for the future (pp.29-45). American Psychological
Association. https://doi.org/10.1037/12063-002
Boyer, E. (1990). Scholarship reconsidered: Priorities of the professoriate. Carnegie Foundation for the
Advancement of Teaching.
Brookfield, S. D. (2006). The skillful teacher: On technique, trust, and responsiveness in the
classroom (2nd ed.). Jossey-Bass.
Brown-Kramer, C. R. (2021). Improving students’ study habits and course performance with a “learning
how to learn” assignment. Teaching of Psychology, 48(1), 48–54.
Brown-Kramer, C. R. (2022). Teaching students to learn better in introductory psychology: A replication
and extension study. Teaching of Psychology, 49(2), 108–117.
Chew, S. L., Bartlett, R. M., Dobbins, J. E., Hammer, E. Y., Kite, M. E., Loop, T. F., Mclntyre, J. G., & Rose,
K. C. (2010). A contextual approach to teaching: Bridging methods, goals, and outcomes. In D. F.
Halpern (Ed.), Undergraduate education in psychology: A blueprint for the future of the
discipline (pp. 95–112). American Psychological Association. https://doi.org/10.1037/12063-006
68
Chick, N. L. (2014). ‘Methodologically sound' Under the ‘big tent': An ongoing
conversation. International Journal for the Scholarship of Teaching and Learning, 8(2), Article 1.
https://doi.org/10.20429/ijsotl.2014.080201
Daniel, D. B., & Chew, S. L. (2013). The tribalism of teaching and learning. Teaching of Psychology, 40(4),
363–367. https://doi.org/10.1177/0098628313501034
Diamond, R. M., & Adam, B. E. (2000). The disciplines speak II: More statements on rewarding the
scholarly, professional, and creative work of faculty. Washington, DC: American Association for
Higher Education.
DiGregorio, N., Maurer, T. W., & Pattanaik, S. (2016). Documentation of scholarship of teaching and
learning trends at National Council on Family Relations Annual Conferences. Family Science
Review, 21(2), 54-73. https://doi.org/10.26536/FSR.2016.21.02.03
Donald, J. G. (2002). Learning to think: Disciplinary perspectives. Jossey-Bass.
Educational Testing Service. (2023a). ETS Major Field Test for Psychology. https://origin-
www.ets.org/mft/about/content/psychology
Educational Testing Service. (2023b). GRE Subject Test Psychology. https://www.ets.org/gre/test-
takers/subject-tests/about/content-structure.html#accordion-792530b110-item-c2a51a1d4e
Felten, P. (2013). Principles of good practice in SoTL. Teaching & Learning Inquiry, 1, 121-125.
https://doi.org/10.2979/teachlearninqu.1.1.121
Fyfe, E. R., de Leeuw, J. R., Carvalho, P. F., Goldstone, R. L., Sherman, J., Admiraal, D., Alford, L. K.,
Bonner, A., Brassil, C. E., Brooks, C. A., Carbonetto, T., Chang, S. H., Cruz, L., Czymoniewicz-
Klippel, M., Daniel, F., Driessen, M., Habashy, N., Hanson-Bradley, C. L., Hirt, E. R., . . . Motz, B. A.
(2021). ManyClasses 1: Assessing the generalizable effect of immediate feedback versus Delayed
feedback across many college classes. Advances in Methods and Practices in Psychological
Science, 4(3). https://doi.org/10.1177/25152459211027575
Glassick, C. E., Huber, M. T., & Maeroft, G. I. (1997). Scholarship assessed: Evaluation of the
professoriate. Jossey-Bass.
Grant, N. K., & McGrath, A. L. (2021). Effects of PowerPoint slides on attendance and learning: If you
share it, they will (still) come. Scholarship of Teaching and Learning in Psychology. Advance
online publication. https://doi.org/10.1037/stl0000241
Gurung, R. A. R. (2014). Getting foxy: Invoking different magesteria in the Scholarship of Teaching and
Learning. Teaching and Learning Inquiry, 2(2), 109 -114.
Gurung, R. A. R. (2017). Predicting learning: Comparing an open educational resource and standard
textbooks. Scholarship of Teaching and Learning in Psychology, 3(3), 233–
Gurung, R. R., Ansburg, P. I., Alexander, P. A., Lawrence, N. K., & Johnson, D. E. (2008). The state of the
scholarship of teaching and learning in psychology. Teaching of Psychology, 35(4), 249-261.
Gurung, R. A. R. & Burns, K. C, (2018). Putting evidence-based claims to the test: A multi-
site classroom study of retrieval practice and spaced practice. Applied Cognitive
Psychology. 33(5). https://doi.org/10.1002/acp.3507
69
Gurung, R. A. R., Chick, N., & Haynie, A. (Eds.) (2009). Exploring signature pedagogies: Approaches to
teaching disciplinary habits of mind. Stylus.
Gurung, R. A. R., Daniel, D. B., & Landrum, R. E. (2012). A multi-site study of learning: A focus on
metacognition and study behaviors. Teaching of Psychology, 39(3), 170-
Gurung, R. A. R., & Hackathorn, J. (2018). Ramp it up: A call for more research in introductory
psychology. Teaching of Psychology, 45(4), 302–311. https://doi-
org.ezproxy.proxy.library.oregonstate.edu/10.1177/0098628318796413
Gurung, R. A. R., & Landrum, R. E. (2103). Bottleneck concepts in psychology. Psychology of Learning and
Teaching, 12(3), 236-245. https://doi.org/10.2304/plat.2013.12.3.236
Gurung, R. A. R., & Landrum, R. E. (2015). Editorial. Scholarship of Teaching and Learning in
Psychology, 1(1), 1-6. https://doi.org/10.1037/stl0000026
Gurung, R. A. R. and Schwartz, B. M. (2010). Riding the third wave of SoTL, International Journal for the
Scholarship of Teaching and Learning: 4(2, article 5), 1-6.
https://doi.org/10.20429/ijsotl.2010.040205
Gurung, R. A. R., Richmond, A., Drouin, M., Landrum, R. E., & Christopher, A. N. (2019). The past,
present, and future of scholarship of teaching and learning in psychology. Scholarship of
Teaching and Learning in Psychology, 5(2), 97–120. https://doi.org/10.1037/stl0000143
Gurung, R. A. R., & Stone, A. M. (2020). You can’t always get what you want and it hurts: Learning during
the pandemic. Scholarship of Teaching and Learning in Psychology. Advance online publication.
https://doi.org/10.1037/stl0000236
Hake, R. R. (2015). What might psychologists learn from Scholarship of Teaching and Learning in
physics? Scholarship of Teaching and Learning in Psychology, 1(1), 100–
Halonen, J. S., Thompson, J. L. W., Whitlock, J., Landrum R. E., & Frantz, S. (2022). Measuring meaning
learning in introductory psychology: The IPI student learning outcomes. In R. A. R. Gurung & G.
Neufeld (Eds.) Transforming introductory psychology: Expert advice on teaching, training, and
assessing the course. American Psychological Association.
Halpern, D. F., Smothergill, D., Allen, M., Baker, S., Baum, C., Best, D., Ferrari, J., . . . & Weaver, K. A.
(1998). Scholarship in psychology: A paradigm for the twenty-first century. American
Psychologist, 53, 1292–1297. https://doi.org/10.1037/0003-066X.53.12.1292
Hard, B. M., Lovett, J. M., & Brady, S. T. (2019). What do students remember about introductory
psychology, years later? Scholarship of Teaching and Learning in Psychology, 5(1), 61–
Hattie, J., & Yates, G. (2014). Visible learning and the science of how we learn. Routledge.
Huber, M. T., & Hutchings, P. (2005). The advancement of learning: Building the teaching
commons. Jossey-Bass.
Hudson, D. L., Whisenhunt, B. L., Shoptaugh, C. F., Visio, M. E., Cathey, C., & Rost, A. D. (2015). Change
takes time: Understanding and responding to culture change in course redesign. Scholarship of
70
Hutchings, P., Huber, M. T., & Ciccone, A. (2011). The scholarship of teaching and learning reconsidered:
Institutional integration and impact. Jossey-Bass.
Hutchings, P., & Shulman, L. (1999). The scholarship of teaching new elaborations, new developments.
Change, 31(5), 11-15. https://doi.org/10.1080/00091389909604218
James, W. (1899). Talks to teachers on psychology--and to students on some of life's ideals. Metropolitan
Books/Henry Holt and Company.
Jhangiani, R. S., & Hardin, E. E. (2015). Skill development in introductory psychology. Scholarship of
Joosten, T. (2020). Learning science research through a social scientific lens. In R. S. Feldman (Ed.),
Learning Science: Theory, Research, & Practice (pp. 75-103). McGraw-Hill.
Komarraju, M., & Nadler, D. (2013). Self-efficacy and academic achievement: Why do implicit beliefs,
goals, and effort regulation matter? Learning and Individual Differences, 25, 67–
72. https://doi.org/10.1016/j.lindif.2013.01.005
Landrum, R. E., & Gurung, R. A. R. (2013). The memorability of introductory psychology
revisited. Teaching of Psychology, 40(3), 222–227. https://doi.org/10.1177/0098628313487417
Lawrence, N. K. (2014). Examining the benefits of cumulative exams. In V. A. Benassi, C. E. Overson, & C.
M. Hakala (Eds.), Applying science of learning in education: Infusing psychological science into
the curriculum (pp. 274–278). Society for the Teaching of Psychology.
Matthews, K. E., Lodge, J. M., & Bosanquet, A. (2014). Early career academic perceptions, attitudes and
professional development activities: questioning the teaching and research gap to further
academic development. International Journal for Academic Development, 19(2), 112-
124. https://doi.org/10.1080/1360144X.2012.724421
McCarthy, M. A., & Frantz, S. (2016). Challenging the status quo: Evidence that introductory psychology
can dispel myths. Teaching of Psychology, 43(3), 211–214.
McKinney, K., Atkinson, M., & Flockhart, T. (2017). A sampling of what psychologists engaged
in SoTL might learn from sociology. Scholarship of Teaching and Learning in Psychology, 3, 178-
Meyer, J.H.F. and Land, R. (2005), Threshold concepts and troublesome knowledge (2): Epistemological
considerations and a conceptual framework for teaching and learning, Higher Education, 49(3),
373−388. https://doi.org/10.1007/s10734-004-6779-5
O’Meara, K., & Rice, R. E. (2005). Faculty priorities reconsidered: Rewarding multiple forms of
scholarship (pp. 17–31). Jossey-Bass.
Pace, D., & Middendorf, J. (2004). Decoding the Disciplines: Helping Students Learn Disciplinary Ways of
Thinking. Jossey-Bass.
Richlin, L. (2001). Scholarly teaching and the scholarship of teaching. New Directions for Teaching and
Learning, 86, 57-68. https://doi.org/10.1002/tl.16
Secret, M., Leisey, M., Lanning, S., Polich, S., & Schaub, J. (2012). Faculty perceptions of the scholarship
of teaching and learning: Definition, activity level and merit considerations at one
university. Journal of the Scholarship of Teaching and Learning, 11(3), 1-20.
71
Soicher, R. N. (2020). Implementation science in teaching and learning in higher education: A mixed-
methods study of the utility value intervention in general psychology courses. Unpublished
Dissertation, Oregon State University, Corvallis, OR.
Soicher, R. N., Becker-Blease, K. A., & Bostick, K. C. P. (2020). Adapting implementation science for
higher education research: The systematic study of implementing evidence-based practices in
college classrooms. Cognitive Research, 5(54). 1-15. https://doi.org/10.1186/s41235-020-00255-
0
Solomon, E. D., Bugg, J. M., Rowell, S. F., McDaniel, M. A., Frey, R. F., & Mattson, P. S. (2021).
Development and validation of an introductory psychology knowledge inventory. Scholarship of
Shulman, L. S. (Summer 2005). Signature pedagogies in the professions. Dædalus, 134(8). pp. 52–59.
Wilson-Doenges, G., & Gurung, R. A. R. (2013). Benchmarks for scholarly investigations of teaching and
learning. Australian Journal of Psychology, 65(1), 63-70.
Wilson-Doenges, G., Troisi, J. D., & Bartsch, R. A. (2016). Exemplars of the gold standard in SoTL for
psychology. Scholarship of Teaching and Learning in Psychology, 2(1), 1–12.
72
In Their Own Words
Part 2
Science of Learning: Principles
and Approaches
Editors
Lauren L. Kordonowy
Victor A. Benassi

73
In Their Own Words The Prequestion Effect: Why It Is Useful to Ask
Students Questions Before They Learn
Shana K. Carpenter
Editors
Iowa State University
Lauren L. Kordonowy
Victor A. Benassi
Quentin King-Shepard and Timothy Nokes-Malach
University of Pittsburgh
Introduction and Overview

Research in cognitive science has brought to light a number of findings that can enhance education. In
particular, studies have revealed specific techniques that significantly enhance students’ learning in a
number of domains. These techniques can be implemented in a variety of course settings, making them
useful tools that students and teachers can incorporate into many different types of lessons.
One of the most powerful techniques for enhancing learning is to ask students questions about what
they are learning. After students are introduced to some new information—for example, through a class
lesson or reading assignment—their learning of that information is significantly boosted by answering
questions that require them to retrieve the information. In one study, when sixth grade students
reviewed information from their social studies lessons by either completing short practice quizzes or by
simply rereading the information from the lessons, they scored significantly higher on later exams when
they had reviewed through quizzing (Roediger et al., 2011, Experiment 2). The benefits of retrieval
practice have been demonstrated in numerous other studies (see Yang et al., this volume), showcasing a
simple but highly effective technique for improving learning and retention.
It makes sense to give students practice questions after they have encountered a lesson. What about
giving students questions before a lesson? This could be a useful approach for teachers to determine
what students already know about a topic, which can help them plan lessons accordingly. A less well-
known fact, however, is that these questions can actually enhance learning of the lesson that follows.
Even if students have not yet learned the information and are merely guessing the answers, engaging
with these prequestions enables them to better learn the information from a later lesson that is relevant
to those questions.
In the following sections, we review the research findings on prequestions and their implications for
education. We begin with an overview of the typical approach to studying prequestions. We then
include a discussion of the research findings illustrating ways in which prequestions can benefit learning,
and why these benefits occur. We conclude with recommendations for teachers about how they can use
prequestions in their courses to boost student learning.
“Pretrieval Practice”: The Benefits of Prequestions

Figure 1 illustrates the design of a typical study on prequestions. Here students are given a learning
opportunity in the form of a reading assignment, video, or class lecture. Prior to that learning
opportunity, students are given questions on information that will appear in the upcoming reading,
video, or lecture. Even if they do not know the answers because they have not yet learned the material,
74
students are encouraged to offer their best attempt at answering the questions. A group of students
that answers prequestions (the Prequestion Group) is most often compared to a group that does not
answer prequestions and does not receive any other kind of preview of the lesson (the Control Group).
Corrective feedback is typically not provided at the time of the prequestions, but instead students in the
Prequestion Group proceed with the learning opportunity right after answering the prequestions.
Figure 1
Overview of the Design of a Typical Study on Prequestions.
After the learning opportunity, students are given a post-test to assess what they have learned. For the
Control Group, none of these post-test questions have been seen before. For the Prequestion Group,
some of the questions have been seen before as prequestions (a, b, and c), and some of the questions
have not been seen before (d, e, and f). For the Prequestion Group, this permits a comparison of post-
test performance for “prequestioned information” (information from the lesson that was directly
addressed ahead of time in the prequestions) versus “non-prequestioned information” (information
from the lesson that was not addressed in the prequestions). For the Control Group, because there were
no prequestions, all of the post-test questions naturally represent non-prequestioned information.
In a prequestion study, three key comparisons are typically made. First, what is the overall score on the
post-test for the Prequestion Group compared to the Control Group? This gives us an idea of the overall
benefit of prequestions for enhancing learning. Second, what is the Prequestion Group’s score on the
post-test specifically for prequestioned information, compared to the Control Group’s overall post-test
score on the entire set of questions? This gives us a closer look at the benefits of prequestions by
focusing on learning of the information from the lesson that was specifically addressed in the
prequestions. If prequestions enhance learning of information that was specifically addressed in those
prequestions, this can be referred to as the “specific benefit” of prequestions. Finally, what is the
Prequestion Group’s score on the post-test for non-prequestioned information, compared to the Control
Group’s overall post-test score on the entire set of questions? This gives us an idea of whether the
benefits of prequestions generalize to information from the lesson that was not addressed in the
prequestions. As such, this can be referred to as the “general benefit” of prequestions. As described
below, the research on prequestions sometimes shows specific benefits, sometimes general benefits,
and sometimes both.
75
Research on Prequestions
Learning From Reading Materials
Some of the earliest research on prequestions explored their effects on learning and comprehension of
reading material. These studies found that prequestions often lead to a specific learning benefit. For
example, in a study conducted by Pressley et al. (1990), students were instructed to read a 30-page
chapter on adult development in the family. Before reading the chapter, students were assigned to
either answer prequestions, read prequestions without trying to answer them, or to a control group in
which they received the chapter but did not receive any prequestions. On the post-test, students who
answered prequestions performed significantly better than the other two groups. However, this benefit
was specific to the prequestioned information. There were no differences in post-test scores between
the three groups for the non-prequestioned information. Further, there were no differences in post-test
scores between the control group and the group that simply read the prequestions without attempting
to answer them. This suggests that attempting to answer the prequestions is a critical ingredient for
producing the learning benefits of prequestions.
Richland et al. (2009) also investigated the effects of prequestions on comprehension and retention of
reading material. Students in this study were instructed to read a two-page text passage about the visual
system. The prequestion group received questions about the text before reading, and the control group
did not receive prequestions but instead received extra time to read the text. Through a series of five
experiments, the prequestioned group consistently outperformed the control group on the post-test.
Similar to the findings of Pressley et. al (1990), the students who received prequestions showed a
specific learning benefit for the prequestioned information but did not show a general learning benefit
for the non-prequestioned information. The specific benefit of prequestions occurred when the post-
test was given immediately after reading the passage (Experiments 1-3, 5), and also when the post-test
was given one week after reading the passage (Experiment 4).
Many additional recent studies have explored the effects of prequestions on learning reading material
(Hausman & Rhodes, 2018; James & Storm, 2019 see Experiments 1-4; Pan & Sana, 2021). The majority
of these studies show that prequestions produce specific benefits on learning of prequestioned
information; however, a few studies also show that prequestions produce general benefits on non-
prequestioned information as well (e.g., Pan & Sana, 2021). Although the majority of studies on
prequestions have examined learning from reading material, this area is only a subset, with recent work
also exploring the effects of prequestions on other learning materials such as videos and class lectures.
Learning From Video Materials
Prequestions can benefit learning of information from video-based materials as well. Carpenter and
Toftness (2017) had students view a brief video presentation on the history of Easter Island. The
prequestion group answered two prequestions prior to each of three segments of the video, whereas
the control group viewed the same video segments without answering any prequestions. On the post-
test right after the video, the prequestion group significantly outperformed the control group, showing
the usual overall advantage of prequestions. The advantage of the prequestion group was particularly
strong for prequestioned information but was also present (albeit less so) for non-prequestioned
information. Thus, this study demonstrated both specific and general benefits of prequestions on
learning from short videos.
Along similar lines, James and Storm (2019, Experiment 5) had students view brief educational videos
about ancient life on earth. Students answered prequestions before viewing some of the videos, and
other videos they simply viewed without answering any prequestions first. A post-test over all of the
76
videos revealed a significant specific benefit of prequestions, but no general benefit. Pan et al. (2020)
had students view an online video-recorded lecture on signal detection theory in which some students
answered prequestions beforehand and other students did not. Answering prequestions led to a
significant overall benefit on the post-test. In addition, answering prequestions produced a specific
benefit on the learning of prequestioned information, as well as a general benefit (although smaller than
the specific benefit) on non-prequestioned information.
Other studies have found that prequestions significantly benefit students’ learning of information from
video-recorded lectures about psychology concepts (Toftness et al., 2018) and information theory (St.
Hilaire & Carpenter, 2020). The specific benefits of prequestions consistently occurred across these
studies, and the general benefits sometimes occurred but were less consistent than the specific
benefits.
Classroom-Based Research
Studies conducted in real classrooms suggest that prequestions can be an effective tool for boosting
students’ learning of their course information. In one of the earliest classroom studies, Swenson and
Kulhavy (1974) had fifth- and sixth-grade students read a text about the history and culture of Ako.
Some students answered prequestions before reading sections of the text, and other students simply
read the text without answering any prequestions. On a post-test given both immediately and one week
later, students who answered prequestions significantly outperformed students who simply read the
text. Answering prequestions produced a strong specific benefit on prequestioned information, but no
general benefit on non-prequestioned information.
More recently, Carpenter et al. (2018) explored the effects of prequestions in an undergraduate
psychology course. Some students answered prequestions immediately before each day’s lesson, and
other students did not. At the end of each day’s lesson, all students completed a post-test over the
information from that lesson. A specific benefit of prequestions occurred in that students who answered
prequestions scored higher on the post-test over prequestioned information. However, no general
benefit of prequestions occurred. That is, answering prequestions at the beginning of class did not boost
learning of non-prequestioned information, relative to a control group of students who received the
same lesson but did not answer prequestions beforehand.
Studies conducted in real classrooms can be subject to scheduling constraints and other factors inherent
to the course that sometimes affect the comparisons that can be made. For example, some studies
incorporated prequestions as part of the normal course routine for all students in a middle school
science course (McDaniel et al., 2011) and an undergraduate chemical engineering course (Geller et al.,
2017). The lack of a true control group (for whom prequestions were withheld) means that direct
comparisons of the effects of prequestions relative to no prequestions are not available from these
studies. However, both of these studies showed significant improvements from students’ performance
on the prequestions (asked at the beginning of class) to the same questions asked again at the end of
class or at the end of a lesson unit, as well as to new never-before-seen questions from the same
lessons. McDaniel et al. (2011) further explored the frequency of practice questions over the course of
an entire academic term, and they found that the more often students encountered a practice question
over a given concept (e.g., as a prequestion before class, then again as a question at the end of the unit,
then again later as a practice question on a review for the exam, etc.), the better students learned that
particular concept.
de Lima and Jaeger (2020) explored the specific benefits of prequestions on fourth- and fifth-grade
students’ learning of a science text passage about rockets. Students first read the text with some of the
information omitted that they had to fill in (e.g., “A revolutionary e_____ has been developed”), and
77
other information intact (e.g., “The Americans sent a rocket called Apollo 11 to the moon”). Afterward
they read the same text again with all of the information intact (including the earlier phrase that they
had to fill in, “A revolutionary engine has been developed”). On a post-test a week later, students scored
significantly higher for the information that they originally had to fill in, compared to the intact
information that they simply read twice. Filling in the omitted information also led students to be more
confident in their learning at the time of the post-test. Although this study does not include measures of
the general benefit of prequestions (due to the fact that all students learned the text passage using both
approaches), it demonstrates the specific benefits of using a method that is well-suited for young
students and appears to increase confidence in learning as well.
In summary, the research on prequestions shows consistent and reliable specific benefits on learning.
Most often, prequestions enhance learning of the information that is directly addressed in the
prequestions. Some studies show that prequestions can also produce more general benefits on the
learning of non-prequestioned information, but this finding is less consistent. There does not appear to
be any detriment to using prequestions.
Why Do Prequestions Benefit Learning?

One proposition is that prequestions are beneficial because they increase students’ engagement with
the learning material. When students receive questions prior to a learning opportunity, those questions
might pique their interest in the information they are about to learn. Students may, as a result, invest
more attention in the learning material and in finding out the answers to the prequestions.
Indeed, a common challenge encountered by students is the need to sustain attention to a learning task.
Even inadvertently, attention often shifts to other things while reading or viewing a lecture. This
tendency is ever-present, with some studies estimating that about one third to one half of students
experience mind wandering during a lecture (Lindquist & McLean, 2011). Fortunately, there is evidence
that prequestions can help with this. One recent study found that students who received prequestions
before a video lecture (compared to students who did not receive prequestions) reported lower rates of
mind wandering during the lecture, and less mind wandering led to significantly better learning of the
lecture material (Pan et al., 2020).
One study used eye-tracking technology to explore where students focus their attention in a reading
passage after receiving prequestions. Lewis and Mensink (2012) had students read a text about
humanitarian efforts of the Gates Foundation. One group received prequestions before reading (e.g.,
“What causes Tuberculosis to spread through the air?”), and the other group simply read the passage
without seeing any prequestions beforehand. While reading, the eye movements of each student were
recorded using a head-mounted eye-tracking device. Students’ eye movements revealed a specific
pattern in which students who received prequestions (compared to students who did not) focused their
attention more on the prequestioned information in the reading. That is, if students received
prequestions before reading, they spent a longer amount of time re-reading parts of sentences that
were directly relevant to those prequestions (e.g., “Tuberculosis is spread through the air when people
who have the disease cough, sneeze, or spit”), and they also spent a longer amount of time looking back
to these relevant sentences after they had moved on to the subsequent parts of the reading passage.
However, for sentences in the reading that were not directly relevant to the prequestions, no
differences in re-reading time occurred for students who received prequestions compared to students
who did not. On an essay test in which students were asked to recall everything they could remember
from the reading passage, the group that received prequestions recalled about twice as much of the
prequestioned information as the control group, but they did not recall more of the non-prequestioned
78
information. Thus, the specific benefits of prequestions occurred (but not the general benefits), and
these specific benefits appear to be linked with where students focused their attention during reading.
Using a different approach, St. Hilaire and Carpenter (2020) also explored the role of attention in
learning from prequestions. Before viewing a video lecture on information theory, one group of
students was given prequestions and the other group was not. All students were then asked to take
notes during the video over any information that they felt would be important to learn. If students had
received prequestions, they were significantly more likely to write down prequestioned information in
their notes while viewing the video—that is, information from the video that was relevant to answering
the prequestions—but they were not more likely to write down non-prequestioned information.
Performance on the post-test revealed a specific benefit of prequestions, such that students who
received prequestions outperformed students who did not, but only on the prequestioned information
and not on the non-prequestioned information. Furthermore, the advantage of the prequestion group
over the control group only occurred when students successfully included the prequestioned
information in their notes. If students who received prequestions happened to not include in their notes
a piece of information from the video that was relevant to a prequestion, they did not remember this
particular piece of information any better than the control group on the post-test. Thus, prequestions
increase the chances that students will notice the prequestioned information, but not to a perfect
degree. As in all learning experiences, some information might fall outside the window of attention, and
if it does, this information would not be learned as well as information that receives full attention.
Thus, prequestions likely benefit learning through focusing students’ attention to parts of the learning
material that are directly relevant to those prequestions. This focused attention appears to coincide
directly with later memory, in that the prequestioned information that is made more noticeable is also
more successfully remembered later on. Such a pattern likely explains the common tendency for
prequestions to have specific benefits, but not always general benefits, on learning. Future research
should explore additional ways that prequestions might benefit learning, such as the potential impact of
prequestions on students’ motivation and goals.
What Kind of Benefits Can Be Expected from Prequestions?

Prequestions work best for enhancing learning of information that is directly presented during a learning
opportunity. Given that prequestions help students focus attention on prequestioned information, it is
important that the prequestioned information be clearly noticeable so that students can attend to it and
readily see the relevance of this information for answering the prequestions. Thus, prequestions are
likely best suited for learning facts and concepts that are explicitly stated in a lesson, rather than
information that is not stated but must be inferred.
Indeed, one study found that prequestions did not benefit learning when the prequestions required
students to use the information in a reading passage to infer answers that were not directly stated.
Hausman and Rhodes (2018) had students read a text passage about glaciers, either receiving
prequestions beforehand or not. Factual prequestions were those for which the answers were directly
stated in the reading (e.g., “How much of the earth is covered by glaciers during an ice age?”), whereas
conceptual prequestions were those for which the answers were not directly stated (e.g., “What is a
consequence of higher levels of carbon dioxide in the atmosphere?”) but could be inferred from various
other information that was stated throughout the text. For example, the text directly stated that higher
levels of carbon dioxide led to warmer temperatures and that cooler temperatures lead to lower sea
levels due to the formation of glaciers that hold frozen water. Together this information could be used
to infer the answer that higher sea levels are a consequence of higher levels of carbon dioxide in the
atmosphere. On a post-test given after the reading, the students who received factual prequestions
79
showed the usual specific benefit of prequestions, in that they scored significantly higher than the
control group on prequestioned information but not on non-prequestioned information. In contrast, the
students who received conceptual prequestions showed no advantage at all over the control group.
Thus, prequestions are beneficial when applied to material in the lesson that can be directly noticed by
students. As we have seen, these benefits often tend to be specific to the prequestioned information.
However, under some circumstances prequestions can produce general benefits as well. Little and Bjork
(2016) had students read a text about geysers, either receiving multiple-choice prequestions ahead of
time (e.g., “What is the tallest geyser in Yellowstone National Park? (a) Old Faithful, (b) Steamboat
Geyser, (c) Castle Geyser, (d) Daisy Geyser”) or no prequestions. Although only one answer is correct
(Steamboat), information about each of the other geysers was included in the reading as well. Post-test
performance showed that students who received prequestions (compared to the control group) not only
performed better on those same questions again, but they also performed better on never-before-seen
questions about the other geysers (e.g., a question about Castle Geyser or Daisy Geyser). By requiring
students to consider each answer option during the reading, these multiple-choice prequestions served
to enhance attention not only to the specific correct answer in the prequestion, but to the other answer
options as well, resulting in both specific and general benefits. Along similar lines, St. Hilaire et al. (2019)
found both specific and general benefits of prequestions when the prequestions were designed to
enhance attention to multiple parts of a reading passage about brakes (e.g., “What is the primary
difference between mechanical brakes and hydraulic brakes?”), compared to prequestions that targeted
a more isolated fact within the reading (e.g., “What type of machine relies on caliper brakes?”). If
prequestions are designed in a way that encourages students to focus their attention beyond the
prequestioned information, then they can produce both specific and general benefits.
Prequestions can be applied to a variety of learning situations. They can benefit students’ learning of
reading material, video presentations, and class lectures. Studies exploring prequestions have also
utilized diverse learning material, including readings or lectures over scientific topics (de Lima & Jaeger,
2020; Hausman & Rhodes, 2018; McDaniel et al., 2011; Richland et al., 2009), history (Carpenter &
Toftness, 2017; James & Storm, 2019), geography (Little & Bjork, 2016), signal detection theory (Pan et
al., 2020; Toftness et al., 2018), engineering (Geller et al., 2017), and psychology (Carpenter et al., 2018).
The benefits of prequestions occur across educational levels, from elementary school children to college
students. Researchers have also observed benefits for different types of prequestions, including
multiple-choice, fill-in-the-blank, and short answer. Prequestions are thus a flexible learning tool that
can be adapted to a number of different lessons and contexts.
Recommendations for Teachers

Prequestions are a straightforward tool that teachers can utilize in a number of ways. The first step is to
create questions over the learning material that is in a given lesson. Such questions can pertain to fairly
discrete facts (e.g., “In what year was the U. S. Declaration of Independence drafted?”), or to more
complex information (e.g., “How does a neurotransmitter enable communication between neurons?”),
so long as the information needed to answer the questions is directly provided in the lesson.
Teachers can then provide these questions to students right before the lesson. Because many students
may not have full knowledge of the answers at this point, the prequestions can be used as a low-stakes
pre-learning activity that encourages students to think about what the answers to the prequestions may
be but does not require them to answer the questions correctly. Knowing that the answers will be
provided in the upcoming lesson, students are then prepared to direct their attention to the lesson, and
in particular to the information that is relevant to the prequestions. Although no direct feedback of the
answers need be provided at the time of the prequestions, teachers may find it useful to re-visit these
80
questions again at the end of the lesson and provide feedback at that time. This helps ensure that
students have fully grasped the concepts and provides an opportunity to resolve any potential
confusions or misconceptions.
The most straightforward way to incorporate prequestions likely depends on how a lesson is delivered.
For an in-class learning opportunity, prior to immersing in the activities teachers can introduce the topic
and present the prequestions at that time. Exactly how the prequestions are presented can vary. For
example, teachers could display the prequestions in a presentation, or write them on a whiteboard and
ask students to write down their answers. Teachers could also distribute sheets of paper with the
prequestions printed on them and ask students to work on these questions at the beginning of class. If
resources are available, teachers could also use “clicker” devices to present the prequestions and collect
students’ responses electronically.
Other times, a lesson could take the form of a reading assignment or a video that students complete
outside of class. In these cases, prequestions could be assigned ahead of time to be completed just prior
to the reading assignment or video. If the reading or video is available in an online course management
system, teachers could include the prequestions in the form of a short online quiz that students
complete before opening the link to the reading or video.
In summary, prequestions can be incorporated as learning activities in a number of different ways. The
exact way that prequestions are delivered can be flexible and could include either high-tech options
(online quizzes or clicker questions) or low-tech options (on the whiteboard or on paper). Whatever the
method that teachers use, “pretrieval” practice is a simple and effective way to engage students in what
they are about to learn.
Author Note
This material is based upon work supported by the James S. McDonnell Foundation 21st Century Science
Initiative in Understanding Human Cognition, Collaborative Grant No. 220020483.
Address correspondence to Shana K. Carpenter, Department of Psychology, Iowa State University, W112
Lagomarcino Hall, 901 Stange Road, Ames, IA 50011, USA. Phone: (515) 294-6385, E-mail:
[email protected]
References
Carpenter, S. K., Rahman, S., & Perkins, K. (2018). The effects of prequestions on classroom learning.
Journal of Experimental Psychology: Applied, 24(1), 34-42. https://doi.org/10.1037/xap0000145
Carpenter, S. K., & Toftness, A. R. (2017). The effect of prequestions on learning from video
presentations. Journal of Applied Research in Memory and Cognition, 6(1), 104-109.
de Lima, N. K., & Jaeger, A. (2020). The effects of prequestions versus postquestions on memory
retention in children. Journal of Applied Research in Memory & Cognition, 9(4), 555-563.
Geller, J., Carpenter, S. K., Lamm, M. H., Rahman, S., Armstrong, P. I., & Coffman, C. R. (2017).
Prequestions do not enhance the benefits of retrieval in a STEM classroom. Cognitive Research:
Principles and Implications, 2(1), 42. https://doi.org/10.1186/s41235-017-0078-z
81
Hausman, H., & Rhodes, M. G. (2018). When pretesting fails to enhance learning concepts from reading
texts. Journal of Experimental Psychology: Applied, 24(3), 331-346.
https://doi.org/10.1037/xap0000160
James, K. K., & Storm, B. C. (2019). Beyond the pretesting effect: What happens to the information that
is not pretested? Journal of Experimental Psychology: Applied, 25(4), 576-587.
Lewis, M. R., & Mensink, M. C. (2012). Prereading questions and online text comprehension. Discourse
Processes, 49(5), 367-390. https://doi.org/10.1080/0163853X.2012.662801
Lindquist, S. I., & McLean, J. P. (2011). Daydreaming and its correlates in an educational environment.
Learning and Individual Differences, 21(2), 158-167. https://doi.org/10.1016/j.lindif.2010.12.006
Little, J. L., & Bjork, E. L. (2016). Multiple-choice pretesting potentiates learning of related information.
Memory & Cognition, 44(7), 1085-1101. https://doi.org/10.3758/s13421-016-0621-z
McDaniel, M. A., Agarwal, P. K., Huelser, B. J., McDermott, K. B., & Roediger, H. L. III. (2011). Test-
enhanced learning in a middle school science classroom: The effects of quiz frequency and
placement. Journal of Educational Psychology, 103(2), 399-414.
Pan, S. C., & Sana, F. (2021). Pretesting versus posttesting: Comparing the pedagogical benefits of
errorful generation and retrieval practice. Journal of Experimental Psychology: Applied, 27(2),
237-257. https://doi.org/10.1037/xap0000345
Pan, S. C., Schmitt, A. G., & Bjork, E. L., & Sana, F. (2020). Pretesting reduces mind wandering and
enhances learning during online lectures. Journal of Applied Research in Memory and Cognition,
9(4), 542-554. https://doi.org/10.1016/j.jarmac.2020.07.004
Pressley, M., Tanenbaum, R., McDaniel, M. A., & Wood, E. (1990). What happens when university
students try to answer prequestions that accompany textbook material? Contemporary
Educational Psychology, 15(1), 27–35. http://dx.doi.org/10.1016/0361-476X(90)90003-J
Richland, L. E., Kornell, N., & Kao, L. S. (2009). The pretesting effect: Do unsuccessful retrieval attempts
enhance learning? Journal of Experimental Psychology: Applied, 15(3), 243-257.
Roediger, H. L. III, Agarwal, P. K., McDaniel, M. A., & McDermott, K. B. (2011). Test-enhanced learning in
the classroom: Long-term improvements from quizzing. Journal of Experimental Psychology:
Applied, 17(4), 382-395. https://doi.org/10.1037/a0026252
St. Hilaire, K. J., & Carpenter, S. K. (2020). Prequestions enhance learning, but only when they are
remembered. Journal of Experimental Psychology: Applied, 26(4), 705-716.
St. Hilaire, K. J., Carpenter, S. K., & Jennings, J. M. (2019). Using prequestions to enhance learning from
reading passages: The roles of question type and structure building ability. Memory, 27(9), 1204-
1213. https://doi.org/10.1080/09658211.2019.1641209
Swenson, I., & Kulhavy, R. W. (1974). Adjunct questions and the comprehension of prose by children.
Journal of Educational Psychology, 66, 212-215. https://doi.org/10.1037/h0036276
Toftness, A. R., Carpenter, S. K., Lauber, S., & Mickes, L. (2018). The limited effects of prequestions on
learning from authentic lecture videos. Journal of Applied Research in Memory and Cognition,
7(3), 370-378. https://doi.org/10.1016/j.jarmac.2018.06.003
82
In Their Own Words
Successive Relearning: An Introduction and
Guide for Educators
John Dunlosky Kathryn T. Wissman
Kent State University North Dakota State University
Editors
Maren Greve Katherine A. Rawson
Lauren L. Kordonowy
Victor A. Benassi
Kent State University Kent State University
Sabrina Badali
Kent State University
Beth and Jeff are both high school basketball coaches and are getting their players ready for the first
game of the season. For over a month, they have worked with individual players and their entire teams,
drilling them on the basics and practicing key skills during each practice until the players get it right.
Their players are ready for the season, yet they still revisit the same drills every practice, so that the
players keep working on these skills until they are overlearned. Beth and Jeff also teach biology in their
schools. When preparing their students for an exam, they also use a typical approach: they introduce
each topic during a lecture and then have students complete a practice worksheet or two over the
relevant material, and later they emphasize that those topics will be on the upcoming exam. An evening
or two before the exam, students review the material. That is, the teachers present most materials only
once and the students essentially cram for the exam. These teachers are not surprised when many of
their students perform poorly, because their students often struggle to retain what they have learned
well enough to excel on the exams. It may be obvious that these teachers’ approach for basketball is
different than their approach for biology. But what is the difference, and does it matter?
What is Successive Relearning?

Without even realizing it, Beth and Jeff were using successive relearning to help their players improve
their basketball skills. In fact, just about any skill that people are good at has arisen from the application
of successive relearning, so let’s take a closer look. Successive relearning involves combining two other
effective strategies – retrieval practice (the “what” to do) and spaced practice (the “when” to do it).
Successive relearning involves practice testing in which learners attempt to retrieve sought-after
information from memory. Examples of retrieval include trying to recall the meaning of a concept or
definition from memory, retrieving the correct name of a structure (e.g., part of a neuron), or even
retrieving the stages in a lengthy process (e.g., photosynthesis) by drawing them on a piece of paper.
After a retrieval attempt, the answer needs to be scored, and if it is correct, then the student is finished
with that particular item for the session. If the answer is incorrect, the student would study the correct
response and then complete another practice test on the same item later in that session. The student
would continue to test and restudy all items until each had been correctly retrieved. After an initial
practice session, the student would return to relearn the same items during two or more subsequent
sessions (i.e., spaced practice) to again practice retrieval (with feedback) until all the items are again
correctly retrieved.
This sequence of processes is perhaps most easily illustrated by the appropriate use of flashcards (for a
detailed flow chart of this sequence, see Figure 1 in Dunlosky & O’Brien, 2020). Each card of a stack
would have a test question on one side and an answer on the other. Although students’ preferred items
83
for flashcard use are foreign-language translation equivalents (Wissman et al., 2012), the question and
answer on each card could also involve other kinds of content (e.g., lengthy, conceptual, and complex
material). As a student takes each test and scores the answer, the current card would either be set aside
(if the answer was correct) or put at the end of the stack to practice again (if the answer was incorrect).
In this way, the practice test for items would be spaced (with the interval between practice attempts
being filled with the practice of other items) within each session. As we discuss under Frequently Asked
Questions, flashcards can be real or virtual; that is, actual cards do not need to be used, and just about
any course material (e.g., notes or even textbook content) can be easily altered to function as
flashcards. For instance, in a textbook, students could place sticky notes over the definitions of key
terms, leaving the key term itself uncovered. They could take a test by trying to write down the meaning
of each concept from memory and then revealing the answers for further study. The students would
work from beginning to the end of a chapter, returning to the beginning until all concepts had been
correctly retrieved.
Although successive relearning involves both retrieval practice and spaced practice, it is critically
different than just combining the two techniques as they are typically used. First, retrieval practice often
involves a single attempt at retrieving a concept (e.g., attempting to answer a single multiple-choice
question for a given item), whether the concept is correctly retrieved or not. This retrieval attempt can
have a positive impact on students’ learning (for reviews, see McDaniel & Little, 2019; Rowland, 2014),
especially when the answers are correctly retrieved (cf. Kornell et al., 2009; Little et al., 2012). By
contrast, for successive relearning, students (a) continue to practice retrieval (with feedback) until they
correctly recall all the information at least one time during a single session and, as important, (b) return
to the same content to relearn it during one or more sessions. As noted above, practice within a session
for a given item would be spaced, because if an answer is incorrect for an item, then that item would
not be tested again until other items within the to-be-learned set (e.g., a flashcard stack) are practiced.
Given that successive relearning (obviously) involves relearning, the practice of all items is also spaced
across days, which would further boost its impact on long-term retention.
How might successive relearning improve student achievement? The answer is simple – after correctly
recalling even simple information a single time (e.g., the answers to questions like “What is the word for
retrieval in Spanish?” Answer: recuperación), people will eventually forget it and need to practice it
again (perhaps across multiple sessions) to retain it over the long term (for further discussion, see
Bahrick, 1979; Dunlosky & O’Brien, 2020). Thus, by successfully practicing the same content across
multiple sessions, retention of that material increases after each one, resulting in enhanced
achievement. In fact, successive relearning is foundational to most expertise. Consider work by Harry
Bahrick and colleagues (Bahrick, 1979; Bahrick & Hall, 1991; Bahrick et al., 1994) who explored why
some people retain competency over a lifetime, whether it be their ability to solve algebra problems or
speak fluently in a language when moving to a foreign country. They discovered that (very) long-term
retention involved using that knowledge across a lifetime.
The Promise of Successive ReLearning: Experimental Demonstrations

With Course-Relevant Content
To date, only a few experiments on successive relearning have explored how it impacts knowledge of
content from courses in which students are enrolled (Higham et al., 2022; Janes et al., 2020; Rawson et
al., 2013). We briefly review these studies. Rawson et al. (2013) and Janes et al. (2020) used similar
methods aimed at improving college students’ achievement in an Introductory Psychology course and in
a Biopsychology course, respectively. Examples of the content are presented in Table 1.
84
Table 1
Example Materials From Classroom-based Research on Successive Relearning
Article Question/Prompt Answer
Rawson et al. (2013) What is the The tendency to estimate the likelihood that
availability heuristic? an event will occur by how easily instances of it
come to mind.
What is the mere The phenomenon whereby the more people
exposure effect? exposed to a stimulus, the more positively they
evaluate that stimulus.
What is hindsight bias? The tendency, once an event has occurred,
to overestimate one's ability to have foreseen
the outcome.
Janes et al. (2020) Image: Arrow pointing to Name/label of region
What are meninges? Protective membranes surrounding the brain
and spinal cord.
Note. For images, students were shown an image with an arrow pointing to a region of it; depending on
the image, students would practice retrieving the name of the region (e.g., frontal lobe) or the process
depicted in the image (e.g., if an arrow pointed to the state of a cell immediately before an action
potential, the answer would be “cell at rest”).
In both studies, each student was assigned to use successive relearning to learn concepts on virtual
flashcard stacks; each stack typically included 6-8 concepts, and concepts in a given stack were taken
from the same unit (e.g., one stack for Introductory Psychology included concepts relevant to judgment
and decision making). Students used successive relearning to learn these concepts on three or more
sessions that were separated by at least one day. Moreover, students began using successive relearning
to learn a set of concepts when the instructor introduced them in class. Note that during retrieval
practice of concepts, the students were told that they did not have to produce a verbatim copy of the
answer, but instead they were encouraged to recall the meaning of concepts in their own words. The
students scored the correctness of an answer, and if correct, the item was dropped from that session; if
the answer was incorrect, the students continued to practice it more. Finally, each student was assigned
other business-as-usual concepts, which the students could learn in any manner that they wanted
(whether concepts were assigned to the successive-relearning condition or the business-as-usual
condition was counterbalanced across students). The rationale for using these business-as-usual
concepts as a baseline condition is straightforward: If successive relearning does not help students
improve their achievement as compared to how they already prepare, then it would not seem prudent
to encourage its use.
85
Figure 1 presents outcomes from
Rawson et al. (2013, Experiment
1), with the left-most bars
highlighting the critical outcome:
Successive relearning boosted
exam performance over 10% as
compared to the baseline
control. Similarly, students who
used successive relearning to
study biopsychology content also
obtained about a 10% boost in
their exam performance (Janes
et al., 2020). Rawson et al. (2013,
Experiment 1) included two
other groups. One group had the
same practice schedule as
students using successive
relearning, but students in this
group used the practice time to
restudy the content instead of
engaging in retrieval practice.
This restudy-only group (Figure 1,
middle bars) also benefited from
spaced restudy (as compared to
the baseline control condition in which participants did not practice concepts but were later tested on
them), but not as much as compared to those who used successive relearning. Thus, practicing retrieval
is an active ingredient for how successive relearning boosts achievement. Another group was given the
same schedule as those using successive relearning, but students in this self-regulated practice group
could practice the content in any way they wanted (i.e., they could test themselves, restudy, or do
nothing). As shown in the right-most bars of Figure 1, these students also benefited from spaced
practice, but not as much as the successive-relearning group (for detailed analyses of how these
students regulated their learning, see Janes et al., 2018). Two other aspects of Rawson et al. (2013) are
also notable. First, although students successively relearned by attempting to retrieve the meaning of
the definitions from memory, none of the questions on the in-class exam involved identifying
definitions; instead, students had to use their understanding of a definition to correctly answer
questions, so successive relearning facilitated the application of those concepts on (near) transfer
questions. Second, another experimenter-developed test that did involve recalling the meaning of
definitions was administered 24 days after the exam. Students recalled over 60% of the concepts
correctly when they used successive relearning to study them, whereas they recalled less than 20%
when they learned the concepts on their own. This outcome highlights the power of successive
relearning to support long-term retention, as well as demonstrating that students’ reliance on cramming
to study course materials results in rapid forgetting.
Higham et al. (2022) also explored the possible benefits of successive relearning while matching time on
task between successively relearned material and material that was learned by restudy alone (i.e., a
well-controlled version of the restudy-only group in Rawson et al., 2013). After each lecture of a large
undergraduate psychology course, students studied concepts from the prior class (e.g., Attitudes are
composed of affective, cognitive, and behavioral components) on three separate days either using
86
successive relearning or restudy. When using successive relearning, students attempted to recall a
missing term (e.g., Attitudes are composed of ? , cognitive, and behavioral components), and
then the correct answer was presented. For the restudy condition, the concept was presented in its
entirety. Across the lectures, students alternated between using successive relearning or restudying
alone to practice particular concepts, so all students used each technique for about half the concepts.
And, for the successive relearning and restudy conditions, the overall time on task was approximately
equated. The criterion tests were administered about 42 and 68 days after the final practice session (no
questions were repeated across these tests). The criterion tests included the following: repeated
questions that involved recalling the terms that were practiced during successive relearning (e.g.,
Attitudes are composed of ? , cognitive, and behavioral components); transfer questions
that involved recalling a different term (e.g., Attitudes are composed of affective, ? , and
behavioral components) or new questions that had not been practiced.
For the repeated questions, students recalled significantly more of the practiced terms 42 days after
practice when they had used successive relearning than when they restudied alone, and the benefit of
using successive relearning (albeit smaller) was also significant 68 days after practice. The benefit was
minimal for transfer and new questions. The authors also reported outcomes from a host of other
measures (see Higham et al., 2022, for details) that further demonstrated the broader impact of
successive relearning. For instance, early in the course, students judged that restudy was more effective
than successive relearning, but after they had used both, they judged successive relearning as more
effective. Students also initially indicated they were more anxious using successive relearning than
restudy, whereas after using both, they reported less anxiety when using successive relearning. These
and other outcomes led the authors to conclude that “[o]ur research indicates that successive relearning
is a valuable addition to any university course and is easy to implement” (Higham et al., 2022).
The final study on successive relearning that we review here (Rawson et al., 2020) was not connected to
a specific course but did involve learning foundational concepts in probability. College students
attempted to learn how to solve four different kinds of probability problems. They attempted to solve a
novel problem of each kind and received feedback (whether their answer was correct and the correct
solution) after each attempt. (So that students would need to solve a novel problem for each attempt,
we developed 17 isomorphic problems for each kind of probability problem.) They continued to cycle
through answering problems (with feedback) until they had correctly answered one problem of each
kind. For the successive relearning group, students then repeated the same learning session on three
separate days – so they continued to practice until they could correctly solve each kind of problem
during three separate sessions. Students in the comparison group completed just one session in which
practice continued until they correctly solved three problems of each kind. One week after the final
practice session, students completed a criterion test that involved solving novel problems. Across
multiple experiments, the successive relearning group demonstrated a significant advantage on the
criterion test, but the effect size was small (Cohen’s d = .28); average performance was also around 50%
or less for the successive relearning group, which means even more (or a different kind of) practice may
be needed to gain mastery. Thus, although successive relearning did help students learn to solve
probability problems, further research will be required to discover how best to use successive relearning
in service of improving students’ problem solving.
Barriers to Using Successive Relearning Effectively and

Recommendations to Overcome Them
What should be evident from the examples discussed above is that successive relearning can have a
positive impact when students are learning complex and conceptual materials; that is, successive
87
relearning can be beneficial for more than just learning foreign-language vocabulary. Even so, we
suspect that a variety of barriers will need to be overcome for teachers and students to fully capitalize
on its power. We consider some of these barriers here, along with possible approaches to overcoming
them.
Barrier #1
Students do use successive relearning, but they report using it largely to learn foreign-language
vocabulary. Certainly, successive relearning is an excellent strategy to learn foreign-language translation
equivalents. However, many students do not realize that they can use (or simply do not try using)
successive relearning to learn conceptual definitions, scientific processes, lengthy text materials, et
cetera. Currently, we are unsure how to motivate students to use successive relearning for learning
content other than foreign-language translations, although instructing students about the benefits of
successive relearning and allowing them to experience those benefits while learning more complex
material may increase the likelihood that they take advantage (cf. McDaniel & Einstein, this volume).
Barrier #2
Using successive relearning can be time consuming. One reason why some students may not use
successive relearning for more complex materials is that it takes time to develop the materials, such as
by writing key terms on one side of each flashcard in a set and their lengthy definitions on the other
side. Moreover, for studying lengthy definitions, students will need to schedule enough time to learn
them well enough to recall their meanings. Concerning the barrier to build the material sets, many
virtual flashcard programs are available (and free), and most allow students to share flashcard stacks
(for details, see Dunlosky & O’Brien, 2020) – so students can share the burden of developing materials.
Concerning the time needed to learn and relearn the materials, we offer three pieces of advice. First, let
your students know that using successive relearning will require time, but if they stick with it, they will
experience the benefits. Second, let them know that relearning is much faster than is initial learning. It
may take students 30 to 40 minutes to initially learn to correctly recall eight to ten lengthy definitions,
but during the first relearning session, they may need only about 15 minutes, and the next session will
likely take fewer than 5 minutes (e.g., Rawson & Dunlosky, 2011). The success experience itself will
contrast a common student experience of studying a lot and gaining little and may encourage some
students to use successive relearning more regularly. Plus, as students use successive relearning, they
will also experience the dramatic speed-up from initial learning to relearning. Finally, we recommend
that every instructor considers “what is worth learning” – what do students need to remember and use
well after your course is done? Consider identifying the smallest number of “must know” concepts from
a course and give your students extra kudos for mastering those by using successive relearning.
Barrier #3
Even if students plan to use successive relearning and develop the materials to do so, they will need to
manage their time to use it with fidelity. One possibility is to have students use a calendar to note
when a particular material set should be started (e.g., when the concepts are introduced in a lecture)
and revisited for relearning sessions (e.g., for three extra sessions separated by 2 days). It would be even
better if their calendar sent reminders about when they should be learning (or relearning) different
material sets. Of course, having multiple study sessions for a given course each week stands in stark
contrast to students’ propensity to wait to cram an evening or two before an exam (Blasiman et al.,
2016; Taraban et al., 1999), so reminders to study (and use successive relearning) regularly may be
valuable. Fortunately, some of the flashcard programs available on the internet have built-in calendars
and messaging systems, so that students can set up a study schedule that will allow them to use
successive relearning with fidelity (Dunlosky & O’Brien, 2020).
88
Barrier #4
Many people believe that successive relearning is just for memorization and cannot help students
develop a deeper understanding of conceptual material. This barrier pertains to any instructor or
student who does not understand that memory is foundational for human performance of any kind. For
instance, to answer conceptual questions about the outcome of a memory experiment about whether
recency and primacy effects will occur when students attempt to recall a list of words from memory, a
student can apply Atkinson and Shiffrin’s (1968) model of memory that includes short-term and long-
term memory. Using this model to develop predictions involves coordinating several concepts, such as
the meanings of recency and primacy effects, the size of short-term memory, the control processes
involved in short-term memory, transfer to long-term memory, et cetera. Doing so would likely be
difficult if all those concepts were not readily accessible from memory – so, in this case, using successive
relearning to enhance memory for the individual concepts promises to impact students’ ability to apply
the model to make predictions. Also, the first author has heard instructors lament that students
memorize all the material but it does not help them; the mistake here is believing that the students are
actually memorizing the content well enough to remember it. Instead, many students do not use
effective techniques to learn course content, so they may believe that they are doing a good job
“memorizing” the content, when in fact their retention of the content is fragile and will not support its
use in answering application questions.
Moreover, your own experiences may highlight that even a solid conceptual understanding of a process
can be forgotten without practice. For instance, after teaching signal detection theory, the first author
developed a relatively solid understanding of the basics, but not so long after teaching it, he quickly
forgot his understanding of this theory. He needed to use successive relearning to revisit his
understanding of signal-detection theory to achieve long-term retention, just like using algebra
throughout one’s life ensures that an understanding of the basics will not be forgotten (as per H.
Bahrick’s research, e.g., Bahrick et al., 1994). Put differently, even comprehension of an idea can be
forgotten, and when it is, it would need to be relearned.
Some Frequently Asked Questions

In discussing effective learning techniques with instructors and students, we hear many of the same
questions, and here are some of them along with our (currently) favored answers.
Is successive relearning just flashcards? No. Flashcards can be used to instantiate successive relearning,
but flashcards can be used in a variety of different ways that will not necessarily produce durable
learning. Moreover, successive relearning can be instantiated using other materials as well, such as
textbooks and notes (for examples, see Dunlosky & O’Brien, 2020). For notes, one recommendation is to
have students write on only one side of each sheet of paper, and then they can use the reverse, blank
side to write down content that they are attempting to retrieve from memory.
Can students accurately score the quality of their responses when practicing retrieval? Having students
score their own responses is essential for using successive relearning in some contexts, such as when
students learn lengthy conceptual definitions or scientific processes. Accurately scoring the responses is
essential, because if a student is overconfident and scores incorrect responses as correct, then the
student may drop items from practice too early, which would result in poor retention (Dunlosky &
Rawson, 2012). Of course, students can compare their responses with the correct answers, but they may
still have difficulties scoring long responses accurately without some form of scaffold or instruction. An
important avenue for future research will be discovering techniques for ensuring that students can
accurately score responses of any kind.
89
Do students need to write down what they are trying to recall, or can they just recall the content
covertly? Students do not need to overtly produce (e.g., write, type, or say aloud) the information they
are trying to recall from memory, but doing so may provide advantages. First, when attempting to
covertly recall lengthy responses, students may not do a full-blown retrieval attempt. By contrast,
writing down responses may increase the chances that they retrieve all they can from memory, which in
turn will enhance their retention of the material (for relevant evidence, see Tauber et al., 2018). As
important, when students write down their best response, they may have an easier time scoring it
accurately, because they would not need to rely on a limited working memory to hold their (potentially
lengthy) response in mind while comparing it to the correct answer.
Will successive relearning benefit younger students? The scant experimental research on successive
relearning has almost exclusively involved adults. Even so, children and younger students show the
benefits of retrieval practice and spaced practice, so it follows that they would also benefit from using
successive relearning. We know of only one experiment evaluating the benefits of successive relearning
for children (Rawson et al., 2021). The experiment was designed to evaluate whether relearning sessions
boosted eighth graders’ retention of concepts from a unit on statistics. The concepts were key term and
definition pairs, such as “What is the mode? The number or object that appears most frequently in a set
of numbers or objects.” One hundred seventy-four students from eight different classes initially learned
to recall the meaning of each definition from memory. For retrieval trials, students were prompted with
a key-term question (e.g., “What is . . . ?”) and were asked to type the meaning of the concept in their
own words. After this initial learning session, the students were randomly assigned to complete zero,
one, two or three relearning sessions. As during initial learning, each relearning session involved practice
test–restudy trials until each definition was correctly recalled once. One month after the final practice
session (either after the initial learning session for the zero relearning group or after the last relearning
session in each of the other groups), students completed a final cued recall test over the practiced
concepts (as well as for baseline control concepts that were included in the unit but were not presented
during any learning sessions; baseline
and practice concepts were counter-
balanced across students).
As shown in Figure 2, performance on
the final test was greater for
concepts that the students practiced
than for the baseline concepts. Most
important, the relearning sessions
significantly boosted performance. A
month after practice, students who
completed three relearning sessions
recalled nearly 60% of the concepts
from memory. But how much extra
time did relearning require? Students
used about 3 minutes to reach a
criterion of one correct recall during
the initial learning session, but they
sped up across relearning sessions.
By the third relearning session,
students used only about a minute
on average to relearn each concept.
90
How many items should be studied within a given session? Or, in terms of using flashcards, how many
cards should be in a given stack? We suspect the answer to this question will depend on many factors,
such as how much time a student is willing to use for each study session, the length and complexity of
materials, and so forth. For instance, if students are using successive relearning to learn foreign-
language translation equivalents, we suspect that a given stack could be rather lengthy. By contrast,
initially learning to recall the meaning of conceptual definitions can be time consuming, so students may
want to include relatively few (8 – 10) in a given stack.
How many relearning sessions should students use for a given set of material? When preparing for
exams in a course, the number of days between exams (and amount of material) will likely limit how
many sessions that students could schedule. For instance, if each exam is separated by four weeks and
covers a great deal of material, that probably leaves just enough time to relearn materials on two
(perhaps three) extra sessions. If the exams are followed by a cumulative final exam, then students
could return to the material previously introduced to engage in one or two relearning sessions near the
end of the semester. If the students previously used successive relearning to learn the material, they will
retain much of the material across the course (Rawson et al., 2013), and for material they have
forgotten, relearning will be rapid. Thus, relearning for these final booster sessions should not take a
great deal of time. If students care to retain that material beyond the course (e.g., for an entrance exam
to medical school), then they will likely need to revisit the material to relearn it. Again, we suspect that
such relearning would be relatively quick and that many students will be amazed by how much they
retained. With this optimism in mind, however, we must emphasize that research has not yet
systematically explored “how much is enough” for producing (extremely) long-term retention of
knowledge. Given the promise of successive relearning, a major challenge for educational research will
be to better understand how much successive relearning is required to enjoy its long-term benefits.
Closing Remarks
Beth and Jeff use successive relearning to coach their basketball teams, which is consistent with this
modified version of an old adage: “try until you succeed, and then try, try again (until you succeed a few
more times).” Just imagine if Beth and Jeff also supported the use of successive relearning in their
biology course. They could develop material sets for their students to learn successively after school,
perhaps even having their students begin learning (and relearning) content while the material is being
first introduced in class lectures. Outside of class, the students could master definitions, the sequences
of biological processes, the names of biological structures, and much more; and then Beth and Jeff could
spend class time helping students more deeply understand and use the content. Of course, we do not
know how the students would fare in such a situation, but given the current evidence on successive
relearning, it seems likely that most students would obtain a boost in their grades and long-term
retention of biology. And, after all, if students already use successive relearning to excel at their
extracurricular activities – such as video games, sports, and music – why shouldn’t instructors help
students use successive relearning with fidelity in their academic learning? We hope that this brief guide
about successive relearning will entice you to try it out yourself, either with your students or while you
are trying to master new content for your discipline or pastime.
Author Note
This research was partially supported by the Institute of Education Sciences and by the National Science
Foundation under grant IUSE-1914499.
91
References
Atkinson, R.C. & Shiffrin, R.M. (1968). Human memory: a proposed system and its control processes. In
K.W. Spence & J.T. Spence (Eds.), The psychology of learning and motivation: Advances in
research and theory. (Vol. 2) (pp. 89–195). Academic Press.
Bahrick, H. P. (1979). Maintenance of knowledge: Questions about memory we forgot to ask. Journal of
Experimental Psychology: General, 108(3), 296–308. https://doi.org/10.1037/0096-
3445.108.3.296
Bahrick, H. P. and Hall, L. K. (1991). Lifetime maintenance of high school mathematics content. Journal of
Experiment Psychology: General, 120(1), 20–33. https://doi.org/10.1037/0096-3445.120.1.20
Bahrick, H.P., Hall, L.K., Goggin, J.P., Bahrick L.E. & Berger, S.A. (1994). Fifty years of language
maintenance and language dominance in bilingual Hispanic immigrants. Journal of Experimental
Psychology: General, 123(3), 264-283. https://doi.org/10.1037/0096-3445.123.3.264
Blasiman, R. N., Dunlosky, J., & Rawson, K. A. (2017). The what, how much, and when of study strategies:
Comparing intended versus actual study behaviour. Memory, 25(6), 784–
Dunlosky, J., & O’Brien, A. (2020). The power of successive relearning and how to implement it with
fidelity using pencil and paper and web-based programs. Scholarship of Teaching and Learning
in Psychology. https://doi.org/10.1037/stl0000233
Dunlosky, J. & Rawson, K. A. (2012). Overconfidence produces underachievement: Inaccurate self
evaluations undermine students’ learning and retention. Learning and Instruction, 22(4), 271-
Higham, P. A., Zengel, B., Bartlett, L., & Hadwin, J. A. (2022). The benefits of successive relearning on
multiple learning outcomes. Journal of Educational Psychology, 114(5), 928–944.
Janes, J., Dunlosky, J., Rawson, K. A., & Jasnow, A. (2020). Successive relearning improves performance
on a high-stakes exam in a difficult biopsychology course. Applied Cognitive Psychology, 34(5),
1118-1132. https://doi.org/10.1002/acp.3699
Janes, J., Dunlosky, J., & Rawson, K. A. (2018). How do students use self-testing across multiple study
sessions when preparing for a high-stakes exam? Journal of Applied Research in Memory and
Cognition, 7(2), 230-240. https://doi.org/10.1016/j.jarmac.2017.11.003
Kornell, N., Hays, M.J., & Bjork, R. (2009). Unsuccessful retrieval attempts enhance subsequent
learning. Journal of experimental psychology. Learning, memory, and cognition, 35(4), 989-98.
Little, J. L., Bjork, E. L., Bjork, R. A., & Angello, G. (2012). Multiple-choice tests exonerated, at least of
some charges: Fostering test-induced learning and avoiding test-induced forgetting.
Psychological Science, 23(11), 1337-1344. https://doi.org/10.1177/0956797612443370
McDaniel, M. A., & Little, J. L. (2019). Multiple-choice and short-answer quizzing on equal footing in the
classroom: Potential indirect effects of testing. In J. Dunlosky & K. A. Rawson (Eds.) The
Cambridge handbook of cognition and education (pp. 480-499). Cambridge University Press.
92
Rawson, K. A., & Dunlosky, J. (2011). Optimizing schedules of retrieval practice for durable and efficient
learning: How much is enough? Journal of Experimental Psychology: General, 140(3), 283-302.
Rawson, K. A., Dunlosky, J., & Janes, J. L. (2020). All good things must come to an end: A potential
boundary condition on the potency of successive relearning. Educational Psychology Review,
32(3), 851-871. https://doi.org/10.1007/s10648-020-09528-y
Rawson, K. A., Dunlosky, J., & Sciartelli, S. M. (2013). The power of successive relearning: Improving
performance on course exams and long-term retention. Educational Psychology Review, 25(4),
523-548. https://doi.org/10.1007/s10648-013-9240-4
Rawson, K. A., Wissman, K. T., & Dunlosky, J. (2021). Grade-school students’ memory for statistical
concepts benefits from successive relearning. Unpublished data.
Rowland C. A. (2014). The effect of testing versus restudy on retention: a meta-analytic review of the
testing effect. Psychological Bulletin, 140(6), 1432–1463. https://doi.org/10.1037/a0037559
Taraban, R., Maki, W.S. & Rynearson, K. (1999). Measuring study time distributions: Implications for
designing computer-based courses. Behavior Research Methods, Instruments, &
Computers 31(2), 63–269. https://doi.org/10.3758/BF03207718
Tauber, S. K., Witherby, A. E., Dunlosky, J., Rawson, K. A., Putnam, A. L., & Roediger, H. L. (2018). Does
covert retrieval benefit learning of key-term definitions? Journal of the Association of Research,
Memory, and Cognition, 7(1), 106-115. https://doi.org/10.1016/j.jarmac.2016.10.004
Wissman, K. T., Rawson, K. A., & Pyc, M. A. (2012). How and when do students use flashcards? Memory,
20(6), 568-579. https://doi.org/10.1080/09658211.2012.687052
93
In Their Own Words Applying the ICAP Framework to Improve
Classroom Learning
to Apply the Science of Learning in Michelene T. H. Chi and Nicole S. Boucher
Arizona State University
Editors
Lauren L. Kordonowy
Victor A. Benassi
Introduction to ICAP
The recent drive to restructure classrooms to improve learning has been shaped by the term “active
learning.” While “active learning” has become a familiar term in education, it has been defined as
referring to just about any learning strategy that is not a lecture (Freeman et al., 2014). Thus, educators
remain without guidance in deciding which types of activities should be included in the classroom and
for what purpose. To help teachers design and implement active learning strategies in the classroom,
ICAP (Interactive, Constructive, Active, and Passive), a science of learning and evidence-based theory,
provides heuristics that can help differentiate and distinguish between different types of active learning
activities in terms of their effectiveness for improving learning (Chi & Wylie, 2014).
ICAP refers to two kinds of engagement: physical and cognitive. Physical engagement refers to the
visible actions or activities students are doing while they interact with instruction—such as taking notes,
listening to a lecture, and so forth (the term “instruction” is used broadly to include both the instructor’s
actions and the instructional materials). Although physical interactions with instruction involve overt
behaviors, it is distinct from the traditional definition of behavioral engagement, which refers to
students’ overt actions that reflect their initiation, persistence, or completion of their interactions with
instruction (Marks, 2000; Skinner & Belmont, 1993). For example, the frequency of “attending class” is
an index of behavioral engagement, and it refers to the initiation of interactions with instruction, while
“turning in homework assignments” refers to the frequency of completing interactions with instruction.
Even in the context of online learning, many indices are measured, and they all refer to behavioral
engagement, such as the frequency of accessing the homework assignments or a specific page of the
instructional materials. We use the term physical engagement to refer to the processes of interacting
with instruction, such as taking notes while reading or listening to a lecture, instead of the initiation and
completion of interactions with instruction. Thus, ICAP is a theory about the processes of how students
learn. Cognitive engagement, on the other hand, refers to students’ underlying thinking processes while
learning. Because cognitive processes are invisible, ICAP uses physical engagement as an index to reflect
cognitive engagement.
The ICAP framework combines observable student physical actions/interactions with instruction, along
with an analysis of student-produced outputs (such as notes or questions) to define four modes of
student engagement: Interactive, Constructive, Active, and Passive. These four engagement modes
correspond to four different levels of learning in the I>C>A>P descending order, with the Interactive
mode leading to deeper and more learning and the Passive mode to shallower and less learning. The
ICAP framework provides heuristics to help teachers differentiate active learning activities and classify
each activity into the associated ICAP learning mode.
94
ICAP is applicable for many content domains, but it is particularly relevant for learning difficult to
understand concepts, such as those in STEM (science, technology, engineering, and math) domains
because ICAP enhances deeper learning. ICAP can be beneficial for students’ learning from pre-K to
college level and beyond, and in many contexts, such as face-to-face or online courses. Whereas existing
overarching frameworks typically used in designing lesson plans or course work start by classifying
instructional objectives or assessment items from the perspective of the instructors (Bloom &
Krathwohl, 1956; Dick & Carey, 2005; Krathwohl, 2002), the ICAP theory designs lesson plans from the
perspective of the students, based on what students are asked to do when interacting with instruction.
ICAP and its supporting evidence is discussed in detail in Chi (2009), Chi et al. (2018), Chi and Wylie
(2014), Fonesca and Chi (2011), and Menekse et al. (2013).
The taxonomy of the four ICAP modes are briefly described below. Each of these modes is operationally
defined by heuristics consisting of two overt indices: the physical actions of what students do and the
visible outputs (if any) of what students produce, while interacting with instruction. In addition, we
allude to the potential underlying thinking processes of each mode. The heuristic indices and the
plausible cognitive processes are shown in Table 1.
Passive
We define the Passive mode of engagement as learners interacting with instruction physically by
orienting towards or paying attention to instruction with no visible outputs produced. For example,
paying attention and listening to a teacher’s lecture without taking notes is considered engaging in the
Passive mode. Thus, the two overt indices of engaging in the Passive mode are paying attention and
producing no visible outputs.
Note that ICAP’s definition of each mode does not allude to the thinking processes, as thinking
processes are invisible to instructors. We assume that the thinking processes occurring while paying
attention might be receiving and storing information. Although it may be possible for students to be
thinking deeply while processing instruction (such as making connections to other course concepts or
revising one’s representation of the concept), ICAP assumes that a student is more likely (but not with
100% certainty) to be simply receiving and storing information. Thus, the student is more likely to be
engaged in the Passive mode when only paying attention and not displaying any other activities.
Active
Learners’ engagement with instructional materials is classified as Active when students physically
manipulate some parts of the instructional materials, such as pointing to or gesturing at parts of what
they are solving or reading (Alibali & DiRusso, 1999), pausing and rewinding parts of a videotape (Chi et
al., 2008), underlining certain text sentences (Katayama et al., 2005), copying some problem solution
steps (VanLehn et al., 2007), mixing certain chemical amounts in a hands-on laboratory (Yaron et al.,
2010), choosing a justification from a menu of options (Conati & VanLehn, 2000), and bookmarking a
page in a textbook. The visible outputs produced from manipulating can be identified as information
already existing within the content materials. For example, if students are highlighting a text, the
highlighted sentences already existed within the instructional materials.
The advantage of manipulating parts of the instructional materials is that it gives students opportunities
to re-focus and pay attention to the selected information again, thus allowing additional thinking
processes of re-activating relevant related information and thereby strengthening it. Selection of the
parts of the instructional materials to manipulate can be based on syntactic cues within the instructional
materials, such as important words like “the principle,” or “the most important equation,” and so forth.
95
Constructive
ICAP defines the Constructive mode as having the characteristics of physically generating some external
outputs and the external outputs contain additional information that goes beyond what was provided in
the learning materials. To meet these criteria for Constructive, the outputs of generative interactions
must contain new ideas or minute inferences not identifiable verbatim in the instructional materials. An
important caveat: when we say an inference or “new” idea, we are not referring to an innovative or
never-before-discovered novel idea in the content domain of instruction. Instead, we mean that the
generated information is new in relation to the instructional materials provided. For example, suppose a
student creates a concept map. If the student creates a concept map in her notes that makes new
connections between the nodes and such connection information was not already presented, then the
student is engaging in the Constructive mode. However, if the student copied a concept map already
existing in the textbook, then the student would be engaged more in the Active mode, as the original
concept map was merely manipulated by being copied.
Many different learning activities fit into our definition of Constructive (generative) activities. Examples
include drawing a concept map (Biswas et al., 2005; Novak, 1990a; 1990b); taking notes in one’s own
words (Trafton & Trickett, 2001); asking questions (Graesser & Person, 1994); comparing-and-
contrasting cases; integrating two texts (Britt & Sommer, 2004), text and diagrams (Butcher, 2006), or
across multimedia resources (Bodemer et al., 2004); inducing hypotheses and causal relations (Suthers
& Hundhausen, 2003); drawing analogies (Chinn & Malhotra, 2002); generating predictions (Schauble et
al., 1995); reflecting and monitoring one’s understanding or other self-regulatory activities (Azevedo et
al., 2006); constructing timelines for historical phenomena (Dawson, 2004), and self-explaining (Chi et
al., 1994). These examples can all be classified as Constructive activities because they satisfy the two
indices in our definition: (1) they require students to be generative, and (2) the outputs must include
additional information not available in the original learning materials (e.g., asking students to compare-
and-contrast two examples requires the learners to say what is the same or different between the
examples, when the instructional materials may not have mentioned their similarities and differences;
similarly, asking students to integrate text and a diagram obviously requires them to articulate relations
about them that were not explicitly presented).
We surmise that the thinking processes involved in being generative is to reason in various ways that
can produce new knowledge, such as inferring new relations, revising one’s existing knowledge,
deducing causal consequences, and so forth. Thus, the Constructive mode involves the fundamental
cognitive processes of learning, consisting of making sense of instructional inputs by activating relevant
prior knowledge, connecting new information with prior knowledge, inferring new knowledge, revising
prior knowledge, and so forth.
Interactive
We define the Interactive mode as collaborations/interactions between students (such as through
dialoging) and with instruction that meet two criterial indices: (1) two (or more) students are engaging
in reciprocally co-generative behaviors; and (2) the outputs contain information that goes beyond the
instructional materials and beyond what each partner contributes individually. The Interactive mode
refers to a unique form of collaboration that is reciprocally co-generative, meaning that not only is each
partner generative in the Constructive sense of going beyond the presented instructional information,
but, moreover, each partner should be generative by building upon the partner’s contributions.
This concept of mutually, reciprocally exchanging ideas can occur in natural dialog or in specifically
formatted dialogs such as debating (defending or arguing a position, Schwarz et al., 2000), critiquing
each other by requesting justification (Okada & Simon, 1997), asking and answering each other’s
96
questions (Webb, 1989), peer tutoring (Roscoe & Chi, 2007), and elaborating on each other’s
contributions, such as clarifying, building upon, correcting, and so forth (Hogan et al., 1999). While
numerous definitions exist to define collaborative behaviors, our definition of Interactive or reciprocally
co-generative collaborative dialogs is more specific by requiring both partners to make Constructive
contributions in a reciprocally co-generative way. However, it is more difficult to determine whether the
second index is met: that the outputs from reciprocally co-generative collaboration are in fact novel and
cannot be produced by one partner alone. In fact, reciprocally co-generative interactions can produce
innovative outputs.
The Interactive mode requires a significant degree of turn-taking so that it allows each partner to
incorporate her partner’s understanding of the domain into her own thinking and to make more
frequent adjustments to her own mental model (Chi, 2000). That is, two students who take long turns
giving mini-lectures to each other, even if they are being Constructive, will likely not reap the same
benefits as two students who take multiple short turns interjecting to ask each other questions, make
clarifications, and so forth. The Interactive mode in ICAP refers to this kind of co-generative collaborative
dialogs.
Table 1
Heuristics of Two Indices to Operationally Define the ICAP Modes
ICAP Modes
2 Heuristic
Passive Active Constructive Interactive
Indices
What physical Orienting or Manipulating Generating Reciprocally co-
behaviors are attending behaviors behaviors generating
present? behaviors behaviors
What visible No visible Visible outputs Visible outputs Visible outputs
outputs (if any) outputs contain contain contain
are produced? produced information information that information that
provided in the goes beyond the goes beyond 1)
instructional existing the instructional
materials instructional material and 2) a
materials partner’s
contributions
Plausible cognitive or thinking processes
Storing new Activating, Inferring new Inferring new
information thereby knowledge knowledge and
strengthening building upon
relevant prior partner’s
knowledge knowledge
ICAP’s Main Hypothesis

The central hypothesis of the ICAP theory is that the levels of learning outcomes are predicted by the
ordering Interactive > Constructive > Active > Passive. That is, an Interactive activity leads to deeper and
more learning than a Constructive activity, and Active activities lead to more learning than Passive
97
activities. This hypothesis is based on both the subsuming hierarchy of the physical interactions and the
plausible hypothetical thinking processes underlying each mode. But the biggest improvement in
learning outcomes occurs between Active and Constructive modes. That is, when students are
Interactive or Constructive, they learn more deeply than when they are Active or Passive. Details on the
thinking process underlying cognitive engagement are described in Chi et al. (2018).
Two important points need to be noted about the ICAP hypothesis. The first is that the deeper learning
achievable in the Constructive and Interactive modes can only be detected if deeper assessment
questions are designed to test students’ understanding. Shallower questions are perfectly adequate to
detect learning from the Passive and Active modes. For example, students are more likely to
demonstrate generated knowledge when answering open-ended questions than answering multiple-
choice assessment items. The second important point to note is that ICAP predicts learning effectiveness
for activities across modes; ICAP cannot make accurate predictions for activities within the same mode.
This is because ICAP’s analyses of engagement are based on overt physical interactions and produced
outputs, indices that can be used in a classroom context. However, in order to make predictions of the
relative benefits of activities within the same mode, an analysis of the underlying cognitive processes is
required.
Empirical Support
The main hypothesis of the ICAP theory is supported by hundreds of laboratory and classroom studies
carried out by other investigators, often comparing two conditions. For many of these studies, we can
typically map the study’s manipulated and control conditions to an ICAP mode and see if the learning
effectiveness is consistent with ICAP’s pair-wise predictions. That is, I>C>A>P implies that pairwise
comparisons should be consistent with ICAP’s predicted descending direction, such as I > C, I > A, I > P, C
> A, and so forth. We describe a simple example from a study that contrasts two conditions to show how
they are easily mapped to two ICAP modes. Henderson and Canning (2013) examined the use of
Interactive activities in college students’ responses to conceptual physics questions that were
administered using classroom clickers. After controlling for all demographic and performance variables,
students who were given an opportunity to verbally discuss their clicker votes with each other
(Interactive) performed significantly better on the Force Concept Inventory than students who received
a supplemental lecture (Passive) between clicker votes. This is just one example of the numerous studies
reviewed in our publications (Chi, 2009; Chi et al., in press; Chi & Wylie, 2014; Fonesca & Chi, 2011). At
the college level, the ICAP hypothesis is consistent with a meta-analysis of 225 college classroom studies
(Freeman et al., 2014). Those 225 studies essentially showed that Interactive, Constructive, and Active
activities are all better than Passive activities of students listening to lectures only.
In conclusion, the strength of the ICAP framework is twofold. First, the framework provides a means of
differentiating between numerous “active learning” strategies outlined in the literature based on simple
heuristics using two indices. That is, ICAP divides “active instruction” or non-lecturing into three modes
and ranks them by effectiveness. Second, the theory applies across content domains, school subjects,
tasks, and is applicable for all age groups and individuals. Thus, ICAP can help instructors decide on class
activities and tasks that optimize students’ cognitive engagement and learning levels.
How to Use ICAP to Design Activities and Exercises, Choose Strategies

and Ask Questions
How can the distinctions pointed out by ICAP in terms of the two heuristic indices of each mode—
students’ physical interactions with instruction and students’ produced outputs (shown in Table 1)—
allow us to systematically translate the four modes (Interactive, Constructive, Active, and Passive) into
98
ways that we can apply ICAP toward the design of activities and exercises, choose learning strategies,
and frame how questions should be asked in class discussions.
How to Design Activities and Exercises Using Verbs Differentiated by ICAP
Activities or exercises undertaken by students are often explained through directives using verbs. The
type of verbs used to explain or describe what students need to do in an activity or exercise can play a
significant role in the level of student learning. We will refer to such explanations as “instructional
directives.” One of our recent ICAP studies (Chi et al., 2018) found that the type of verbs used by
teachers when giving directives significantly impacted the mode of students’ engagement and, thus,
their level of learning in the classroom. ICAP can classify the type of verbs used by teachers into ICAP’s
four different modes. For example, the two components of Passive mode require students to be paying
attention, and no external outputs are produced. Passive verbs are ones that lead students to enact
attentive behaviors, such as “listen to me,” “look at the board, or “watch the video.” In these directives,
the verbs only require students to pay attention to the task at hand, but do not require students to
produce any output. More examples of Passive verbs are shown in Table 2.
On the other hand, Active mode verbs should require students to engage in some behavior that
manipulates the course materials and requires an output of some kind that can be identified as from
within the course content. So, Active verbs should lead students to manipulative behaviors. Examples of
Active verbs include describe, match, and pick (see Table 2 for a list of ICAP verbs). Following the
operational definitions and heuristics of ICAP, Constructive verbs should lead student to enact
generative behaviors and produce outputs that contain new information. Examples of these ICAP verbs
include design, evaluate, explain, and paraphrase in your own words, as shown in Table 2.
It is important to remember that the biggest jump in learning level is from the Active to the Constructive
mode. Because of this, changing the verbs used in directives and worksheets from Active to Constructive
can be easily done and can significantly improve the student engagement mode and the level of
students’ learning outcomes. For example, suppose an instructor originally used a worksheet that was in
the Active mode because it asked students to match photos of a molecular model to the correct
chemical name. Match is an Active verb and only requires students to choose a photo from the ones
presented. However, the teacher could upgrade this activity to Constructive by asking students to draw
the molecular model based on the name of the chemical. Assuming that the students do not have access
to the image of the model in their textbook or notes, the students are now generating a new output not
already given in the course material. Thus, these students are engaging at a higher mode.
Finally, Interactive verbs should require students to engage in reciprocally co-generative behaviors and
produce outputs that are beyond both the course content and their peer’s contributions. Examples of
these phrases include “exchange ideas,” “debate with a peer,” “answer peer questions,” and “expand on
your peer’s reasoning.” These phrases can lead students to engage with a partner in a reciprocally co-
generative way. One important caveat about the Interactive mode is that instructors have a common
tendency to try to upgrade an activity to Interactive simply by adding the phrase “work with your
partner” to an Active or a Constructive activity. “Work with your partner” is a very general directive; it
does not specify how students should work with a partner in order to be reciprocally co-generative.
In sum, the fact that verbs can be differentiated into the four ICAP modes means that when designing an
exercise or activity for students to carry out, instructors can choose the verbs that correspond to the
mode of engagement (and its corresponding level of learning) that they wish to obtain from students.
99
Table 2
Classifying Verbs Using ICAP
Passive Verbs Active Verbs Constructive Verbs Interactive Verbs

Verbs that can lead Verbs that can lead Verbs that can lead Verbs that can lead
students to students to students to generative students to
attentive behaviors manipulative behaviors. collaborative
behaviors. behaviors
• Listen • Arrange • Analogize • Agree upon
• Look • Assign • Compose • Answer
• Observe • Classify • Decipher peer’s
• Read • Describe • Design questions
• View • Duplicate • Differentiate • Argue
• Watch • Fill in • Evaluate • Consult with
• Indicate • Explain • Critique
• Mark • Hypothesize • Debate
• Match • Invent • Discuss
• Number • Paraphrase • Exchange
ideas
• Pick out • Propose
• Point • Rationalize
How to Select Which Strategies to Use

Instead of designing one’s own activities, instructors often implement conventional activities and
strategies that have been used extensively in classrooms and explored in a multitude of research
studies. However, there has not been any systematic or theoretical way of differentiating the
comparative benefits of these strategies for learning. In principle, we can now examine a strategy by
analyzing the verbs or verb phrases used in the strategies in terms of their ICAP modes. But many of
these conventional and well-used strategies are described by some non-verb labels such as “clickers.”
Therefore, in order to know the ICAP mode of these strategies, we need to analyze what the students
are supposed to do in these activities in terms of ICAP’s heuristic indices of physical interactions and
outputs. In this section, we evaluate the ICAP mode of three common strategies by considering how the
students interact with instruction and what outputs they produced in their physical interactions, the
heuristic indices listed in Table 1.
The first example is a common classroom activity—fill-in-the blank worksheets (or guided notes, Konrad
et al., 2009; Kotsopoulos et al., 2022; Neef et al., 2006). Guided notes require students to write
down information that is being displayed during a lecture onto the provided worksheets with blanks,
which clearly indicate where and what to write. The physical interactions of the students are
manipulative because they are only copying information onto the worksheet that was already
presented. Therefore, they are not providing any new information outside of the given instructional
materials. Thus, this activity can be considered engaging in the Active mode.
But not all forms of notetaking are Active. For example, taking notes using the Cornell note-taking
format (Pauk & Owens, 2010, in which a structure is provided on each page, is a much more
Constructive activity. The structure provided by Cornell notes consist of segmenting a blank page with
100
different sections allocated for different goals, such as reserving the left column, which is about 30% of
the page, for students to write questions and main points. The right column, which is the other 70% of
the page, is reserved for students to take their primary notes from lecture or reading, which
corresponds to those questions or key terms. At the bottom of the page, students are asked to
summarize the main points and notes from the page (Quintus et al., 2012). It is obvious that Cornell
notes is an activity in the Constructive mode because it asks students to produce questions, main points,
and so forth. Thus, it is not surprising that taking Cornell notes is more effective for learning than taking
guided notes (Jacobs, 2008).
A second example is the “muddiest point” activity. The “muddiest point” learning strategy requires
students to reflect on a point of uncertainty existing in the instructional materials covered in that class
(Akhtar & Saeed, 2020). This requires students to think about what aspects of the instructional materials
were unclear, manifested in the physical behavior of pausing and reflecting on what they do not
understand, and then writing down what aspects of instruction were confusing. The output produced in
this instance is the articulated statement of the muddiest point. This muddiest point is likely not
explicitly stated in the instructional materials. Thus, this learning activity can be considered Constructive.
The third commonly used activity is clickers. Clickers can be used by students to respond to a question
the instructor posed, often by choosing one of four provided answer choices. The way clickers were
used originally and that showed substantial learning benefits (Hake, 1998), start with students in a class
choosing an answer from the four options. Once chosen, the instructor can display everyone’s combined
answers and determine whether the correct option was chosen by a majority of students, thereby
indicating if the students have understood the materials embedded in the question. If the responses
were distributed across various incorrect options, then the next step of using clickers is to ask students
to discuss with their neighbors, especially a neighbor who had chosen an opposing option. The students
are presumed to discuss and resolve their different responses. Carried out this way, using clickers is
definitely an Interactive mode of co-generative collaboration. Unfortunately, instructors often reduce
clickers to an Active mode activity by only having students choose a response and omitting the
collaborative discussion. Consistent with ICAP’s predictions, using clickers the correct way to include
collaborative activities (the Interactive mode) seems to be better for learning than without (the Active
mode; Chien et al., 2016; MacArthur & Jones, 2008). Thus, depending on how they are used, clickers can
be an Active or an Interactive activity.
The heuristics of analyzing what students are asked to do, and what they produce to determine the ICAP
mode of a strategy, can be applied to analyze almost any learning strategy for classifying it as either
Interactive, Constructive, Active or Passive. Table 3 below provides several additional examples of
learning strategies for each mode. A more complete list of learning strategies can be found in Chi et al.
(in press).
101
Table 3
Classifying Examples of Learning Strategies Using ICAP
Passive Active Constructive Interactive

• Listening (Feng & • Clickers and polling • Concept-mapping • Asking and
Webb, 2020) (Arthurs & Kreager, (Chang et al., 2002; answering each
• Observing and 2017) Novak, 1990a; other’s questions
watching and • Following step-by- 1990b) (Rivers et al., 2017);
looking (Cocco et al., step instructions • Explaining to or reciprocal
2021) (Liefooghe et al., another (Biswas et questioning (King,
• Reading and re- 2018). al., 2005; Roscoe & 1990a; 1990b).
reading (Rawson & • Guided notes with Chi, 2007) • Debating with a
Kintsch, 2005) fill-in-the-blank • Hypothesizing partner, offering
• Tactile learning (Kotsopoulos et al., explanations for a claims and
(Savaiano et al., 2022) set of data (Hulsizer justifications
2016) • Listing (generating a et al., 2018) (Schwarz et al.,
• Self-explaining and 2000)
• Turn-taking reading list) from memory
(since as a whole, (Epstein, 1969; elaborating (Chi et • Improv theater
each individual Roberts, 1972) al., 1989; 1994) (Rossing & Hoffman-
student only speaks • Mapping or aligning • Taking notes or Longtin, 2016)
aloud for a short two sets of given paraphrasing in own • Peer tutoring
portion) (Duffy, information (Yin et words; writing (Beasley, 1997)
1983) al., 2005). interpretive • Reciprocal learning
summary (Trafton & (Palinscar & Brown,
Trickett, 2001). 1984)
How to Ask Questions in the Classroom

Constructive questions should be the most common type of question asked in the classroom, but in
practice we often primarily see Active questions (Morris & Chi, 2020). The learning benefits of moving
from asking Active questions to Constructive questions indicate a need for teachers to prioritize
increasing the proportion of Constructive questions asked in the classroom in relation to Active
questions.
ICAP can also classify questions using the same heuristics as applied to our classification of verbs. Active
questions lead students to respond in a manner that only manipulates the course materials and not to
provide new information, such as asking student to recall or re-state information already provided (e.g.
“Describe what we talked about yesterday,”) or asking students to choose an option from a set of
provided options or to make a perceptual comparison or judgment of what is presented (e.g., such as
“Can you identify which one is bigger?”), or asking students to do a simple calculation (e.g., “Can you tell
me the number of mice in location A?”). Retrieving and carrying out an already known computation is
Active.
On the other hand, Constructive questions ask students to generate beyond the given materials, often
requiring students to infer information or to justify their explanation (e.g., “What evidence do you have
to support that?”), or requiring students to make a connection or link between given course materials
(e.g. “How does this concept relate to what we learned yesterday?”), or requiring students to make a
prediction based on the given information or perhaps by combining the given information with some
102
prior knowledge to make an educated inference (e.g., “What do you think will happen to the mice that
have the darker color mutation?”). Commonly, Constructive questions require students to predict,
justify, explain, plan, or make connections in their responses.
We have identified a few types of Passive questions from middle school teachers (Morris & Chi, 2020).
Passive questions do not expect students to answer the question, besides perhaps a shake of the head.
For example, rhetorical questions (e.g., “Does this make sense?”) are often asked by teachers who are
taking a brief pause to “check in” before moving onto the next part of the lesson, without waiting to
receive a response. Another common Passive question occurs when a teacher asks a question, but then
immediately answers the question—essentially turning the question into lecture (e.g., “How many
polygons are there? Four, right?”). Passive questions should be avoided in the classroom because they
do not offer substantial value to students’ engagement or learning.
Interactive questions are questions that elicit students’ interaction with their peers or other
collaborative behaviors (e.g., “Does anyone disagree with Student A, why or why not?”; “What do you
think about Student B’s opinion?”; “How can you defend Student C’s answer?”). Interactive question-
asking often occurs when teachers are helping to guide a group discussion or debate in a facilitated
manner. When asking Interactive and Constructive questions, it is important that instructors allow
sufficient time for students to think and answer the questions because generating new ideas tend to
require more think-time than Active responses.
Examples of Upgrades from Post-secondary and K-12 Classes

We describe briefly two examples of how two college instructors (Professor Semken and Professor
Krause at Arizona State University) chose to upgrade their course activities, and one example from a K-
12 class, from Active to Constructive,
Professor S. Semken (personal communication, October 17, 2019) teaches a geology course every
semester which originally consisted of two, short in-class activities during each class meeting. In one
instance of this in-class activity, students were asked to match sedimentary depositional environments
and sediments to the correct rock type. Matching is an Active activity because students only manipulate
the existing course content. After ICAP-upgrading his course, the same in-class activity was changed so
students were asked to: 1) identify (Active) the sedimentary rock specimen, 2) predict (Constructive)
what type of environment that rock could be found in, and 3) then interpret (Constructive) which
environment matches their own prediction for each rock. While the activity still includes some Active
components, the overall ICAP mode of the task moved from Active to Constructive.
Our second example comes from a class taught by Professor Krause in an undergraduate engineering
course. Originally, one type of activity Professor Krause gave his students required them to match the
property and change, unit cell transformation, condition for change, and original processing method—
which were each provided in content banks organized by each topic—to five provided objects. In the
upgraded version, the students were provided with the same five objects and their corresponding
conditions for change. Based on this information, for each object, students had to 1) state and explain
the properties and change structure, 2) draw the unit cell transformation, and 3) state and explain the
original processing method. Because students were now generating answers and drawing the molecular
structures rather than matching the same information, students were engaging in the Constructive
mode rather than the Active mode (Menekse et al., 2013).
The third activity is a pre-college ICAP upgrade example. In this math class, the instructor improved a
worksheet on fractions from Active to Constructive. The original worksheet provided students with tiles
that had written on them various fractions, decimals, and percentages. For example, the tiles said .75,
103
3/4, 75%. The students had to organize the tiles so all of the values that represented the same numerical
amount were grouped together. When the worksheet was upgraded, the tiles were removed. Instead,
students were given one value (3/4, .8, 9%, etc.) and had to generate various different versions of the
given decimal or fraction. For example, a student sees 3/4 written on the worksheet and then needs to
generate different ways to represent the same numerical amount.
ICAP in Practice: Issues and Challenges

We discuss several caveats and challenges that can arise as instructors think about implementing
activities, guided by ICAP, to improve student learning.
Deciding on Which ICAP Mode to Implement
Although ICAP emphasizes the importance of learning in the Interactive and Constructive modes, not all-
learning objectives require a high-level of cognitive engagement. For example, when students need to
memorize a set of procedural steps or memorize associated information, Active is sufficient (e.g.,
learning the name of a word in another language). This also applies to some domains for which no
generative inferences, justifications, or reasoning can be provided, such as the rules of grammar. In
learning such domains, such as formal logic, activities in the manipulative Active mode is adequate.
Moreover, there is some interesting evidence to show that sequencing of different engagement modes
might make the Passive mode more effective. For example, the Passive mode (lecturing) can be valuable
to learning if a Passive activity comes after a Constructive activity, such as having students struggle to
solve a problem first and then listening to an instructor’s explanation (Kapur, 2012; Schwartz &
Bransford, 1998; Kapur & Roll, this volume). One explanation might be that by allowing students to be
generative first, before hearing a lecture, provides the opportunity for students to figure out what gaps
they have and what they misunderstand, which can then be filled in or corrected by the subsequent
lecture. Not enough research exists to deeply explore various sequencing effects of one mode followed
by another mode of activity.
Ambiguity of a Mode
Sometimes it is somewhat ambiguous which ICAP mode an activity is. There are several reasons for such
ambiguity, such as students’ prior knowledge, additional conditions or constraints, and activities that are
simply ambiguous between modes.
Ambiguity in terms of students’ prior knowledge can be illustrated by the task of problem solving.
Suppose an instructor assigns a set of problems for her students to solve. If a student already knows
how to solve the assigned problems and she is simply applying the steps in a methodical manner, then
for her this problem-solving task is only Active because she is only recalling information she has learned
from instruction and not producing any new information. However, if the problem is considered a
transfer problem, and the student does not know how to solve the problem without making further
reasoning, then the student is generating changes to a known procedure to adapt the procedure for this
new type of problem. Then the student would be generative and in the Constructive mode. Thus, the
ICAP mode of a problem is individualized to the students’ prior knowledge. A simple solution is to
provide a range of problems in difficulty.
A second way that an activity can be ambiguous is when additional conditions or constraints are needed
for accurate classification. A good example is sorting items into categories. If a student is given a set of
items to sort, with the categories to sort them into, then this is an Active task because all of the
materials are given, and students are merely manipulating the to-be-sorted items into categories. On
104
the other hand, if students are given a set of items and asked to sort them into categories that they have
to create, then this is clearly a Constructive activity.
Ambiguity can also occur for some activities that are genuinely difficult to classify, because they have
characteristics of two modes, such as asking students to read out-loud an entire passage. Although this
seems to be an Active activity because students are articulating the words on a page, it also seems
Passive because it lacks the re-focusing opportunities to manipulate portions of instruction, as provided
by other Active activities, since the entire passage is being read out-loud rather than a selected portion.
The ambiguity in the modes for such examples can be easily addressed by considering an activity as
falling in between two modes (in this reading case, between Passive and Active). In practice, instructor
need not worry about the precise mode of an ambiguous activity. Rather, instructors should focus on
upgrading the mode of an ambiguous activity, such as from an in-between Passive-Active activity to a
Constructive activity.
Students’ Compliance
Another challenge in practicing ICAP includes student compliance issues. That is, students themselves
can downgrade a higher-mode activity—intentionally or unintentionally. For example, if an instructor
asks her students to summarize a text in their own words (a Constructive activity), a student may instead
use the copy-paste method of summarizing (which is an Active activity). In these instances, an instructor
can check compliance by examining the student-produced output. Unfortunately, ICAP cannot handle
the compliance issues associated with students not following directions.
However, ICAP underscores the findings in Chi et al. (2018) that even with frequent non-compliance,
giving students higher ICAP mode activities improves the odds that students are more likely to respond
within the higher modes of cognitive engagement and should improve the students learning overall. It is
safe to assume that a majority of the students are likely to comply, or that the individual students might
comply a majority of the times.
Conclusion
ICAP is a theory that can define and differentiate active learning and recommend how to put these
classifications into practice, such as designing in-class and homework activities, selecting learning
strategies, and framing questions. Moreover, ICAP can also be used in myriad other ways that are not
discussed in this chapter, such as evaluating videos, digital tools, and lesson plans. In short, ICAP allows
instructors to know how to evaluate their own classrooms and how to apply ICAP in their classrooms, so
that they will be better able to guide their students toward higher cognitive engagement and,
ultimately, higher learning gains.
Author Note
The research reported here was supported by the Institute of Education Sciences, U.S. Department of
Education, through grants R305A110090 and R305A150432 to Michelene T. H. Chi at Arizona State
University. The opinions expressed are those of the authors and do not represent views of the Institute
or the U.S. Department of Education.
References
Akhtar, M. & Saeed, M. (2020). Effect of Frayer model, choral response and muddiest-point on students’
academic achievement. Journal of Educational Science and Research, 7(2), 71-86.
105
Alibali, M., & DiRusso, A. (1999). The function of gesture in learning to count: More than keeping track.
Cognitive Development, 14(1), 37–56. https://doi.org/10.1016/S0885-2014(99)80017-3
Arthurs, L. A. & Kreager, B. Z. (2017). An integrative review of in-class activities that enable active
learning in college science classroom settings. International Journal of Science Education, 39(15),
2073-2091. https://doi.org/10.1080/09500693.2017.1363925
Azevedo, R., Greene, J. A., Moos, D. C., Winters, F. I., Cromley, J. G., & Godbole-Chaudhuri, P. (2006). Is
externally-regulated learning by a human tutor always effective in facilitating learning with
hypermedia? In S. Barab, K. Hay, & D. Hickey (Eds.), Proceedings of the 7th International
Conference on Learning Sciences (pp. 16–22). Erlbaum.
Beasley, C. (1997). Students as teachers: The benefits of peer tutoring. Learning Through Teaching, 21-
30.
Biswas, G., Leelawong, K., Schwartz, D., & Vye, N. (2005). Learning by teaching a new agent paradigm for
education software. Applied Artificial Intelligence, 19(3/4), 363–392.
Bloom, B. S., & Krathwohl, D. (1956). Taxonomy of educational objectives: The classification of
educational goals, by a committee of college and university examiners. Handbook I: Cognitive
Domain. Longman.
Bodemer, D., Ploetzner, R., Feuerlein, I., & Spada, H. (2004). The active integration of information during
learning with dynamic and interactive visualizations. Learning and Instruction, 14(3), 325–341.
https://doi.org/10.1016/j.learninstruc.2004.06.006
Britt, M. A. & Sommer, J. (2004). Facilitating textual integration with macro-structure focusing task.
Reading Psychology, 25(4), 313-339. https://doi.org/10.1080/02702710490522658
Butcher, K. R. (2006). Learning from text with diagrams: Promoting mental model development and
inference generation. Journal of Educational Psychology, 98(1), 182–197.
Chang, K.-E., Sung, Y.-T., and Chen, I.-D. (2002). The effect of concept mapping to enhance text
comprehension and summarization. The Journal of Experimental Education, 71(1), 5–23.
Chi, M. T. H. (2000). Self-explaining expository texts: The dual processes of generating inferences and
repairing mental models. In R. Glaser (Ed.), Advances in Instructional Psychology (pp. 161-238).
Lawrence Erlbaum Associates, Inc.
Chi, M. T. H. (2009). Active-constructive-interactive: A conceptual framework for differentiating learning
activities. Topics in Cognitive Science, 1(1), 73-105. https://doi.org/10.1111/j.1756-
8765.2008.01005.x
Chi, M. T. H., & Wylie, R. (2014). The ICAP framework: Linking cognitive engagement to active learning
outcomes. Educational Psychologist, 49(4), 219-243.
Chi, M. T. H., Bassok, M., Lewis, M. W., Reimann, P., & Glaser, R. (1989). Self–explanations: How
students study and use examples in learning to solve problems. Cognitive Science, 13(2), 145–
182. https://doi.org/10.1207/s15516709cog1302_1
106
Chi, M. T. H., Boucher, N., & Ha, J. (in press). The efficacy of learning strategies from the ICAP
perspective. In R. Tierney, F. Rizvi, & K. Ericikan (Eds.) International Encyclopedia of Education
(4th edition).
Chi, M. T. H., de Leeuw, N., Chiu, M. H., & LaVancher, C. (1994). Eliciting self-explanations improves
understanding. Cognitive Science, 18(3), 439–477.
https://doi.org/10.1207/s15516709cog1803_3
Chi, M. T. H., Roy, M., & Hausmann, R. G. M. (2008). Observing tutorial dialogues collaboratively:
Insights about human tutoring effectiveness from vicarious learning. Cognitive Science, 32(2),
Chi, M.T.H., Adams, J., Bogusch, E.B., Bruchok, C., Kang, S., Lancaster, M., Levy, R., McEldoon, K., Stump,
G.S., Wylie, R., Xu, D., & Yaghmourian, D.L. (2018). Translating the ICAP theory of cognitive
engagement into practice. Cognitive Science, 42(6), 1777-1832.
Chien, Y. T., Chang, Y. H., & Chang, C. Y. (2016). Do we click in the right way? A meta-analytic review of
clicker-integrated instruction. Educational Research Review, 17, 1-18.
https://doi.org/10.1016/j.edurev.2015.10.003
Chinn, C. A., & Malhotra, B. A. (2002). Epistemologically authentic inquiry in schools: A theoretical
framework for evaluating inquiry tasks. Science Education, 86(2), 175-218.
https://doi.org/10.1002/sce.10001
Cocco, V.M., Bisagno, E., Di Bernardo, G.A., Cadamuro, A., Riboldi, S.D., Crapolicchio, E., Trifiletti, E.,
Stathi, S. & Vezzali, L. (2021). Comparing story reading and video watching as two distinct forms
of vicarious contact: An experimental intervention among elementary school children. British
Journal of Social Psychology, 60(1), 74-94. https://doi.org/10.1111/bjso.12404
Conati, C. & VanLehn, K. (2000). Toward computer-based support of meta-cognitive skills: A
computational framework to coach self-explanation. International Journal of Artificial
Intelligence in Education, 11, 389-415. https://doi.org/10.1007/s40593-015-0074-8
Dawson, I. (2004). Time for chronology? Ideas for developing chronological understanding. Teaching
History, 117, 14–24.
Dick, W. & Carey, S. (2005). The systematic design of instruction (6th edition). Allyn and Bacon.
Duffy, G. G. (1983). First in series-factor related to reading performance from turn taking to sense
making broadening the concept of reading teacher effectiveness. The Journal of Educational
Research, 76(3), 134-139. https://doi.org/10.1080/00220671.1983.10885438
Epstein, W. (1969). Recall of word lists following learning of sentences and of anomalous and random
strings. Journal of Verbal Learning and Verbal Behavior, 8(1), 20-25.
https://doi.org/10.1016/S0022-5371(69)80005-8
Feng, Y., & Webb, S. (2020). Learning vocabulary through reading, listening, and viewing: Which mode of
input is most effective? Studies in Second Language Acquisition, 42(3), 499-523.
https://doi.org/10.1017/S0272263119000494
Fonesca, B. & Chi, M. T. H. (2011). The self-explanation effect: A constructive learning activity. In R.
Mayer, & P. Alexander (Eds.), The Handbook of Research on Learning and Instruction (pp. 270-
321). Routledge Press.
107
mathematics. Proceedings of the National Academy of Sciences, 111(23), 8410-8415.
Graesser, A. C., & Person, N. K. (1994). Question asking during tutoring. American Educational Research
Journal, 31(1), 104–137. https://doi.org/10.3102/00028312031001104
Hake, R. R. (1998). Interactive-engagement versus traditional methods: A six-thousand-student survey of
mechanics test data for introductory physics courses. American journal of Physics, 66(1), 64-74.
https://doi.org/10.1119/1.18809
Henderson, J.B. & Canning, B. (2013, February). Brain candy: Utilizing technology and peer learning to
improve science learning. Paper presented at the 2013 International Teacher-Scientist
Partnership Conference (ITSPC), Boston, MA.
Hogan, K., Nastasi, B. K., & Pressley, M. (1999). Discourse patterns and collaborative scientific reasoning
in peer and teacher-guided discussions. Cognition and Instruction, 17(4), 379–432.
https://doi.org/10.1207/S1532690XCI1704_2
Hulsizer, H., Huq, A., & Wei, W. (2018). Early introduction of hypothesis testing in introductory statistics:
A pilot study. The Online Journal of New Horizons in Education, 8(3), 10-19.
Jacobs, K. (2008). A comparison of two note taking methods in a secondary English classroom. In
Proceedings: 4th Annual Symposium: Graduate Research and Scholarly Projects. Witchita State
University, p. 119-120. http://hdl.handle.net/10057/1388
Kapur, M. (2012). Productive failure in learning the concept of variance. Instructional Science, 40(4), 651-
672. https://doi.org/10.1007/s11251-012-9209-6
Katayama, A. D., Shambaugh, R. N., & Doctor, T. (2005). Promoting knowledge transfer with electronic
note taking. Teaching of Psychology, 32(2), 129–131.
https://doi.org/10.1207/s15328023top3202_9
King, A. (1990a). Enhancing peer interaction and learning in the classroom through reciprocal
questioning. American Educational Research Journal, 27(4), 664-687.
King, A. (1990b). Reciprocal peer-questioning a strategy for teaching students how to learn from
lectures. The Clearing House, 64(2), 131-135. https://doi.org/10.1080/00098655.1990.9955828
Konrad, M., Joseph, L. M., & Eveleigh, E. (2009). A meta-analytic review of guided notes. Education and
Treatment of Children, 32(3), 421-444. https://doi.org/10.1353/etc.0.0066
Kotsopoulos, D., Weatherby, C., & Woolford, D. G. (2022). Using guided notes to support learning in
first-year calculus. International Journal of Mathematics Education in Science and Technology,
53(6), 1629-1644. https://doi.org/10.1080/0020739X.2021.1910742
Krathwohl, D. (2002). A revision of Bloom’s taxonomy: An overview. Theory into Practice, 41(4), 212-218.
https://doi.org/10.1207/s15430421tip4104_2
Liefooghe, B., Braem, S., & Meiran, N. (2018). The implications and applications of learning via
instructions. Acta Psychologica, 184, 1-3. https://doi.org/10.1016/j.actpsy.2017.09.015
108
MacArthur, J. R., & Jones, L. L. (2008). A review of literature reports of clickers applicable to college
chemistry classrooms. Chemistry Education Research and Practice, 9(3), 187-195.
https://doi.org/10.1039/B812407H
Marks, H. M. (2000). Student engagement in instructional activity: Patterns in the elementary, middle,
and high school years. American Educational Research Journal, 37(1), 153–184.
Menekse, M., Stump, G. S., Krause, S., & Chi, M. T. (2013). Differentiated overt learning activities for
effective instruction in engineering classrooms. Journal of Engineering Education, 102(3), 346-
374. https://doi.org/10.1002/jee.20021
Morris, J., & Chi, M. T. H. (2020). Improving teacher questioning in science using ICAP theory. The
Journal of Educational Research, 113(1), 1-12. https://doi.org/10.1080/00220671.2019.1709401
Neef, N. A., McCord, B. E., & Ferreri, S. J. (2006). Effects of guided notes versus completed notes during
lectures on college students' quiz performance. Journal of Applied Behavior Analysis, 39(1), 123-
130. https://doi.org/10.1901%2Fjaba.2006.94-04
Novak, J. D. (1990a). Concept mapping: A useful tool for science education. Journal of Research in
Science Teaching, 27(10), 937-949. https://doi.org/10.1002/tea.3660271003
Novak, J. D. (1990b). Concept maps and Vee diagrams: Two metacognitive tools to facilitate meaningful
learning. Instructional Science, 19(1), 29-52. https://doi.org/10.1007/BF00377984
Okada, T., & Simon, H. A. (1997). Collaborative discovery in a scientific domain. Cognitive Science, 21(2),
109-146. https://doi.org/10.1016/S0364-0213(99)80020-2
Pauk, W., & Owens, R. J. Q. (2010). How to study in college (10 ed.). Chapter 10. (pp. 235-277).
Wadsworth.
Palinscar, A.S. & Brown, A.L. (1984). Reciprocal teaching of comprehension-fostering and
comprehension-monitoring activities. Cognition and Instruction, 1(2), 117-175.
https://doi.org/10.1207/s1532690xci0102_1
Quintus, L., Borr, M., Duffield, S., Napoleon, L., & Welch, A. (2012). The Impact of the Cornell Note-
Taking Method on Students’ Performance in a High School Family and Consumer Sciences
Class. Journal of Family & Consumer Sciences Education, 30(1), 27–38.
Rawson, K.A. & Kintsch, W., (2005). Rereading effects depend on time of test. Journal of Educational
Psychology, 97(1), 70. https://doi.org/10.1037/0022-0663.97.1.70
Rivers, J., Smith, A., Higgins, D., Mills, R., Maier, A. G., & Howitt, S. M. (2017). Asking and answering
questions: Partners, peer learning, and participation. International Journal for Students as
Partners, 1(1). https://doi.org/10.15173/ijsap.v1i1.3072
Roberts, W. A. (1972). Free recall of word lists varying in length and rate of presentation: A test of total-
time hypotheses. Journal of Experimental Psychology, 92(3), 365-372.
Roscoe, R. D., & Chi, M. T. H. (2007). Understanding tutor learning: Knowledge-building and knowledge-
telling in peer tutors’ explanations and questions. Review of Educational Research, 77(4), 534–
109
Rossing, J. P. & Hoffmann-Longtin, K. (2016). Improv(ing) the academy: Applied improvisation as a
strategy of educational development. To Improve the Academy, 35(2), 303-325.
https://doi.org/10.1002/tia2.20044
Savaiano, M.E., Compton, D.L., Hatton, D.D. & Lloyd, B.P. (2016). Vocabulary word instruction for
students who read braille. Exceptional Children, 82(3), 337-353.
http://digitalcommons.unl.edu/specedfacpub/96
Schauble, L., Glaser, R., Duschl, R. A., Schulze, S., & John, J. (1995). Students’ understanding of the
objectives and procedures of experimentation in the science classroom. Journal of the Learning
Sciences, 4(2), 131–166. https://doi.org/10.1207/s15327809jls0402_1
Schwartz, D. L., & Bransford, J. D. (1998). A time for telling. Cognition and Instruction, 16(4), 475-523.
Schwarz, B. B., Neuman, Y., & Biezuner, S. (2000). Two wrongs may make a right ... if they argue
together! Cognition and Instruction, 18(4), 461–494.
https://doi.org/10.1207/S1532690XCI1804_2
Skinner, E. A., & Belmont, M. J. (1993). Motivation in the classroom: Reciprocal effects of teacher
behavior and student engagement across the school year. Journal of Educational Psychology,
85(4), 571. https://doi.org/10.1037/0022-0663.85.4.571
Suthers, D. D., & Hundhausen, C. D. (2003). An experimental study of the effects of representational
guidance on collaborative learning processes. Journal of the Learning Sciences, 12(2), 183–218.
https://doi.org/10.1207/S15327809JLS1202_2
Trafton, J. G., & Trickett, S. B. (2001). Note-taking for self-explanation and problem solving. Human–
Computer Interaction, 16(1), 1–38. https://doi.org/10.1207/S15327051HCI1601_1
VanLehn, K., Graesser, A. C., Jackson, G. T., Jordan, P., Olney, A., & Rosé, C. P. (2007). When are tutorial
dialogues more effective than reading? Cognitive Science, 31(1), 3–62.
Webb, N. M. (1989). Peer interaction, problem solving, and cognition: Multidisciplinary perspectives.
International Journal of Educational Research, 13(1), 1–119. https://doi.org/10.1016/0883-
0355(89)90009-8
Yaron, D., Karabinos, M., Lange, D., Greeno, J. G., & Leinhardt, G. (2010). The ChemCollective—virtual
labs for introductory chemistry courses. Science, 328(5978), 584–585.
https://doi.org/10.1126/science.1182435
Yin, Y., Vanides, J., Ruiz-Primo, M. A., Ayala, C. C., & Shavelson, R. J. (2005). Comparison of two concept-
mapping techniques: Implications for scoring, interpretation, and use. Journal of Research in
110
In Their Own Words Spaced and Interleaved Mathematics Practice

Doug Rohrer and Marissa K. Hartwig
You to Know About Why and How University of South Florida
Editors
Lauren L. Kordonowy
Victor A. Benassi
Education experts have given considerable advice on how students should best learn, but many of these
recommended interventions are ineffective. For instance, schools spend small fortunes on computers
for their students despite the lack of evidence for computer-based instruction (Roediger & Pyc, 2012).
Other interventions are ostensibly supported by empirical research, but a closer look reveals either
flawed research designs or one-off findings that do not replicate (Kirschner & van Merriënboer, 2013).
Still other learning interventions have no empirical support and are instead solely based on platitudes
and armchair theories (Robinson & Levin, 2019). Fortunately, researchers have identified several
effective learning strategies that have consistently proven effective in both the laboratory and the
classroom (e.g., Carpenter, 2014; Dunlosky et al., 2014; Pan, 2015).
Two of these are mathematics learning interventions that are the focus of this chapter, and each has
produced test benefits in randomized studies. The benefits were found with students between first
grade and college, and the learning materials were drawn from a variety of topics, including arithmetic,
algebra, and geometry. Studies took place in either the laboratory or classroom, and the delay between
practice and test often exceeded several weeks – long enough to be educationally meaningful.
The two interventions are also feasible. Neither requires teacher training, and each allows teachers to
present material as they ordinarily would. The interventions can be used in class or at home, with or
without computers, and each is free. In this chapter, we describe the interventions, briefly review the
evidence for each, and list recommendations on how mathematics teachers can implement the
interventions in their courses.
The Interventions: Spaced and Interleaved Practice

Most mathematics textbooks and courses are divided into many short lessons, each followed by a group
of practice problems devoted to that lesson – an approach known as blocked practice. For instance, a
lesson on parabolas is usually followed by a block of parabola problems. Of course, not every problem in
the set necessarily takes the same form. Some parabola problems require only an understanding of
procedures, whereas other problems might demand a deeper conceptual understanding. Nevertheless,
every problem in the assignment relates to a parabola, and students usually realize this before they
begin the assignment, often because they have just seen a lesson on parabolas and several worked
examples. To be sure, not all assignments are blocked, as most textbooks include periodic review
assignments, but even the review assignments usually consist of small blocks of problems. For instance,
a chapter review assignment typically begins with several problems relating to the first lesson of the
chapter, followed by another small block of problems relating to the second lesson, and so forth. In fact,
the majority of problems in most mathematics textbooks can be solved by the same strategy used to
solve the immediately preceding problem, as we detail further below. In short, it appears that the
111
majority of mathematics students devote most of their practice effort to working similar kinds of
problems in immediate succession.
In an alternative approach that we endorse, a portion of the practice problems in the course are
rearranged so that assignments incorporate the two interventions espoused in this chapter. In the first
of these interventions, practice problems relating to a particular skill or concept – say, parabolas –
should be distributed or spaced across many assignments spanning a long period of time. For example,
whereas a lesson on parabolas is ordinarily followed by a large block of parabola problems that
comprises the majority of the parabola problems in the course, we instead recommend that a majority
of the parabola problems be spaced (Figure 1).
Figure 1
Spaced Practice
A. Lightly Spaced Practice (typical approach)
Unit 3 Course
1 2 3 4 Review Review
1
Practice 2
Problems 3
4
Most parabola 5
problems appear 6
immediately after
7
the lesson on
parabolas (3-2). 8
9
10
B. Heavily Spaced Practice (recommended)

Unit 3 Unit 4 Unit 5 Course
1 2 3 4 Review 1 3 4 Review
1
2
Practice
3
Problems
4
Parabola 5
problems are
6
distributed
across many 7
assignments. 8
9
10
Note. Blue disks represent parabola problems.
112
By the second intervention recommended here, most of the practice problems in a course should be
mixed, or interleaved, with different kinds of problems, rather than grouped together into a block of
problems requiring the same strategy. For instance, rather than arranging problems so that nearly every
circumference problem is preceded by another circumference problem, problems should be arranged so
that most of the circumference problems are followed immediately by an unrelated problem, such as a
problem about circle area or a problem unrelated to circles. To be clear, at least some blocked practice
might be beneficial when students first encounter a new skill or concept, but the data, described in the
next section, suggest that a large proportion of problems should be interleaved rather than blocked
(Figure 2).
Figure 2
Interleaved Practice
A. Lightly Interleaved Practice (typical approach)
Unit 3 Course
1 ooË
Practice 2 Ëo
Problems 3 o o
4 oË
lesson is
Each lesson is followed
followed byby 5 Ë o
block of
a block of problems
problems
6 o
devotedto
devoted tothat
that lesson,
lesson,
and unit reviews
reviewsusually
usually 7 ËË
include
includesmall
smallblocks.
blocks. 8 oË
9 oË
10 ËË
 Unit 1 concepts
B. Mostly Interleaved Practice (recommended)  Unit 2 concepts
 Unit 3 concepts
Unit 3 Course
1 ooË
Practice 2 Ëo
Problems
3 o o
4 oË
Each lesson is followed by a 5 Ë o
small block of problems
and several interleaved
6 Ë o Ë o o
problems drawn from prior 7 o Ë Ë ËË
lessons or last school year. 8 o o oË
9 Ë o o oË
10 o Ë Ë ËË
Note. Each symbol represents a unique kind of problem. For example, an orange disk and a blue disk
represent two different kinds of problems introduced in Unit 3, whereas an orange square represents yet
another kind of problem introduced in a previous unit.
113
Spaced practice and interleaved practice are similar interventions, but there is a critical distinction
between the two. Spaced practice describes the arrangement of a single kind of problem (e.g.,
parabolas), whereas interleaved practice describes the arrangement of multiple kinds of problems. In
fact, it is possible that problems of a single kind could be spread across time (spaced) without appearing
in an assignment that mixes multiple kinds of math problems (not interleaved). That said, the
interleaving of practice problems guarantees that problems are spaced as well, meaning that the
recommendation to interleave is inherently a recommendation to both space and interleave.
The benefits of interleaving are likely due, at least in part, to spacing, but interleaving provides another
benefit that is illustrated by the following problem:
A bug crawls 24 cm east and then 7 cm north. How far is the bug from where it started?
The problem is solved by the Pythagorean Theorem (242 + 72 = c2, so c = 25 cm), and students must
choose this strategy (Pythagorean Theorem) before they can use the strategy. Yet this choice of strategy
is not obvious, because the problem excludes cues such as triangle or hypotenuse. Throughout much of
mathematics, in fact, the choice of an appropriate strategy is more difficult than it might first seem
because problems that look alike sometimes require different strategies (Figure 3).
Figure 3
Strategy Choice. Students must discriminate between superficially similar problems.
Problem Strategy
Arithmetic
Jo ran ¼ km. Mo ran ½ km farther than Jo ran. How far did Mo run? Addition
Jo ran ¼ km. Mo ran ½ as far as Jo ran. How far did Mo run? Multiplication
Algebra
Solve. x2 – x – 1 = 0 Quadratic Formula
3
Solve. x – x = 0 Factor
Calculus
∫x(e + 1)x dx Integration by Parts
∫e(x + 1)e dx U-Substitution
Although mathematics students must learn to choose an appropriate strategy, they do not have a
chance to practice doing so when every problem in an assignment can be solved with the same strategy.
They can instead safely assume that the problem relates to the same skill or concept as the immediately
preceding problem. In effect, blocked practice is a kind of scaffolding that has utility, especially when
students first encounter a new skill or concept, but blocked practice is not sufficient. Students must
ultimately learn to choose a strategy on the basis of the problem itself, just as they must do when they
sit for a cumulative exam or other high-stakes test (Higgins, 2019; Rohrer, 2012). In short, interleaved
practice provides students with an opportunity to practice doing what they are expected to know.
Evidence for Spaced and Interleaved Mathematics Practice

The spacing effect is arguably the largest and most robust effect known to learning scientists, and
several randomized studies have shown that spacing improves mathematics learning. The earliest
mathematics spacing studies were conducted in the laboratory with college students (e.g., Gay, 1973;
114
Rohrer & Taylor, 2006, 2007). More recently, the benefits of spacing mathematics were shown in several
randomized studies in the classroom, including one with third-grade students learning multiplication
and seventh-grade students learning probability (Barzagar Nazari & Ebersbach, 2019), students in a
college statistics course (Ebersbach & Barzagar Nazari, 2020), UK students in Year 7 learning about
permutations and sets (Emeny et al., 2021), young children learning arithmetic facts (Schutte et al.,
2015), and three studies with college mathematics students (Butler et al., 2014; Hopkins et al., 2016;
Lyle et al., 2020).
A review of this literature is beyond the scope of this chapter, but for the sake of illustration, we
describe the college classroom study by Lyle et al. (2020). This study was fully embedded within a pre-
calculus course, and the practice problems related to several of the course objectives (e.g., simplifying
radical expressions). Students were randomly assigned to one of two conditions (Figure 4). In the
Baseline condition, practice was spaced to a small degree, as is ordinarily the case. For instance, each
kind of problem is seen during lectures, practice assignments, and exams, and thus, these exposures are
naturally spaced to some degree. The Increased Spacing condition was identical to the Baseline
condition with one exception: the practice problems relating to a particular learning objective were
distributed across three sessions spaced one week apart. Critically, the students in both groups worked
the same problems – only the practice schedule varied. The results showed that students who received
Increased Spacing, in comparison to the Baseline, scored significantly higher on both the final exam and
a follow-up exam given four weeks later.
Figure 4
Classroom Study of Spaced Practice (Lyle et al., 2020)
Increased Spacing
Final Follow-Up
⎯⎯→
Exam Exam
Baseline 4 weeks
⎯⎯⎯ 15-week course ⎯⎯⎯→
Apart from these mathematics spacing studies, still other randomized studies have found test score
benefits of interleaving. These studies include both laboratory experiments with college students (e.g.,
Foster et al., 2019; Rohrer & Taylor, 2007; Sana et al., 2017) and classroom studies with college students
needing remedial mathematics instruction (Mayfield & Chase, 2002), fourth-grade students learning
geometry (Taylor & Rohrer, 2010), fifth- and sixth-grade students learning fractions (Rau et al., 2013),
seventh-grade students learning elementary algebra (Rohrer et al., 2014, 2015, 2020b), and college
students in quantitative science courses such as chemistry (Eglington & Kang, 2017) and physics (Samani
& Pan, 2021). Moreover, in many of the studies, a greater dose of interleaving produced particularly
large test benefits.
In the study by Rohrer et al. (2020b), for instance, each of 54 seventh-grade classrooms was randomly
assigned to one of two groups (Figure 5). One group received worksheets providing mostly blocked
practice, and the other received worksheets providing mostly interleaved practice. Every student saw
the same practice problems; only their order varied. Students received these assignments periodically
for several months, and then every student completed the same (interleaved) review assignment. One
115
month later, students took an unannounced test, and the results showed that the higher dose of
interleaved practice produced much higher test scores, 61% vs. 38%.
Figure 5
Classroom Study of Interleaved Practice (Rohrer et al., 2020b)
Mostly Interleaved Practice

Review Unannounced
⎯⎯→ ⎯⎯→
Assignment Test
Mostly Blocked Practice 10 33
days days
⎯⎯⎯ 103 days ⎯⎯⎯→
In short, strong evidence shows that both spaced practice and interleaved practice can dramatically
improve test scores in both the laboratory and classroom. Both effects are robust, too, as benefits have
been found with a variety of student populations and with diverse kinds of materials. For these reasons,
spacing and interleaving have been endorsed by numerous learning scientists (e.g., Bahrick et al, 1993;
Bjork & Bjork, 2019; Carpenter, 2014; Dempster, 1988; Dunlosky et al., 2013; Kang, 2016; Pan, 2015;
Pashler et al., 2007; Rohrer & Hartwig, 2020; Willingham, 2014).
Apart from improving test scores, spaced and interleaved mathematics practice offer yet another
advantage: they help students and teachers more accurately monitor students’ learning. For instance,
with spaced practice, students revisit a particular kind of practice problem after a period of days or
weeks, which can help reveal whether students have truly learned the skill and retained it across time.
With interleaved practice, a mix of problems forces students to choose an appropriate strategy for each
problem, revealing whether they have learned to recognize when each strategy is appropriate. In
contrast, a block of a single kind of problem permits repetition of the same strategy over and over,
without revealing whether students have learned to choose appropriate strategies or whether they will
retain the skills across time. In fact, blocked practice can create an illusion of mastery (Bjork, 1999; Bjork
et al., 2013; Kornell & Bjork, 2008), in which the ease of repeating a strategy throughout a practice
session gives the impression that the strategy is well-learned – even when it is not. In other words,
blocked practice promotes overconfidence, which is problematic because overconfidence can lead
students and teachers to cease practice prematurely and, later, feel surprised and disappointed when
the students perform poorly on exams. With spaced and interleaved practice, however, overconfidence
is reduced (e.g., Emeny et al., 2021).
The Prevalence of Blocked Practice

Despite the benefits of spaced and interleaved mathematics practice, blocked practice remains
widespread. One reason for the prevalence of blocked practice may be that both students and teachers
are often unaware of the benefits of spaced and interleaved math practice. For example, in one recent
study, college students were asked to create schedules of math practice they believed would be most
effective for learning (Hartwig et al., 2022). The schedules they created predominantly featured blocked
practice and incorporated only small amounts of spacing and interleaving. Even when students were
asked to choose from several practice schedules with varying amounts of spacing and interleaving, most
selected heavily blocked practice. Other surveys of students and teachers have similarly found that
116
blocked practice is often incorrectly perceived to be a superior strategy (e.g., Halamish, 2018; McCabe,
2011; Morehead et al., 2016). Thus, greater awareness of the benefits of spaced and interleaved
practice is needed to correct the misconception that practice should be predominantly blocked.
The prevalence of blocked practice can also be seen in mathematics curricula, as shown by a recent
evaluation of six widely used seventh-grade U.S. mathematics textbooks (Rohrer et al., 2020a). Raters
examined every practice problem in each text and categorized each problem as either blocked (one that
can be solved by the same strategy used to solve the preceding problem), interleaved (not blocked), or
ambiguous (not easily classified as either blocked or interleaved). Blocked practice predominated each
text (Figure 6).
Figure 6
The Scarcity of Interleaved Practice in Six Popular Mathematics Textbooks
Blocked Ambiguous Interleaved
Big Ideas Math 81% 9%
Connected Math 55% 31%
Glencoe 86% 7%
Go Math 83% 8%
Holt McDougal 85% 8%
SpringBoard 81% 7%
0 500 1000 1500 2000 2500 3000 3500

Total Number of Practice Problems
Note. Adapted from “The scarcity of interleaved practice in mathematics textbooks,” by Rohrer et al,
2020a, Educational Psychology Review, 32, p. 873-883.). Used with permission.
The reason for the prevalence of blocked practice in mathematics textbooks is unclear—perhaps it is
simply the default arrangement. Regardless of the reason, the heavy reliance on blocked mathematics
practice is not optimal for students’ learning. Thus, teachers may need to take steps to increase the
amount of spaced and interleaved practice their students receive. We provide guidance below.
Recommendations for Teachers

1. Most of the practice problems in a mathematics course should be spaced and interleaved. In
fact, in nearly every study cited here, the students who performed the best were the ones
whose practice assignments were arranged so that at least two-thirds of the problems were
spaced or interleaved. However, there is no magic proportion because the optimal mixture of
spaced, interleaved practice and blocked practice depends on numerous factors, including the
nature of the material and the mathematics proficiency of the students. For instance, weaker
students might need more blocked practice than do stronger students before confronting
interleaved practice.
117
2. Students should be shown the solution to every practice problem soon after they attempt the
problem – perhaps no later than the next school day. We suspect that most teachers already do
this, but this kind of feedback is especially important when practice is spaced and interleaved.
With blocked practice, students often zip through the assignment by simply repeating the same
procedure and are thus less likely to need to see the solutions. With spaced and interleaved
practice, however, students more often fail to solve problems, and they cannot learn from such
failures unless they receive feedback that provides the correct answer or solution (e.g., Guthrie,
1971; Pashler et al., 2005). In short, a practice problem serves little or no purpose if students do
not ultimately understand its solution.
3. Two consecutive interleaved problems need not be entirely unrelated. For instance, a problem
solved by the Law of Sines might be followed by one that is solved by the Law of Cosines. This
juxtaposition of two fundamentally-distinct, superficially-similar problems might help students
identify the critical features of each problem that indicate which strategy is appropriate for each
problem (e.g., Dunlosky et al., 2013; Sana et al., 2017). Alternatively, a set of interleaved
problems can relate to a single scenario, such as a histogram followed by problems about the
mean, median, range, and standard deviation.
4. If the students’ textbook and other school-supplied curricula provide mostly blocked practice,
teachers can supplement these resources with assignments providing spacing and interleaved
practice. For instance, teachers can create their own assignments by choosing one problem from
each of a dozen blocked practice assignments in the students’ textbook (e.g., a problem from
each of 10 different chapters). Teachers also can search the internet for interleaved sets of
problems, including practice tests for standardized exams.
Final Remarks
Spaced and interleaved practice are not flashy, but each is strongly supported by empirical research.
Both interventions have dramatically boosted test scores in randomized studies in both the laboratory
and classroom, and each strategy can be used in nearly any mathematics course. The rationale is
straightforward. Whereas blocked practice allows students to infer the relevant concept or strategy
needed to solve a problem before they even read the problem, spaced and interleaved practice require
that students choose the appropriate strategy for the problem on the basis of the problem itself, just as
they must do when given an exam that evaluates multiple skills and concepts. In short, spacing and
interleaving give students an opportunity to learn what they are expected to know.
Author Note
The research reported here was supported by the Institute of Education Sciences, U.S. Department of
Education, through grant R305A160263 to the University of South Florida. The opinions expressed are
those of the authors and do not represent the views of the U.S. Department of Education.
References
Bahrick, H. P., Bahrick, L. E., Bahrick, A. S., & Bahrick, P. E. (1993). Maintenance of foreign language
vocabulary and the spacing effect. Psychological Science, 4(5), 316-321.
118
Barzagar Nazari, K., & Ebersbach, M. (2019). Distributing mathematical practice of third and seventh
graders: Applicability of the spacing effect in the classroom. Applied Cognitive Psychology, 33(2),
288–298. https://doi.org/10.1002/acp.3485
Bjork, R. A., & Bjork, E. L. (2019). The myth that blocking one’s study or practice by topic or skill
enhances learning. In C. Barton (Ed.), Education myths: An evidence-informed guide for teachers
(pp. 57-70). John Catt Educational Ltd.
Annual Review of Psychology, 64, 417–444. https://doi.org/10.1146/annurev-psych-113011-
143823
Butler, A. C., Marsh, E. J., Slavinsky, J. P., & Baraniuk, R. G. (2014). Integrating cognitive science and
technology improves learning in a STEM classroom. Educational Psychology Review, 26(2), 331–
Carpenter, S. K. (2014). Spacing and interleaving of study and practice. In V. A. Benassi, C. E. Overson, &
C. M. Hakala (Eds.), Applying the science of learning in education: Infusing psychological science
into the curriculum (pp. 131–141). Society for the Teaching of Psychology.
Dempster, F. N. (1988). The spacing effect: A case study in the failure to apply the results of
psychological research. American Psychologist, 43(8), 627–634. https://doi.org/10.1037/0003-
066X.43.8.627
Ebersbach, M., & Barzagar Nazari, K. (2020). Implementing distributed practice in statistics courses:
Benefits for retention and transfer. Journal of Applied Research in Memory and Cognition, 9(4),
532-541. https://doi.org/10.1016/j.jarmac.2020.08.014
Eglington, L. G., & Kang, S. H. K. (2017). Interleaved presentation benefits science category learning.
Journal of Applied Research in Memory and Cognition, 6(4), 475–485.
Emeny, W. G., Hartwig, M. K., & Rohrer, D. (2021). Spaced mathematics practice improves test scores
and reduces overconfidence. Applied Cognitive Psychology, 35(4), 1082-1089.
https://doi.org/10.1002/acp.3814
Foster, N. L., Mueller, M. L., Was, C., Rawson, K. A., & Dunlosky, J. (2019). Why does interleaving
improve math learning? The contributions of discriminative contrast and distributed
practice. Memory & Cognition, 47(6), 1088-1101. https://doi.org/10.3758/s13421-019-00918-4
Gay, L. R. (1973). Temporal position of reviews and its effect on the retention of mathematical rules.
Journal of Educational Psychology, 64(2), 171–182. https://doi.org/10.1037/h0034595
Guthrie, J. T. (1971). Feedback and sentence learning. Journal of Verbal Learning and Verbal
Behavior, 10(1), 23-28. https://doi.org/10.1016/S0022-5371(71)80088-9
Halamish, V. (2018). Pre-service and in-service teachers’ metacognitive knowledge of learning strategies.
Frontiers in Psychology, 9, 2152. https://doi.org/10.3389/fpsyg.2018.02152
119
Hartwig, M. K., Rohrer, D., & Dedrick, R. F. (2022). Scheduling math practice: Students’
underappreciation of spacing and interleaving. Journal of Experimental Psychology: Applied,
28(1), 100–113. https://doi.org/10.1037/xap0000391
Higgins, E. J. (2019). The complexities of learning categories through comparisons. In Psychology of
learning and motivation (Vol. 66, pp. 43-77). Academic Press.
https://doi.org/10.1016/bs.plm.2016.11.002
Hopkins, R. F., Lyle, K. B., Hieb, J. L., & Ralston, P. A. (2016). Spaced retrieval practice increases college
students’ short-and long-term retention of mathematics knowledge. Educational Psychology
Review, 28, 853–873. https://doi.org/10.1007/s10648-015-9349-8
Kang, S. H. (2016). Spaced repetition promotes efficient and effective learning: Policy implications for
instruction. Policy Insights from the Behavioral and Brain Sciences, 3(1), 12–19.
Kirschner, P. A., & van Merriënboer, J. J. (2013). Do learners really know best? Urban legends in
education. Educational Psychologist, 48(3), 169-183.
Psychological Science, 19(6), 585–592. https://doi.org/10.1111/j.1467-9280.2008.02127.x
Lyle, K. B., Bego, C. R., Hopkins, R. F., Hieb, J. L., & Ralston, P. A. S. (2020). How the amount and spacing
of retrieval practice affect the short- and long-term retention of mathematics knowledge.
Educational Psychology Review, 32, 277–295. https://doi.org/10.1007/s10648-019-09489-x
Mayfield, K. H., & Chase, P. N. (2002). The effects of cumulative practice on mathematics problem
solving. Journal of Applied Behavior Analysis, 35(2), 105–123.
https://doi.org/10.1901/jaba.2002.35-105
McCabe, J. (2011). Metacognitive awareness of learning strategies in undergraduates. Memory &
Cognition, 39, 462–476. https://doi.org/10.3758/s13421-010-0035-2
Morehead, K., Rhodes, M. G., & DeLozier, S. (2016). Instructor and student knowledge of study
strategies. Memory, 24(2), 257–271. https://doi.org/10.1080/09658211.2014.1001992
Pan, S. C. (August 4, 2015). The interleaving effect: Mixing it up boosts learning. Scientific American
Mind. https://www.scientificamerican.com/article/the-interleaving-effect-mixing-it-up-boosts-
learning/
Organizing instruction and study to improve student learning - IES practice guide. Washington,
DC: National Center for Education Research. https://files.eric.ed.gov/fulltext/ED498555.pdf
Pashler, H., Cepeda, N. J., Wixted, J. T., & Rohrer, D. (2005). When does feedback facilitate learning of
words? Journal of Experimental Psychology: Learning, Memory, and Cognition, 31(1), 3–8.
Rau, M. A., Aleven, V., & Rummel, N. (2013). Interleaved practice in multi-dimensional learning tasks:
Which dimension should we interleave? Learning and Instruction, 23, 98–114.
Robinson, D. H., & Levin, J. R. (2019). Quackery in educational research. In J. Dunlosky & K. A. Rawson
(Eds.), The Cambridge handbook of cognition and education (pp. 35–48). Cambridge University
Press. https://doi.org/10.1017/9781108235631.003
120
Roediger, H. L., & Pyc, M. A. (2012). Inexpensive techniques to improve education: Applying cognitive
psychology to enhance educational practice. Journal of Applied Research in Memory and
Cognition, 1(4), 242–248. https://doi.org/10.1016/j.jarmac.2012.09.002
Rohrer, D., & Hartwig, M. K. (2020). Unanswered questions about spaced and interleaved mathematics
practice. Journal of Applied Research in Memory and Cognition, 9(4), 433–438.
Rohrer, D., & Taylor, K. (2006). The effects of overlearning and distributed practice on the retention of
mathematics knowledge. Applied Cognitive Psychology, 20(9), 1209–1224.
Rohrer, D., & Taylor, K. (2007). The shuffling of mathematics practice problems boosts learning.
Instructional Science, 35, 481–498. https://doi.org/10.1007/s11251-007-9015-8
Rohrer, D., Dedrick, R. F., & Burgess, K. (2014). The benefit of interleaved mathematics practice is not
limited to superficially similar kinds of problems. Psychonomic Bulletin & Review, 21, 1323–1330.
Rohrer, D., Dedrick, R. F., & Hartwig, M. K. (2020a). The scarcity of interleaved practice in mathematics
textbooks. Educational Psychology Review, 32, 873-883. https://doi.org/10.1007/s10648-020-
09516-2
Rohrer, D., Dedrick, R. F., & Stershic, S. (2015). Interleaved practice improves mathematics learning.
Journal of Educational Psychology, 107(3), 900-908. https://doi.org/10.1037/edu0000001
Rohrer, D., Dedrick, R. F., Hartwig, M. K., & Cheung, C. N. (2020b). A randomized controlled trial of
interleaved mathematics practice. Journal of Educational Psychology, 112(1), 40–52.
Samani, J. & Pan, S. C. (2021). Interleaved practice enhances memory and problem-solving ability in
undergraduate physics. NPJ Science of Learning, 6(32). https://doi.org/10.1038/s41539-021-
00110-x
Sana, F., Yan, V. X., & Kim, J. A. (2017). Study sequence matters for the inductive learning of cognitive
concepts. Journal of Educational Psychology, 109(1), 84–98.
Schutte, G. M., Duhon, G. J., Solomon, B. G., Poncy, B. C., Moore, K., & Story, B. (2015). A comparative
analysis of massed vs. distributed practice on basic math fact fluency growth rates. Journal of
School Psychology, 53(2), 149-159. https://doi.org/10.1016/j.jsp.2014.12.003
Taylor, K., & Rohrer, D. (2010). The effect of interleaved practice. Applied Cognitive Psychology, 24(6),
837–848. https://doi.org/10.1002/acp.1598
Willingham, D. T. (2014). Strategies that make learning last. Educational Leadership, 72(2), 10–15.
121
In Their Own Words Task Complexity, Learner Expertise, and
Instructional Goals in Managing Instructional
Guidance
Slava Kalyuga
Editors
The University of New South Wales (UNSW Sydney), Australia
Lauren L. Kordonowy
Victor A. Benassi
Overview
Instructional guidance (instructional assistance, instructional support) is generally intended to provide
learners with information they have not yet learned to expedite completing a learning task or attaining a
learning goal (e.g., Clark, 2009). Various instructional means could be used for this purpose—such as
providing explicit instruction, demonstrating required action steps (e.g., worked examples), offering
partial guidance (e.g., completion tasks, faded worked examples, action prompts, hints), or providing
feedback on attempted steps or giving answers to student questions. In contrast, with unguided or
minimally guided learning approaches, learners are assumed to invent the solution steps on their own
and in the process of invention, implicitly acquire new knowledge. Another frequently discussed form of
instructional guidance is scaffolding—a temporary instructional support that should be gradually
eliminated when learners are capable of dealing with the task on their own (Pea, 2004). For efficient
teaching and learning, it is important to understand the principles and rationales for using different
types and levels of instructional guidance.
The importance of choosing the optimal degree of guidance by providing instructional assistance or
withdrawing it (‘assistance dilemma,’ according to Koedinger & Aleven, 2007) has been highlighted in
the continuing arguments about the benefits and drawbacks of explicit instruction in contrast with
minimal-guidance approaches, such as inquiry, problem-based or discovery learning (Clark, 2009;
Hmelo-Silver et al., 2007; Kirschner et al.,, 2006; Klahr, 2009; Lee & Anderson, 2013; Mayer, 2004;
Schmidt et al., 2007). Two specific points of disagreement are 1) whether learners should be given
explicit instruction in concepts or solution steps prior to problem-solving activities and, if yes, 2)
whether they should be involved in inventing or problem-solving attempts prior to the explicit
instruction. For example, while cognitive load theory (Ashman & Sweller, this volume; Sweller et al.,
2011) views explicit instruction as a compulsory initial stage of any cognitively optimal instructional
sequence (at least, for novice learners), inquiry-oriented instructional approaches (e.g., discovery
learning, problem-based learning) suggest minimally guided problem-solving activities as the legitimate
basis of meaningful learning and instruction (Kuhn, 2007). A few approaches (e.g., invention learning:
Schwartz & Bransford, 1998; productive failure: Kapur, 2008, Kapur & Roll, this volume) consider explicit
instruction as an essential component, but suggest it should always be preceded by learners’ own
attempts at solving problems or inventing concepts or solutions first.
Understanding the essential factors that need to be considered while navigating through such
contradictory viewpoints and making decisions on optimal levels of guidance in specific instructional
situations is critical, especially considering that most of these conflicting instructional approaches offer
plenty of empirical evidence in their support, and appear to be proven positions rather than only
theoretical speculations. This chapter is focused on three main factors – complexity of the learning task,
learner prior knowledge (expertise), and instructional goals to be achieved. The role of the first two of
122
these factors is well investigated within a cognitive load framework, while the significance of the third
factor is its role in resolving the above controversies between alternative views on the optimal levels of
instructional guidance. The next section of the chapter will provide a brief overview of the first two
factors, while the following section will focus on the role of instructional goals. Instructional implications
and recommendations conclude the last section of the chapter.
Task Complexity and Learner Expertise: A Cognitive Load Perspective

It is commonly accepted that relatively more instructional guidance should be provided for (1) more
complex tasks (which are more difficult for learners to complete on their own) than simpler tasks and
for (2) less knowledgeable, novice learners (who are less likely to be able to learn without external help)
than those more experienced in the task domain, expert learners. Cognitive load theory (see Ashman
and Sweller, this volume) provides rational scientific explanations of these guidelines from the
perspective of human cognitive architecture and cognitive mechanisms of learning. According to this
theory, the role of instructional guidance is to manage learner cognitive load by providing the necessary
amount of information at the right time in a suitable format. In particular, this theory generated two
principles that are directly relevant to this issue: the worked example effect (Renkl, this volume) and the
expertise reversal effect (Kalyuga, 2007; Lee & Kalyuga, 2014). These two principles/effects are usually
utilized in cognitive load theory to predict instructional consequences of using explicit instructional
guidance vs. unguided problem solving and to describe the differential roles of instructional guidance for
novice and expert learners.
In cognitive load theory, processing limitations of human cognitive architecture are considered a main
factor influencing learning and instruction. One major component of this architecture, working memory,
is responsible for conscious processing of information within the focus of attention, and it is severely
limited in both capacity and duration (only a few units of information could be processed concurrently
and for no longer than a matter of seconds) when dealing with novel information (Baddeley, 1986;
Miller, 1956). When such limited working memory resources are engaged in processing non-essential
information or in irrelevant cognitive activities (e.g., random searches for solution moves when dealing
with a new problem), learning could become inefficient and delayed or impossible.
Another major component of human cognitive architecture is long-term memory, the storage of learned
information (our knowledge base). It is unlimited in capacity and duration, and the extent, structure,
and the degree of automation of domain-specific knowledge in long-term memory is determined by
expertise in the corresponding domain—whether it is a broad professional field of knowledge (e.g.,
molecular biology) or a narrow task domain (e.g., solving linear algebra equations in school
mathematics). The level of learner expertise is a major factor influencing learning and performance
(Bransford et al., 1999; Chase & Simon, 1973). Organized generic knowledge structures (frequently
called schemas) in long-term memory enable expert learners to circumvent the working memory
limitations by encapsulating multiple elements of information into larger units to be treated as single
elements in working memory (Ericsson & Kintsch, 1995). As a result, learning characteristics of experts
(more knowledgeable learners) are significantly different from those of novices.
The notion of element interactivity (or interconnectedness) is used in cognitive load theory to address
the effects of task complexity. The more interactive elements that need to be considered concurrently
which are involved in a task, the more complex the task. More working memory resources would be
required to make sense of a complex task or to find a solution; thus, working memory load (cognitive
load) would increase. The subjective difficulty of a task is determined not only by the level of element
interactivity, but also by the learner’s level of expertise. In fact, the level of element interactivity is
always relative to a particular learner’s level of expertise, determined by the previously acquired
123
domain-specific organized knowledge structures (schemas). Many interactive elements of information in
a difficult task for a novice learner could involve only a single higher-level unit for an expert, whose
schemas would allow encapsulating the interactive elements into this unit, thus making the task much
easier for an expert to handle in working memory.
Therefore, the learner’s level of expertise is considered to be another essential factor influencing the
instructional support decision, and the expertise reversal effect in cognitive load theory provides an
important guiding principle. This effect (Kalyuga et al., 2003; Lee & Kalyuga, 2014) refers to interactions
between levels of learner expertise and effectiveness of alternative instructional methods, and one of its
main instructional implications is the need to tailor levels of instructional guidance to levels of learner
expertise. For example, novice learners may considerably benefit from various forms of explicit
instructional support, while for learners with higher levels of prior knowledge, these types of support
could become redundant and interfere with the optimal trajectories of improvement of their knowledge
and skills. Accordingly, the cognitively optimal transition from explicit, meticulously guided instruction to
unguided problem-solving learning tasks needs to be a gradual process involving different, learner-
tailored levels of instructional guidance based on monitoring changes in the learner’s knowledge base.
Even from this brief summary, it is clear that according to cognitive load theory, novices learn most
efficiently with comprehensive instructional guidance (e.g., explicit instruction in the form of worked
examples) provided before they attempt to do the task on their own (e.g., worked example effect; see
Sweller et al., 2011, for an overview). Such explicit guidance allocates learners’ limited cognitive
resources to learning the solution moves directly relevant to the target schema rather than to irrelevant
and resource-consuming activities, such as a random search for appropriate solution steps—what
presumably occurs with various forms of inquiry-based instructional approaches based on minimally-
guided problem-solving activities. Only after the learners acquired some relevant task-specific
knowledge (e.g., at least partial schemas) could the levels of guidance be gradually reduced through
various partial forms of support, eventually transforming into minimal-guidance, problem-solving tasks
when learners gain higher levels of expertise.
Still, a number of studies within the frameworks of productive failure and the preparation for future
learning or invention learning (e.g., Kapur, 2008; 2011; 2014; Schwartz & Martin, 2004; Sinha et al.,
2021; Kapur & Roll, this volume) disputed this point of view by showing that minimally guided problem-
solving learning tasks provided prior to explicit instruction were beneficial for learners. It should be
noted that these approaches, in contrast to some pure inquiry-based frameworks, do not deny the
essential role of explicit instruction, which is their important component. The disagreement relates
mostly to the sequence of activities: what should go first – a problem-solving activity or explicit
instruction? Some of the studies within these frameworks were criticized in cognitive load theory for not
always adhering to strict experimental control of the compared conditions (e.g., by not using identical
learning tasks in different conditions) (Hsu, Kalyuga, & Sweller, 2015; Sweller, Kirschner, & Clark, 2007).
Increasing empirical support for such frameworks may need to be reconciled with cognitive load theory
on a deeper theoretical level.
One way for reconciliation is from the productive failure approach recently suggested by Ziegler et al.
(2021). They demonstrated that in cases of procedural knowledge, the benefits of productive failure
were observed only for simple component procedural steps (‘micro productive failure’)—that is, from a
cognitive load perspective, engaging learners in low element interactivity tasks rather than high element
interactivity tasks. This is exactly the position of cognitive load theory: problem-first instruction could be
effective only for low element interactivity tasks. Another way of reconciling these two approaches,
through cognitive load theory, is by clarifying the broader characteristics of learning and instruction
124
situations for which this theory works best, as well as those for which it may not be effective; in other
words, outlining the boundaries of applicability of cognitive load theory.
Instructional Goals of Learning Activities as a Unifying Factor

Two components— 1) levels of element interactivity determining the complexity of learning tasks, and
2) learner prior knowledge in a domain determining the level of expertise in the task domain— have
been examined considerably as the factors determining the degree of required instructional guidance in
cognitive load theory (e.g., Ashman et al., 2020; Chen, et al., 2017). The overwhelming evidence from
research in the worked example effect and expertise reversal effect showed that novice learners dealing
with complex, high-element interactivity learning tasks required explicit instruction with high levels of
instructional guidance. Only when learning tasks were inherently low in element interactivity (e.g.,
studying lists of mostly unconnected elements, formulas, vocabulary items), or had become such
because of increased levels of learner expertise, were problem-solving activities and problem-first
sequences effective instructional methods (Chen, Kalyuga, & Sweller, 2015; 2016; 2017). For example,
multiple studies in the expertise reversal effect (see Lee & Kalyuga, 2014, for an overview)
demonstrated that while expert learners benefited from problem solving tasks, explicit instruction could
inhibit their learning, as it essentially provided redundant information to be processed that
unnecessarily detracted from available cognitive resources. Chen et al. (2015, 2016) demonstrated that
self-generation tasks, such as recalling previously learned math formulas, were more effective than
explicitly viewing these formulas— these types of learning materials were obviously low in element
interactivity.
These well empirically supported rules, however, do not fit the studies within the frameworks of
productive failure and the preparation for future learning or invention learning, according to which
initial problem-solving activities could be effective for complex learning tasks, particularly for novice
learners. Therefore, I will address another important factor that may help to outline the boundaries of
cognitive load theory— specific instructional goals of learning activities. An expected result of this focus
on clearly differentiating instructional goals of learning activities, especially in cases of complex learning
tasks, is a dismissal of the explicit instruction vs. unguided learning dichotomy in favor of a more flexible,
unifying and integrative framework. In particular, with this flexible approach, providing novice learners
with initial problem solving or exploratory learning tasks may be an effective means for achieving some
specific instructional goals that are different from the follow-up goals of learning domain-specific
schemas. For example, in the productive failure and invention learning approaches, such additional
goals are achieved by initial problem-solving activities prior to the following explicit instruction and
problem-solving practice.
Kalyuga (2015) and Kalyuga and Singh (2016) proposed that taking into account the differences between
specific instructional goals of various learning tasks could potentially resolve the above contradictions
between the problem-first and explicit instruction-first approaches. Various learner activities involved in
complex learning tasks may be aimed at achieving different specific instructional goals, such as
motivating and engaging learners, activating their prior knowledge, acquiring specific solution schemas,
or automating the acquired schemas. Cognitive load theory has always focused on the goal of learning
domain-specific schemas and has consistently demonstrated the inhibiting effects of working memory
overload on achieving this goal. Most cognitive load effects essentially reveal such phenomena.
However, this evidence and the corresponding instructional implications do not necessarily apply to
other types of specific instructional goals. For example, if the goal is the activation of relevant prior
knowledge, there is no clear evidence indicating that high levels of cognitive load experienced by novice
learners attempting to solve novel problems on their own would adversely affect the achievement of
this goal, even if it would be ineffective for the acquisition of the corresponding solution schemas.
125
Viewed from this perspective, the goals of the initial problem-solving attempt (the generation phase) in
the productive failure or invention learning frameworks are different from the acquisition of domain-
specific solution schemas. For example, Kapur and Bielaczyc (2012), Kapur and Rummel (2012), and
Schwartz and Martin (2004) indicated the goal of activating and differentiating learner prior knowledge.
In addition, Loibl and Rummel (2014a; 2014b) studied enhancing learner awareness of knowledge gaps
and problem situations before the following phase of explicit instruction (with the goal of the acquisition
of solution schemas). Several goals for the generation phase (e.g., activating relevant prior knowledge,
focusing on deep features, enhancing awareness of knowledge gaps, triggering learner motivation and
interest in the problem etc.) could co-exist (Glogger-Frey et al., 2015). Such goals might be effectively
accomplished by instructional tools different from those used in explicit instruction, which is the best
initial approach for the acquisition of domain-specific schemas according to cognitive load theory. The
increased levels of cognitive load during the generation or problem-solving phase would not necessarily
prevent achieving the specific instructional goals of this phase.
The optimal levels of guidance and the corresponding instructional techniques might depend on specific
instructional goals to be achieved. The initial problem exploration by novice learners in the productive
failure approach is effective for achieving the goal of activating learner prior knowledge or making them
better aware of the problem situations (even though it is not suitable for teaching domain-specific
solution schemas, due to excessive cognitive load caused by problem-solving search processes which are
unproductive for achieving this goal). Complex learning tasks may include learner activities with
different instructional goals. On the other hand, the same type of learner activities may be used for
achieving different instructional goals too. For example, a problem-solving task in an example-problem
pair is effective in reinforcing or automating the solution schema learned by studying the worked
example (according to cognitive load theory). Similar problem-solving tasks are used in productive
failure, invention learning, or other problem-first frameworks for achieving the above mentioned non-
schema acquisition goals. Such goals could be referred to as ‘pre-instruction’ goals—the goals associated
with creating cognitive or motivational prerequisites for further learning (Kalyuga & Singh, 2016).
Various search-based, minimally-guided, exploratory problem-solving activities (especially in well-
contextualized, engaging, emotionally charged simulation or game-based learning environments) may
be best used for achieving such ‘pre-instruction’ goals.
Educational Implications
According to the arguments provided in the previous section, the management of instructional guidance
should take into account not only levels of task complexity and learner expertise but also specific
instructional goals and the chosen sequence of learning activities. In the end, the combination of all
these factors—specific instructional goals and levels of task complexity/learner expertise—define the
degree of required instructional guidance (comprehensive, partial, or minimal guidance).
Although most of the empirical support for this integrative goal-driven approach comes from research in
problem-first approaches (e.g., productive failure or invention learning frameworks), some support was
also obtained within a cognitive load framework. Kalyuga and Hsu (2019) compared four different levels
of instructional guidance during the initial problem-solving phase prior to explicit instruction provided to
high school students who studied how to solve physics problems. The different levels of guidance
included: fully unguided problem solving, partial principle-based guidance, partial principle-based
guidance with reflection on solution attempts, and an entirely worked-out solution procedure (a worked
example). The second phase (explicit instruction in the form of a worked example) was the same for all
the compared conditions. The results of the delayed transfer post-test showed similar effects of
instructional conditions across all these groups, with similar means and relatively low effect sizes. There
was no superiority of worked example-first instruction in comparison to unguided or partially guided
126
initial problem solving (even though the partial principle-based guidance with reflection group
outperformed other two problem-first groups). When considering specific goals of learner activities
within complex learning tasks, these results were not unexpected: participants apparently learned the
required problem-solving procedure in different ways, through various sequences of specific goals in
different phases of instruction. The reduced-guidance initial problem-solving activities could have
facilitated creating cognitive or motivational prerequisites to learning, such as achieving pre-
instructional goals, including activating prior knowledge or engagement with the task, and problem
solving with partial principle-based guidance and reflection was possibly better in this than other
problem-solving activities. These activities were not directly aimed at the acquisition of domain-specific
schemas using worked examples. The compared learning pathways based on different sequences of
specific instructional goals appeared to be equally effective.
Another similar study explored different instructional pathways to learning by comparing the
effectiveness of unguided problem solving, partially-guided problem solving (with some hints in the
form of final steps of the worked-out solutions), and comprehensive guidance (worked example) prior to
explicit instruction in the standard solution procedures (Likourezos & Kalyuga, 2017). The participants
were secondary school students learning a specific type of task in geometry (I.e., constructing a
perpendicular to the interval from a point below using a pair of compasses, a ruler, pencil and eraser),
and learning success was judged by improved engagement and transferable knowledge (measured by a
delayed post-test). The results showed no significant differences between the groups, with very similar
means and low effect sizes, suggesting that all three groups performed at the same level. When the
students’ solutions in the post-test were evaluated as either canonical (i.e., the same as those provided
during the learning phase) or creative (I.e., different from those previously provided), the results
indicated that the conditions with more instructional guidance showed significantly more cases of
creative thinking, with the worked-example group providing the largest number of creative answers,
followed by the partial-guidance group, and the no-guidance group last. Indicators of motivation
showed that the fully-guided initial instruction phase increased learner interest and confidence in
success, while the minimally-guided initial phase motivated the acceptance of challenge, which
apparently overrode any negative effects of the increased cognitive load. In summary, the reduced
levels of cognitive load did not result in higher performance in the worked example-first group in
comparison with problem-first groups. Again, the alternative sequences of learner activities with varying
instructional goals lead to similar overall outcomes.
The suggested general procedure in making decisions regarding appropriate levels of instructional
guidance is represented in Figure 1. If the learning task is a simple, low-element interactivity task,
minimal guidance should be provided. The same approach applies to learners highly knowledgeable in
the task domain because for them, the task will likely be simple due to their high level of expertise.
When the task is complex and the learners are novices, the decision depends on the intended
instructional goal of the planned activity – if the goal is domain schema acquisition, high levels of
instructional guidance (e.g., explicit instruction in the form of worked examples) should be used. If the
goal is non-schema related (e.g., one of the ‘pre-instruction’ goals such as motivation/engagement or
knowledge activation), then minimal guidance could be appropriate.
During this decision-making process, instead of rigidly adhering to either explicit instruction or to
inquiry-based learning approaches, the focus should be on selecting the most efficient ways of achieving
specific instructional goals by combining various methods of explicit instruction (e.g., worked examples,
demonstration, partial forms of guidance) and inquiry learning (e.g., exploratory, problem-based,
discovery). Any available and forthcoming evidence of the effectiveness of different instructional
127
approaches should be viewed from the perspective of specific goals to be achieved by learners with
specific cognitive characteristics (e.g., level of expertise).
In conclusion, when selecting optimal instructional methods, in addition to the task complexity and
levels of learner expertise, the goals of the intended learner activities should be considered. In many
instructional situations with complex tasks and novice learners, alternative strategies reflecting different
sequences of instructional goals may work equally effectively.
Figure 1
The Process of Selecting Levels of Instructional Guidance
References
Ashman, G., Kalyuga, S., & Sweller, J. (2020). Problem-solving or explicit instruction: which should go
first when element interactivity is high?, Educational Psychology Review, 32(3), 229–247.
Baddeley, A. D. (1986). Working memory. Oxford University Press.
Bransford, J.D., Brown, A.L., & Cocking, R.R. (Eds.). (1999). How people learn: Mind, brain, experience,
and school. National Academy Press.
128
Chase, W. G., & Simon, H. A. (1973). Perception in chess. Cognitive Psychology, 4(1), 55-81.
https://doi.org/10.1016/0010-0285(73)90004-2
Chen, O., Kalyuga, S., & Sweller, J. (2015). The worked example effect, the generation effect, and
element interactivity. Journal of Educational Psychology, 107(3), 689-704.
Chen, O., Kalyuga, S., & Sweller, J. (2016). Relations between the worked example and generation
effects on immediate and delayed tests. Learning and Instruction, 45, 20-30.
Chen, O., Kalyuga, S., & Sweller, J. (2017). The expertise reversal effect is a variant of the more general
element interactivity effect. Educational Psychology Review, 29, 393-405.
Clark, R. E. (2009). How much and what type of guidance is optimal for learning from instruction. In S.
Tobias & T. M. Duffy (Eds.), Constructivist instruction: Success or failure? (pp. 158-183).
Routledge.
Ericsson, K. A., & Kintsch, W. (1995). Long-term working memory. Psychological Review, 102(2), 211-245.
https://doi.org/10.1037/0033-295X.102.2.211
Glogger-Frey, I., Fleischer, C., Grüny, L., Kappich, J., & Renkl, A. (2015). Inventing a solution and studying
a worked solution prepare differently for learning from direct instruction. Learning and
Instruction, 39, 72-87. https://doi.org/10.1016/j.learninstruc.2015.05.001
Hmelo-Silver, C. E., Duncan, R. G., & Chinn, C. A. (2007). Scaffolding and achievement in problem-based
and inquiry learning: A response to Kirschner, Sweller, and Clark (2006). Educational
Psychologist, 42(2), 99-107. https://doi.org/10.1080/00461520701263368
Hsu, C.-Y., Kalyuga, S., & Sweller, J. (2015). When should guidance be presented during physics
instruction? Archives of Scientific Psychology, 3(1), 37-53. https://doi.org/10.1037/arc0000012
Kalyuga, S. (2007). Expertise reversal effect and its implications for learner-tailored instruction.
Educational Psychology Review, 19, 509-539. https://doi.org/10.1007/s10648-007-9054-3
Kalyuga S. (2015). Instructional guidance: A cognitive load perspective. Information Age Publishing.
Kalyuga, S., Ayres, P., Chandler, P., & Sweller, J. (2003). The expertise reversal effect. Educational
Psychologist, 38(1), 23-31. https://doi.org/10.1207/S15326985EP3801_4
Kalyuga, S., & Hsu, C-Y. (2019). What should students do first when learning how to solve a physics
problem: Try to solve it themselves or study a worked example? In S. Tindall-Ford, S. Agostinho,
and J. Sweller (Eds.), Advances in cognitive load theory: Rethinking teaching (pp. 209-220).
Routledge.
Kalyuga, S., & Singh, A-M. (2016). Rethinking the boundaries of cognitive load theory in complex
learning. Educational Psychology Review, 28, 831-852. https://doi.org/10.1007/s10648-015-
9352-0
Kapur M. (2008). Productive failure. Cognition and Instruction, 26(3), 379–424.
Kapur M. (2011). A further study of productive failure in mathematical problem solving: unpacking the
design components. Instructional Science, 39, 561–579. https://doi.org/10.1007/s11251-010-
9144-3
129
Kapur, M. (2014). Productive failure in learning math. Cognitive Science, 38(5), 1008-1022.
Kapur, M., & Bielaczyc, K. (2012). Designing for productive failure. The Journal of the Learning Sciences,
21(1), 45–83. https://doi.org/10.1080/10508406.2011.591717
Kapur, M., & Rummel, N. (2012). Productive failure in learning from generation and invention activities.
Instructional Science, 40, 645–650. https://doi.org/10.1007/s11251-012-9235-4
Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why minimal guidance during instruction does not
work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and
inquiry-based teaching. Educational Psychologist, 41(2), 75-86.
https://doi.org/10.1207/s15326985ep4102_1
Klahr, D. (2009). "To every thing there is a season, and a time to every purpose under the heavens":
What about direct instruction? In S. Tobias & T. M. Duffy (Eds.), Constructivist instruction:
Success or failure? (pp. 291-310). Routledge.
Koedinger, K. R., & Aleven, V. (2007). Exploring the assistance dilemma in experiments with cognitive
tutors. Educational Psychology Review, 19, 239–264. https://doi.org/10.1007/s10648-007-9049-
0
Kuhn, D. (2007). Is direct instruction an answer to the right question? Educational psychologist, 42 (2),
109-113. https://doi.org/10.1080/00461520701263376
Lee, H. S., & Anderson, J. R. (2013). Student learning: What has instruction got to do with it? Annual
Review of Psychology, 64(1), 445-469. https://doi.org/10.1146/annurev-psych-113011-143833
Lee, C. H. & Kalyuga, S. (2014). Expertise reversal effect and its instructional implications. In V.A. Benassi,
C.E. Overson, & C.M. Hakala (Eds.), Applying science of learning in education: Infusing
psychological science into the curriculum (pp. 31-44). Society for the Teaching of Psychology.
https://teachpsych.org/ebooks/asle2014/index.php
Likourezos, V., & Kalyuga, S. (2017). Instruction-first and problem-solving-first approaches: alternative
pathways to learning complex tasks. Instructional Science, 45, 195-219.
Loibl, K. & Rummel, N. (2014a). Knowing what you don't know makes failure productive. Learning and
Loibl, K., & Rummel, N. (2014b). The impact of guidance during problem-solving prior to instruction on
students’ inventions and learning outcomes. Instructional Science,42, 305-326.
Mayer, R. E. (2004). Should there be a three-strikes rule against pure discovery learning? American
Psychologist, 59(1), 14-19. https://doi.org/10.1037/0003-066x.59.1.14
Miller, G. A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for
processing information. Psychological Review, 63(2), 81-97. https://doi.org/10.1037/h0043158
Pea, R. (2004). The social and technological dimensions of scaffolding and related theoretical concepts
for learning, education, and human activity. Journal of the Learning Sciences, 13(3), 423-451.
https://doi.org/10.1207/s15327809jls1303_6
130
Schmidt, H. G., Loyens, S. M. M., Van Gog, T., & Paas, F. (2007). Problem-based learning is compatible
with human cognitive architecture: Commentary on Kirschner, Sweller, and Clark (2006).
Educational Psychologist, 42(2), 91-97. https://doi.org/10.1080/00461520701263350
Schwartz, D., & Bransford, J. D. (1998). A time for telling. Cognition and Instruction, 16(4), 475–522.
Schwartz, D. L., & Martin, T. (2004). Inventing to prepare for future learning: The hidden efficiency of
encouraging original student production in statistics instruction. Cognition and Instruction,
22(2), 129–84. https://doi.org/10.1207/s1532690xci2202_1
Sinha, T., Kapur, M., West, R. Catasta, M., Hauswirth, M. & Trninic, D. (2021). Differential benefits of
explicit failure-driven and success-driven scaffolding in problem-solving prior to
instruction. Journal of Educational Psychology, 113(3), 530-555.
Sweller, J., Ayres, P., & Kalyuga, S. (2011). Cognitive load theory. Springer.
Sweller, J., Kirschner, P. A., & Clark, R. E. (2007). Why minimally guided teaching techniques do not work:
A reply to commentaries. Educational Psychologist, 42(2), 115-121.
Ziegler, E., Trninic, D. & Kapur, M. (2021). Micro productive failure and the acquisition of algebraic
procedural knowledge. Instructional Science, 49, 313-336. https://doi.org/10.1007/s11251-021-
09544-7
131
In Their Own Words Interleaved Training and Category Learning

Sean H. K. Kang
to Apply the Science of Learning in The University of Melbourne
Editors
Lauren L. Kordonowy
Victor A. Benassi
Overview
The ability to categorize is fundamental to human existence. Learning to assign objects in our
environment to separate categories allows us to recognize the things around us, to use language (e.g.,
common nouns refer to classes of objects rather than a specific object), and to respond appropriately to
different organisms (e.g., edible vs. poisonous plants). Category learning is also integral to achievement
across academic disciplines, whether in the arts and humanities (e.g., learning to identify a style of
painting or music; learning grammatical categories and their associated rules), or in STEM (e.g., learning
taxonomic classification in biology; learning to identify the kind of problem one is trying to solve so that
the right strategies might be employed).
In many real-world situations, and especially in education, it is more useful to identify patterns and
principles from past experiences and be able to apply this learning to new circumstances than to be able
to remember specific past examples, simply because it is less likely that one would face the exact same
example again in the future. Inductive learning of relevant previous encounters allows the formation of
mental categories, and this category learning then equips the learner to go beyond the prior cases and
categorize novel examples as being a member of the category (or not). In other words, the learning of
categories is an important way by which learners become able to generalize or transfer their learning to
new instances.
Cognitive scientists over the past decade have shown renewed interest in how category learning might
ibe improved or facilitated, and the sequencing of examples during training has been a particular area of
focus. Specifically, researchers have examined whether grouping together examples from the same
category during study (e.g., A1 A2 A3 A4 B1 B2 B3 B4 C1 C2 C3 C4), otherwise known as blocked training, or
mixing examples from different categories (e.g., A1 B1 C1 B2 A2 C2 A3 C3 B3 C4 A4 B4), also referred to as
interleaved training, is advantageous for learning. The bulk of the evidence supports the superiority of
interleaved over blocked training, although there are noteworthy boundary conditions (for recent meta-
analyses, see Brunmair & Richter, 2019; Firth et al., 2021). These findings have important potential
application in educational practice, especially considering that modifying how examples are presented in
an instructional sequence is often within the control of the teacher and does not require additional
resources or study time. In other words, with a bit of planning, interleaved training can be a feasible,
cost-effective strategy for improving student learning (Roediger & Pyc, 2012). In the remainder of this
chapter, I discuss the key research findings related to interleaved training and category learning (e.g.,
when and why interleaving is beneficial), and the practical implications for the design of instruction.
132
Research Evidence
Learners’ Beliefs and Perceptions
Imagine that you have not read the preceding chapter overview and are naïve to the topic of interleaved
training. Now, consider the following learning scenario (Morehead et al., 2016 [taken from McCabe,
2011]):
Two art history professors present 6 paintings by each of 12 artists (72 paintings total).
Professor A presents all six paintings by a single artist consecutively (i.e., grouped), and then
moves on to the next artist's six paintings, and so on, until all paintings have been presented.
Professor B presents the various artists' paintings in an intermingled fashion (i.e., mixed), such
that a single painting by a particular artist would be followed by a different artist’s. (p. 270)
Would you rate the teaching strategy of Professor A or B to be more effective for students’ learning to
recognize each artist’s style? If you thought that Professor B’s method was superior, you would be in the
minority. In a survey of college students and instructors, only 16% and 13%, respectively, gave a higher
effectiveness rating to Professor B than Professor A (Morehead et al., 2016; see also McCabe, 2011).
What is particularly interesting about these survey results is that the learning scenario closely resembles
the procedure of an actual experiment that was conducted (Kornell & Bjork, 2008), so we can compare
subjective beliefs or perceptions (i.e., survey responses) against objective learning (i.e., test
performance in the experiment).
In a seminal study, Kornell and Bjork (2008, Experiment 1b) presented college participants with 6
paintings each by 12 artists in either a blocked or interleaved fashion (the artist’s name accompanied
each painting during training). When later asked to classify previously unseen paintings by the same 12
artists, participants in the interleaved training group were more accurate at identifying the correct artist
than those in the blocked training group (59% vs. 36%). In a similar experiment (Kornell and Bjork, 2008,
Experiment 1a), participants experienced both training conditions (the paintings by half of the artists
were studied in an interleaved manner while the paintings by the other half were blocked), and again
interleaved training produced better category learning. A striking mismatch between subjective and
actual learning was also revealed: despite the majority of participants exhibiting better learning after
interleaved training, when surveyed after the test whether they thought interleaved or blocked training
was more effective for their learning, most judged that blocked training was superior.
Learners also show a preference for blocking when they have control over the sequencing of examples
during training. Tauber et al. (2013) found that when learning to categorize different types of birds,
participants overwhelmingly chose to view multiple examples of a particular bird family before switching
to another bird family. The preference for blocking or the belief that it is more effective for learning
(relative to interleaving) is likely due to (1) a greater fluency experienced when several examples of the
same category are studied consecutively, wherein this subjective ease of processing is interpreted as
learning, and (2) the common use of blocked training in education (Yan et al., 2016). Inaccurate
metacognitive beliefs or judgments, such as these, can be resistant to correction; I will discuss the
pedagogical implications later.
Interleaving Benefit for Category Learning
Since the study by Kornell and Bjork (2008), dozens of other studies have replicated the main finding
that interleaved (relative to blocked) training yields superior category learning (e.g., Kang & Pashler,
2012; Kornell et al., 2010). A key point is that across studies, the interleaved and blocked training
conditions each involved the same amount of training, identical presentation durations, and the exact
133
same examples; the only difference was the sequencing of the examples. Most of these studies have
investigated the learning of visual categories, in which participants were presented with examples of
images belonging to different categories during training and later asked to categorize previously unseen
examples from these categories. The visual materials have included paintings (e.g., Sana et al., 2018),
birds (e.g., Walheim et al., 2011), butterflies (e.g., Birnbaum et al., 2013), and various artificially
generated categories (e.g., Zulkiply & Burt, 2013b).
I will draw attention to a couple of other studies featuring visual categories that have clear educational
relevance. Eglington and Kang (2017) examined college students’ learning to categorize diagrams of
organic chemical compounds (e.g., alkanes, alkynes, carbamates), and they found that interleaving the
categories benefited learning, even when the diagnostic features were highlighted during study. This
finding is notable because it suggests that (1) the interleaving advantage can generalize to rule-based
categories (i.e., category membership can be defined by verbalizable rules; cf. much of the materials
used in previous studies, such as paintings, birds), and (2) interleaving is still effective when paired with
another useful instructional strategy (highlighting or using visual cues to draw learners’ attention to
particular features). There is also evidence that interleaving (relative to blocking) examples of different
radiographic (or x-ray) patterns (e.g., emphysema, pleural effusion, pneumonia) during training
enhances medical students’ ability to correctly interpret x-rays (Rozenshtein et al., 2016). Importantly,
the presentation of the x-ray examples during training was accompanied by a voice-over description of
the disease and the radiographic pattern, akin to how the cases would be presented in typical medical
education. Similar results have been found with medical students learning to interpret
electrocardiograms (Hatala et al., 2003).
Researchers have also examined the effects of interleaved training on auditory category learning. In an
auditory analogue of Kornell and Bjork’s (2008) study, Wong et al. (2020) found that interleaving
(relative to blocking) excerpts of classical music pieces by various composers during training facilitated
listeners’ ability to correctly classify novel pieces by the same composers. A follow-up study found an
interleaved training advantage for identifying musical intervals (Wong et al., 2021). Being able to
recognize composers’ styles and musical intervals are important skills for musicianship, and these
findings suggest that interleaved training can have useful applications in music education. In addition, a
study found that nursing students’ auscultation skills (using a stethoscope) improved more with
interleaved than blocked training, and students were better able to accurately classify various cardiac
and respiratory sounds, especially novel sounds (Chen et al., 2015, Study 3). These studies
demonstrating interleaved training benefits for auditory categorical learning provide a nice parallel to
the previously mentioned findings on interpreting x-rays and electrocardiograms.
Another broad type of material that has been researched in interleaved training studies are verbal
concepts. Zulkiply et al. (2012) examined college students’ learning of six types of psychological
disorders by studying three case studies of each type. The case studies were presented either
interleaved or blocked by disorder type and accompanied by made-up names of the disorders to
minimize the influence of prior knowledge or assumptions. Participants were then asked to categorize
new case studies, and the interleaved training group exhibited superior performance relative to the
blocked group. A similar pattern of results was observed by Sana et al. (2017, Experiment 1) in their
study of college students’ learning of statistics concepts (three nonparametric statistical tests). Word
problems were presented interleaved or blocked by concept during training, and each problem would
describe the research design appropriate for the listed statistical concept. Test results demonstrated
that participants in the interleaved training group were more accurate at classifying new problems. In
the above two studies, the definitions of the verbal concepts were not provided during the case study or
word problem presentations. It is probably more common for educators to present the definition before
134
giving examples to illustrate the concept. There is evidence that when students first study the
definitions for each concept followed by the presentation of illustrative examples, interleaving
(compared to blocking) the examples leads to more accurate categorization of new examples (Rawson
et al., 2015, Experiment 2).
A relatively new area of research on interleaved training has targeted second language (L2) grammar
learning, and the emerging findings are promising. In an experiment conducted with Japanese college
students who had been studying English for the past 6 years, Nakata and Suzuki (2019) found that
interleaved practice on five different English grammatical categories (simple tense, past tense, first,
second, and third conditional) produced more accurate grammatical judgments than blocked training
immediately and 1-week after training. In another study conducted with US college students who were
complete novices in Spanish (Pan, Lovelett, et al., 2019), when learning to conjugate Spanish verbs, an
interleaving advantage (over blocking) on a test (2-days later) emerged only when there was systematic
alternation (ABABAB) during the initial study trials and randomization during the subsequent practice
trials. This finding suggests that how interleaving is implemented may matter, especially when you
consider that novices may benefit from greater predictability during initial study. Given that many
factors influence L2 learning (Long & Doughty, 2009), more research is needed in order to gain greater
clarity on the optimal way to interleave for grammatical category learning.
The selection of studies discussed above demonstrates the breadth of learning materials that have been
featured in the research supporting an interleaving advantage for category learning, which increases
confidence that the effect will likely generalize to a wide range of educationally relevant materials. In
addition, a few of these studies had the categorization test occur a couple of days to a week after
training (rather than shortly after training), and the interleaving effect did replicate even with the
greater test delays (e.g., Eglington & Kang, 2017; Zulkiply, 2013; Zulkiply & Burt, 2013a), providing some
assurance that the benefit of interleaving persists over educationally-relevant time. There are, however,
a few known moderating factors which may eliminate or reverse the interleaving advantage; these will
be acknowledged in the next section.
A limitation of the existing research is that most of it was conducted on college students in lab-based
studies. There is evidence that the interleaving benefit for visual category learning and verbal concept
learning does not vary as a function of working memory capacity – i.e., individuals who measure high or
low in working memory capacity both benefit from interleaving (Sana et al., 2018; Yan & Sana, 2021).
Also, the interleaving advantage for learning painting styles has been replicated in older adults (Kornell
et al., 2010), and there is strong evidence that interleaved practice benefits children’s mathematics
problem solving (e.g., Nemeth et al., 2019; Rohrer & Hartwig, this volume). Future classroom studies
assessing the impact of interleaved training on the learning of curriculum-related categories and in
specific populations (e.g., students with learning difficulties, autism) would facilitate greater translation
of research findings into educational practice.
Why Does Interleaving Promote Category Learning, and What are the Boundary
Conditions?
Although the interleaving benefit is generally reliable and robust across individuals in the specific
domains and with the learning materials described earlier in this chapter, there are certain boundary
conditions under which interleaving is no longer advantageous or blocking becomes superior. In order to
better appreciate what these conditions are (or when they arise), it is helpful to understand why
interleaving promotes category learning. The dominant theory is that interleaving juxtaposes examples
from different categories, which facilitates appreciation of differences among categories (discriminative
contrast; Kang & Pashler, 2012; Wahlheim et al., 2011).
135
In learning a category, it is not enough to figure out how it differs from others; it is also crucial to
identify the commonalities within each category, a process that is facilitated when examples are
blocked. Therefore, whether a blocked or interleaved sequence will be optimal for learning depends
critically on where the difficulty of categorization lies (Carvalho & Goldstone, 2015). If the categories are
very similar to each other and it is very challenging to tell them apart, then interleaved training will be
superior, as it promotes discrimination. However, if within each category there is high variability, then
blocked training will be beneficial, as it directs attention toward the similarities within a category.
Indeed, there is evidence to support these predictions. Carvalho and Goldstone (2014) showed that
when they used artificially created visual categories that had low within- and between-category
similarity, blocking was superior for learning (see also Zulkiply & Burt, 2013b).
Category similarity also likely explains other prominent failures to obtain an interleaving benefit. For
instance, Carpenter and Mueller (2013) examined the learning of various French spelling-to-sound rules,
and found that blocked (relative to interleaved) presentation of French words pronounced aloud,
accompanied by their spelling, produced better learning of French pronunciation rules. Importantly,
there was no highlighting in the visual presentation of the spelling to indicate which part of the word
might be relevant for the target rule. It is likely that the French spelling-to-sound rules had minimal
overlap (low between-category similarity), and the items also had low within-category similarity (each
word contained other sounds and letters aside from the target spelling-to-sound rule), and interleaving
interfered with commonality abstraction and thus the learning of the categories.
In another study, Yan and Schuetze (2022) presented US college students with Chinese characters and
their English translations. Participants were told that the Chinese characters came from 5 different
categories (e.g., branch, bridge, cabinet, chair, pole belong to the WOOD category), but they were not
told what the categories were or of which category each character was a member. Perhaps
unsurprisingly, category learning was low, and interleaved training did not yield an advantage. However,
when participants were made aware that the characters contained a visual feature (a radical) that
indicated category membership, and they were further informed about the semantic category to which
each character belonged, interleaved training produced a strong benefit over blocked training for
accuracy at identifying the meaning of new characters and the semantic category of each of the 5
radicals. It seems that for these verbal categories, if the categories are too opaque, there needs to be
scaffolding to help learners become aware of the categories (e.g., making clear during training the
category to which each example belongs), without which interleaved training will not be effective.
Another factor that presents a boundary condition for the interleaving advantage is the time gap
between examples during training. If the advantage derives from the juxtaposition of examples from
different categories, then adding a time delay between successive examples would hinder the
comparing and contrasting process which promotes discrimination. Birnbaum et al. (2013) found that
inserting a trivia question between each successive example eliminated the interleaving advantage (see
also Sana et al., 2017), thus providing a warning against the inadvertent disruptive effects of superfluous
additions to the learning sequence.
In the next section, I will generate implications for instruction by synthesizing the empirical findings,
theoretical accounts, and boundary conditions related to interleaved training.
Practical Advice for Educators

There is substantial research literature on interleaved vs. blocked training and its impact on category
learning, a topic that has garnered considerable interest over the past 10 plus years. Most of the extant
research has been conducted in lab settings, and although that may raise some questions about
generalizability to educational settings, these lab studies often have strong methodological rigor and
136
control extraneous variables. They include replications of past work, allow for causal conclusions (in the
case of experiments), and contribute to theoretical advancement. Overall, the weight of the evidence
favors interleaved training as a beneficial instructional strategy (e.g., Firth et al., 2021), especially given
the ubiquity of category learning across academic disciplines and the low cost of incorporating
interleaving in one’s teaching practice.
The learning of verbal concepts (e.g., new terms and their definitions, examples which elucidate
concepts) is part and parcel of every subject or discipline. Some disciplines go further and require
perceptual category learning (e.g., learning to recognize visual images, objects). As such, there are many
opportunities for instructors to consider how examples should be sequenced during study or training.
Listed below are a few considerations.
Which Concepts or Content Should I Interleave?
Based on theory and empirical findings (e.g., Brunmair & Richter, 2019; Kang & Pashler, 2012),
interleaving is most effective when the categories are similar to each other. There is not much use in
intermixing highly disparate content (e.g., Indonesian vocabulary and human anatomy; Hausman &
Kornell, 2014). As a teacher, you are a subject matter expert and should have an idea (from assignments,
assessments, student feedback) of which concepts or categories your students tend to confuse.
Remember, though, that the confusion can sometimes be within a category (e.g., the category is highly
variable, and a student may have difficulty figuring out what the different examples from the category
have in common). Perhaps a simpler rule-of-thumb would be to juxtapose confusing examples. If the
examples come from different categories, then you would be interleaving the different categories,
which would draw students’ attention to the differences among the categories. If the examples come
from the same category, then you would be blocking (i.e., grouping the examples from that one
category together), which would draw students’ attention to the similarities within the category. Both
these complementary processes should help reduce confusion.
Also, keep in mind that the research literature I reviewed focused on the presentation of examples (e.g.,
images, brief case studies, vignettes), and we should not assume that the benefits of interleaved training
would extend to reading of longer texts, such as a book chapter. After all, longer chunks of content
would mean that there are fewer chances for juxtaposition, which prevents the comparing across
examples. Additionally, when presenting the examples, explicitly state the category name and make key
features salient to students when possible (Eglington & Kang, 2017; Yan & Schuetze, 2022).
When Should I Interleave?
Perhaps the obvious choice would be during classroom instruction when introducing a new concept.
Most of the reviewed research examined novices provided with new material to learn. But there is no
compelling reason to expect that interleaved training needs to be confined to the classroom and only at
the juncture when a new concept is being taught. There are strategies that instructors can use to
incorporate interleaved training (exposing students to examples intermixed across categories) in
practice assignments or homework, which could amplify interleaving benefits. For example, Pan, Tajran,
et al. (2019) found that interleaved training produced better Spanish grammatical category learning (i.e.,
ability to conjugate verbs in different tenses) than blocked training, but only if the training was across
two sessions that were 1 week apart. Interleaving across multiple sessions typically means that there is
an added benefit of spaced practice, which is another powerful strategy to enhance learning (see Kang,
2016, for a review).
137
How Should I Interleave?
If you rely heavily on traditional textbooks and learning resources for your teaching, you probably will
want to set aside a bit of time to plan. The conventional structure for most textbooks tends to feature a
linear, topical arrangement, with relatively few interleaved examples (Rohrer et al., 2020), which means
you will likely have to intentionally select and rearrange the sequencing of examples in your teaching
materials.
Aside from the planning, the other critical aspect is the selection or operationalisation of the actual
learning sequence. Although the vast majority of past research has compared (pure) interleaved vs.
(pure) blocked training, those two options are obviously not the only way to sequence examples. For
instance, having a mix of blocked and interleaved training could perhaps allow learners to benefit from
both the commonality abstraction supported by blocking and the discriminative contrast fostered by
interleaving. Yan et al. (2017) examined the efficacy of various hybrid schedules on participants’ learning
of painting styles, and found that a blocked-to-interleaved schedule (i.e., start out with blocked training
and then transition to interleaved training) was just as effective as an interleaved schedule.
Interestingly, when participants were asked to judge which of 6 different schedules was most effective
for learning, the blocked-to-interleaved option was most frequently endorsed, suggesting perhaps some
appreciation of the benefit of interleaving (but the pure interleaved schedule was the least endorsed,
even though it was one of the most effective schedules).
Metacognitive Considerations Surrounding Interleaved Training
As discussed earlier, there is ample evidence that students (and also instructors) tend to perceive
blocked training as more effective than interleaved training (e.g., Kornell et al., 2010; Morehead et al.,
2016). This metacognitive illusion is a barrier to greater uptake of interleaved practice, as students are
unlikely to embrace learning strategies that they think are unproductive, and they might also regard
teachers who promote such strategies as ineffective. Learning strategies that feel more effortful are
interpreted by students as less effective for learning, and hence, less likely to be selected in the future
(Kirk-Johnson et al., 2019). Therefore, it is crucial to find ways to correct learners’ faulty beliefs. But
these beliefs can be very resistant to change. Yan et al. (2016) found that when participants experienced
blocked and interleaved training for learning painting styles, the majority (72%) judged blocked training
to be more effective for their learning, despite actual learning showing an interleaving advantage. They
tried to correct this metacognitive illusion by providing participants with information about interleaved
training being more effective for 90% of people, asking them to ignore their sense of fluency during
blocked training, and giving an explanation for why interleaving is beneficial, yet almost half still stuck to
their belief that blocked training was superior. To change more minds, it was necessary to also have the
blocked and interleaved training and test be experienced separately, so that test performance could be
connected unambiguously to the appropriate training condition (Yan et al., 2016, Experiment 6).
One approach is to capitalize on students’ greater openness to hybrid blocked-to-interleaved schedules
(Yan et al., 2017). In other words, start with blocked training (something to which students are already
quite accustomed), and then incorporate progressively greater interleaving. Another possibility is for
teachers to conduct demonstrations of the interleaving effect in their classrooms with their students,
either using paintings by different artists (like in Kornell & Bjork, 2008) or using other kinds of categories
that might be more germane to the teacher’s subject. Importantly, divide categories into two lists and
have the blocked and interleaved training and test done separately. The point of the demonstration is
for students to be able to clearly link their test outcome to a given training condition. Hopefully, a better
outcome after interleaved training is obtained, which together with a debriefing containing information
138
about blocked vs. interleaved training and the purpose of the class mini-experiment, will help establish
correct beliefs and knowledge about interleaved training among your students.
The prevailing teaching norms favor blocked training (Kang, 2017). Blocking feels effective for learning,
instructional materials facilitate it, and teachers and students are used to it. But there is a growing body
of research showing that increasing the degree of interleaving during training is often beneficial for
inductive learning. As a teacher, you can decide how examples are presented in class and how practice is
sequenced. Although incorporating greater interleaving will require some effort and planning, it does
not require a dramatic change in your current practice and need not be onerous. May you encounter
many successes as you explore inventive ways of including and promoting interleaved training and other
empirically supported strategies in your classroom.
Author Note
Correspondence concerning this article should be addressed to Sean Kang, Melbourne Graduate School
of Education, 100 Leicester Street, Level 6, Carlton, VIC 3053, Australia. E-mail:
[email protected]
References
Birnbaum, M. S., Kornell, N., Bjork, E. L., & Bjork, R. A. (2013). Why interleaving enhances inductive
learning: The roles of discrimination and retrieval. Memory & Cognition, 41(3), 392–402.
Brunmair, M., & Richter, T. (2019). Similarity matters: A meta-analysis of interleaved learning and its
moderators. Psychological Bulletin, 145(11), 1029–1052. https://doi.org/10.1037/bul0000209
Carvalho, P. F., & Goldstone, R. L. (2014). Putting category learning in order: Category structure and
temporal arrangement affect the benefit of interleaved over blocked study. Memory &
Cognition, 42(3), 481–495. https://doi.org/10.3758/s13421-013-0371-0
Carvalho, P. F., & Goldstone, R. L. (2015). What you learn is more than what you see: What can
sequencing effects tell us about inductive category learning? Frontiers in Psychology, 6, 505.
Chen, R., Grierson, L., & Norman, G. (2015). Manipulation of cognitive load variables and impact on
auscultation test performance. Advances in Health Sciences Education, 20(4), 935–952.
Eglington, L. G., & Kang, S. H. K. (2017). Interleaved presentation benefits science category
learning. Journal of Applied Research in Memory and Cognition, 6(4), 475-485.
Firth, J., Rivers, I., & Boyle, J. (2021). A systematic review of interleaving as a concept learning strategy.
Review of Education, 9(2), 642–684. https://doi.org/10.1002/rev3.3266
Hatala, R. M., Brooks, L. R., & Norman, G. R. (2003). Practice makes perfect: The critical role of mixed
practice in the acquisition of ECG interpretation skills. Advances in Health Sciences
Education, 8(1), 17–26. https://doi.org/10.1023/a:1022687404380
139
Hausman, H., & Kornell, N. (2014). Mixing topics while studying does not enhance learning. Journal of
Applied Research in memory and Cognition, 3(3), 153–160.
Kang, S. H. K. (2016). Spaced repetition promotes efficient and effective learning: Policy implications for
instruction. Policy Insights from the Behavioral and Brain Sciences, 3(1), 12–19.
Kang, S. H. K. (2017). The benefits of interleaved practice for learning. In J. C. Horvath, J. M. Lodge, & J.
Hattie (Eds.), From the laboratory to the classroom: Translating science of learning for teachers
(pp. 79–93). Routledge.
Kang, S. H. K., & Pashler, H. (2012). Learning painting styles: Spacing is advantageous when it promotes
discriminative contrast. Applied Cognitive Psychology, 26(1), 97–103.
Kirk-Johnson, A., Galla, B. M., & Fraundorf, S. H. (2019). Perceiving effort as poor learning: The
misinterpreted-effort hypothesis of how experienced effort and perceived learning relate to
study strategy choice. Cognitive Psychology, 115, 101237.
https://doi.org/10.1016/j.cogpsych.2019.101237
Psychological Science, 19(6), 585–592. https://doi.org/10.1111/j.1467-9280.2008.02127.x
Kornell, N., Castel, A. D., Eich, T. S., & Bjork, R. A. (2010). Spacing as the friend of both memory and
induction in young and older adults. Psychology and Aging, 25(2), 498–503.
Long, M. H., & Doughty, C. J. (Eds.). (2009). The handbook of language teaching. Wiley-Blackwell.
Nakata, T., & Suzuki, Y. (2019). Mixing grammar exercises facilitates long‐term retention: Effects of
blocking, interleaving, and increasing practice. The Modern Language Journal, 103(3), 629-647.
https://doi.org/10.1111/modl.12581
Nemeth, L., Werker, K., Arend, J., Vogel, S., & Lipowsky, F. (2019). Interleaved learning in elementary
school mathematics: Effects on the flexible and adaptive use of subtraction strategies. Frontiers
in Psychology, 10(article 86). https://doi.org/10.3389/fpsyg.2019.00086
Pan, S. C., Lovelett, J. T., Phun, V., & Rickard, T. C. (2019). The synergistic benefits of systematic and
random interleaving for second language grammar learning. Journal of Applied Research in
Memory and Cognition, 8(4), 450-462. https://doi.org/10.1016/j.jarmac.2019.07.004
Pan, S. C., Tajran, J., Lovelett, J., Osuna, J., & Rickard, T. C. (2019). Does interleaved practice enhance
foreign language learning? The effects of training schedule on Spanish verb conjugation
skills. Journal of Educational Psychology, 111(7), 1172–1188.
Rawson, K. A., Thomas, R. C., & Jacoby, L. L. (2015). The power of examples: Illustrative examples
enhance conceptual learning of declarative concepts. Educational Psychology Review, 27(3),
140
Roediger, H. L. III, & Pyc, M. A. (2012). Inexpensive techniques to improve education: Applying cognitive
Cognition, 1(4), 242–248. https://doi.org/10.1016/j.jarmac.2012.09.002
Rohrer, D., Dedrick, R. F., & Hartwig, M. K. (2020). The scarcity of interleaved practice in mathematics
textbooks. Educational Psychology Review, 32(3), 873–883. https://doi.org/10.1007/s10648-
020-09516-2
Rozenshtein, A., Pearson, G. D., Yan, S. X., Liu, A. Z., & Toy, D. (2016). Effect of massed versus interleaved
teaching method on performance of students in radiology. Journal of the American College of
Radiology, 13(8), 979–984. https://doi.org/10.1016/j.jacr.2016.03.031
Tauber, S. K., Dunlosky, J., Rawson, K. A., Wahlheim, C. N., & Jacoby, L. L. (2013). Self-regulated learning
of a natural category: Do people interleave or block exemplars during study? Psychonomic
Bulletin & Review, 20(2), 356–363. https://doi.org/10.3758/s13423-012-0319-6
Sana, F., Yan, V. X., & Kim, J. A. (2017). Study sequence matters for the inductive learning of cognitive
concepts. Journal of Educational Psychology, 109(1), 84–98.
Sana, F., Yan, V. X., Kim, J. A., Bjork, E. L., & Bjork, R. A. (2018). Does working memory capacity moderate
the interleaving benefit? Journal of Applied Research in Memory and Cognition, 7(3), 361–369.
Wahlheim, C. N., Dunlosky, J., & Jacoby, L. L. (2011). Spacing enhances the learning of natural concepts:
An investigation of mechanisms, metacognition, and aging. Memory & Cognition, 39(5), 750–
763. https://doi.org/10.3758%2Fs13421-010-0063-y
Wong, S. S. H., Chen, S., & Lim, S. W. H. (2021). Learning melodic musical intervals: To block or to
interleave? Psychology of Music, 49(4), 1027–1046.
Wong, S. S. H., Low, A. C. M., Kang, S. H. K., & Lim, S. W. H. (2020). Learning music composers’ styles: To
block or to interleave? Journal of Research in Music Education, 68(2), 156–174.
Psychology: General, 145(7), 918–933. https://doi.org/10.1037/xge0000177
Yan, V. X., & Sana, F. (2021). The robustness of the interleaving benefit. Journal of Applied Research in
Memory and Cognition, 10(4), 589–602. https://doi.org/10.1037/h0101863
Yan, V. X., & Schuetze, B. A. (2022). Not just stimuli structure: Sequencing effects in category learning
vary by task demands. Journal of Applied Research in Memory and Cognition, 11(2), 218–
Yan, V. X., Soderstrom, N. C., Seneviratna, G. S., Bjork, E. L., & Bjork, R. A. (2017). How should exemplars
be sequenced in inductive learning? Empirical evidence versus learners’ opinions. Journal of
Experimental Psychology: Applied, 23(4), 403–416. https://doi.org/10.1037/xap0000139
Zulkiply, N. (2013). Effect of interleaving exemplars presented as auditory text on long-term retention in
inductive learning. Procedia-Social and Behavioral Sciences, 97(6), 238–245.
https://doi.org/10.1016/j.sbspro.2013.10.228
141
Zulkiply, N., & Burt, J. S. (2013a). Inductive learning: Does interleaving exemplars affect long-term
retention? Malaysian Journal of Learning and Instruction, 10, 133-155. https://e-
journal.uum.edu.my/index.php/mjli/article/view/7655
Zulkiply, N., & Burt, J. S. (2013b). The exemplar interleaving effect in inductive learning: Moderation by
the difficulty of category discriminations. Memory & Cognition, 41(1), 16–27.
Zulkiply, N., McLean, J., Burt, J. S., & Bath, D. (2012). Spacing and induction: Application to exemplars
presented as auditory and visual text. Learning and Instruction, 22(3), 215–221.
142
In Their Own Words Research-Based Principles for Designing
Multimedia Instruction
to Apply the Science of Learning in Richard E. Mayer
University of California, Santa Barbara
Editors
Lauren L. Kordonowy
Victor A. Benassi
Overview of Multimedia Instruction

People learn more deeply from words and graphics than from words alone. This assertion can be called
the multimedia principle, and it forms the basis for using multimedia instruction—that is, instruction
containing words (such as spoken text or printed text) and graphics (such as illustrations, charts, photos,
animation, or video) that is intended to foster learning (Mayer, 2021, in press-a).
For example, Figure 1 shows frames from a narrated animation on how a tire pump works. In this case
the words are spoken, and the graphics are presented as an animation. Other examples include
textbook lessons presented on paper, slideshow presentations presented face-to-face, instructional
video or narrated animation or interactive tutorials presented via computer, educational games and
simulations presented on hand-held devices, and immersive virtual reality or augmented reality learning
experiences presented via head mounted displays. Regardless of presentation medium, what makes all
these examples of multimedia instruction is that they use words and graphics to promote learning.
Figure 1
Frames from a Narrated Animation on How a Tire Pump Works
143
What is the evidence for the multimedia principle? In a series of 13 experimental comparisons my
colleagues and I have found that students perform much better on a transfer test when they learn from
words and graphics than from words alone (e.g., narration and animation versus narration alone, or text
and illustrations versus text alone), yielding a median effect size of d = 1.35 (Mayer, 2021). For example,
students are better able to answer troubleshooting questions about tire pumps after viewing the
narrated animation shown in Figure 1 than after solely hearing the narration (Mayer & Anderson, 1991).
In short, research shows that multimedia instruction has the potential to greatly improve student
understanding of how things work, including tire pumps, brakes, generators, and lightning (Mayer, 2021,
in press-a).
Although instruction has traditionally emphasized verbal modes of presentation (such as lectures and
printed books), recent advances in graphics technology now allow more widespread incorporation of
visual modes of presentation including illustrations, charts, photos, animation, and video in
presentations and in interactive venues such as intelligent tutoring systems, games, simulations, and
immersive virtual reality. However, not all graphics are equally effective so careful research is needed to
pinpoint principles of multimedia instructional design. The goal of this chapter is to provide a brief
overview of research-based principles for how to design effective instruction that uses words and
graphics, based on research carried out by my colleagues and me and described more fully in Mayer
(2021). A broader review of multimedia design principles is available in Mayer and Fiorella (in press-a).
An important starting point is to examine principles that are based on an understanding of how people
learn from words and graphics. Figure 2 summarizes the cognitive theory of multimedia learning (Mayer,
2021, in press-b), which is based on three core assumptions derived from the science of learning (Mayer,
2011): dual channel assumption—people have separate channels for processing visual and verbal
material (Paivio, 1986); limited capacity assumption—people can process only a limited amount of
material in a channel at any one time (Baddeley, 1999; Sweller et al., 2011); and active processing
assumption—meaningful learning occurs when learners select relevant material, organize it into a
coherent structure, and integrate it with relevant prior knowledge (Fiorella & Mayer, 2015; Mayer,
2021; Wittrock, 1989).
Figure 2 represents memory stores as rectangles: sensory memory, which temporarily holds incoming
images and sounds; working memory, which allows for mentally manipulating a small amount of the
incoming visual and verbal material; and long-term memory, which is the learner’s permanent
storehouse of knowledge. Figure 2 represents cognitive processing as arrows: selecting, which transfers
some of the incoming images and sounds to working memory for additional processing; organizing,
which organizes the images into a pictorial model and the words into a verbal model in working
memory; and integrating, which connects the pictorial and verbal models with each other and with
relevant knowledge activated from long-term memory. A multimedia message enters the cognitive
system through the learner’s ears and eyes. The top row represents the verbal channel (into which
spoken words and sounds enter) and the bottom row represents the visual channel (into which graphics
and printed words enter), although in working memory printed words can be converted into sounds and
images can be converted into spoken words. In working memory, the learner selects relevant words and
images, organizes them into coherent verbal and pictorial representations, and integrates the
representations with each other and with relevant knowledge activated from long-term memory.
144
Figure 2:
Cognitive Theory of Multimedia Learning
MULTIMEDIA SENSORY LONG-TERM
PRESENTATION MEMORY WORKING MEMORY MEMORY
selecting Sounds organizing Verbal

Words Ears words words Model
integrating
Prior
Prior
Knowledge
Knowledge
selecting Images organizing Pictorial
Pictures Eyes images images Model
Table 1 summarizes three kinds of processing that can occur during multimedia instruction—extraneous
processing, which drains limited cognitive processing capacity without contributing to learning and is
caused by poor instructional design; essential processing, which involves selecting relevant information
and organizing it in working memory as presented and is caused by the complexity of the material for
the learner; and generative processing, which involves making sense of the material by reorganizing it
into a coherent structure and integrating it with relevant prior knowledge and is caused by the learner's
motivation to exert effort towards understanding. This analysis is similar to that proposed in cognitive
load theory (Paas & Sweller, in press; Sweller, Ayres, & Kalyuga, 2011) and suggests the need for three
kinds of instructional design goals—reducing extraneous processing, which applies when extraneous
processing and required essential processing exceed the learner’s cognitive capacity; managing essential
processing, which applies when the required essential processing exceeds the learner’s cognitive
capacity; and fostering generative processing, which applies when the learner has processing capacity
available but chooses not to exert the effort to use it for making sense of the material. These three types
of goals form the basis for three kinds of instructional design principles for multimedia learning, which
are presented in the next section.
Table 1
Three Kinds of Cognitive Processing During Learning
Cognitive Description Instructional goal

processing
Extraneous Not related to instructional goal, Reduce extraneous
caused by poor instructional design processing
Essential Aimed at representing essential material, Manage essential
caused by complexity of material processing
Generative Aimed at making sense of essential Foster generative
material, caused by learner’s effort processing
Research on Design Principles for Multimedia Instruction

In this section I examine 13 research-based principles for how to design multimedia, including five
principles aimed at reducing extraneous processing, three principles aimed at managing essential
processing, and five principles aimed at fostering generative processing. For each principle, we
145
conducted a meta-analysis and computed the median effect size based on Cohen’s d (Cohen, 1988).
Following Hattie (2009), we consider any effect size greater than d = 0.40 to be educationally important.
Reduce Extraneous Processing

Table 2 summarizes five principles aimed at reducing extraneous processing based on research
conducted by my colleagues and me: the coherence, signaling, redundancy, spatial contiguity, and
temporal contiguity principles. The table specifies the number of experimental tests in which positive
results were obtained and provides the median effect size based on a meta-analysis by Mayer (2021). A
learner experiences extraneous overload when essential cognitive processing required to understand
the essential material in a multimedia message and extraneous cognitive processing required to process
extraneous material exceeds the learner's cognitive capacity. These five principles are intended to
address the problem of extraneous overload by reducing extraneous cognitive processing.
Table 2
Five Research-Based Principles Based on Reducing Extraneous Processing
Principle Description Tests Effect size

Coherence Delete extraneous material 18 of 19 0.86
Signaling Highlight essential material 26 of 28 0.70
Redundancy Don’t add onscreen captions 8 of 12 0.10
to narrated graphics
Spatial contiguity Place printed words near 9 of 9 0.82
corresponding part of graphic
Temporal contiguity Present spoken words at same 8 of 8 1.31
time as corresponding graphics
Note. Based on Mayer (2021).
The Coherence Principle

The first technique for reducing extraneous processing is the coherence principle: People learn more
deeply from a multimedia message when extraneous material is excluded rather than included. The
rationale for the coherence principle is that people are better able to focus on the essential material if
we eliminate extraneous material that could distract them. This principle was supported in 18 out of 19
experimental tests, yielding a median effect size of 0.86. For example, students who learned from a
multimedia lesson on how a virus causes a cold performed better on a transfer test if the lesson did not
contain seductive details—sentences that gave interesting but irrelevant facts about viruses (Mayer,
Griffith, Jurkowitz, & Rothman, 2008). Similarly, students who learned from a narrated animation on
lightning formation performed better on a transfer test if the lesson did not also contain short video
clips depicting lightning strikes (Mayer, Heiser, & Lonn, 2001). Overall, similar effects have been noted in
the broader literature (Fiorella & Mayer, in press-a; Rey, 2012; Sundarajan & Adesope, 2020).
Concerning boundary conditions, reviews of the coherence principle suggest the effects may be
strongest for learners with low rather than high working memory capacity, when the lesson is system-
paced rather than learner-paced, and when the extraneous material is highly interesting rather than
neutral (Mayer & Fiorella, in press-a; Rey, 2012; Sundarajan & Adesope, 2020).
146
The Signaling Principle
The second technique for reducing extraneous processing is the signaling principle: People learn more
deeply from a multimedia message when cues are added that highlight the organization of the essential
material. The rationale for the signaling principle is that people will learn more efficiently if the lesson is
designed to call their attention to the important material in the lesson and how it is organized. This
principle was supported in 26 out of 28 experimental tests, yielding a median effect size of 0.70.
Signaling of the verbal material includes using an outline, headings, highlighting (such as underlining)
and pointer words (such as first, second, third), as well as placing stress on spoken words. Verbal
signaling was effective in 15 out of 16 tests, yielding a median effect size of 0.69. For example, in a
narrated animation on how an airplane achieves lift, students performed better on a transfer test if the
narration included an initial outline, headings, and voice emphasis on key words (Mautone & Mayer,
2001). Signaling of visual material includes arrows, flashing, spotlighting, and pointing. Visual signaling
was effective in 11 of 12 tests, yielding a median effect size of 0.71. Overall, signaling effects have been
documented in the broader research literature (Richter, Scheiter, & Eitel, 2016; Schneider, Beege, Nebel,
& Rey, 2018; van Gog, in press). Concerning boundary conditions, the signaling principle may apply most
strongly when the learner might otherwise be overwhelmed with extraneous processing—such as, for
low-knowledge learners rather than high-knowledge learners, for complex material rather than simple
material, and when visual and verbal signals are used sparingly (Fiorella & Mayer, in press-a; Mayer,
2021).
Redundancy Principle
The third technique for reducing extraneous processing is the redundancy principle: People do not learn
more deeply when on-screen text is added to narrated graphics. The rationale is that with redundant
presentations people may waste precious processing capacity by trying to reconcile the two verbal
streams of information (i.e., printed and spoken text) or may focus on the printed words rather than the
relevant portions of the graphics. This principle was supported in 8 out of 12 experimental tests, yielding
a median effect size of 0.10 favoring no onscreen text. For example, Moreno and Mayer (2002) reported
that students performed better on a transfer test about lightning if they received a narrated animation
about lightning formation rather than the same narrated animated with concurrent onscreen text
inserted at the bottom of the screen. This work is consistent with the broader research literature on
redundancy (Adesope & Nesbit, 2012; Kalyuga & Sweller, in press). Concerning boundary conditions, the
redundancy effect can be diminished or even reversed when the lesson is slow-paced or under learner
control, the on-screen text is short or in a reworded form, the words are in the learner's second
language, or the material lacks graphics (Fiorella & Mayer, in press-a; Mayer, 2021).
The Spatial Contiguity Principle
The fourth technique for reducing extraneous processing is the spatial contiguity principle: People learn
more deeply from a multimedia message when corresponding printed words and graphics are presented
near rather than far from each other on the page or screen. The rationale is that spatial contiguity helps
learners build connections between corresponding words and graphics. This principle was supported in
9 out of 9 experimental tests, yielding a median effect size of 0.82. For example, Moreno and Mayer
(1999) found that students performed better on a transfer test after viewing an animation about
lightning in which printed words were placed next to the part of the lightning system they described
than when printed words were placed at the bottom of the screen as a caption. Similar results are
reported in meta-analyses by Ginns (2006) and Schroder and Cenkci (2018) as well as in a review by
Ayres and Weller (in press). Fiorella and Mayer (in press-a) report there is preliminary evidence that the
147
spatial contiguity principle may be strongest for low prior knowledge learners, non-redundant text and
pictures, complex lessons, and interactive formats.
The Temporal Contiguity Principle
The fifth technique for reducing extraneous processing is the temporal contiguity principle: People learn
more deeply from a multimedia message when corresponding graphics and narration are presented
simultaneously rather than successively. The rationale is that temporal contiguity helps learners build
connections between corresponding words and graphics. This principle was supported in 8 out of 8
experimental tests, yielding a median effect size of 1.31. For example, students performed better on a
transfer test when they received a narrated animation on how a tire pump works than when they heard
the narration before or after the animation (Mayer and Anderson, 1991). Researchers have noted that
the temporal contiguity principle may apply most strongly when lessons are long rather than short and
when the lesson is system-paced rather than learner paced (Fiorella & Mayer, in press-a; Mayer, 2021).
Manage Essential Processing

Table 3 summarizes three principles aimed at managing essential processing: the segmenting, pre-
training, and modality principles. The table specifies the number of experimental tests in which positive
results were obtained and provides the median effect size based on a meta-analysis by Mayer (2021).
These principles are intended to address the instructional problem of essential overload, which can
occur when a fast-paced multimedia lesson contains material that is complicated for the learner. A
learner experiences essential overload when the amount of essential cognitive processing required to
understand the multimedia instructional message exceeds the learner's cognitive capacity.
Table 3
Three Research-Based Principles Based on Managing Essential Processing

Segmenting Break lesson into learner-paced parts 7 of 7 0.67
Pre-training Present characteristics of key concepts 10 of 10 0.78
before lesson
Modality Use spoken words rather than printed 18 of 19 1.00
words

The Segmenting Principle
The first technique for managing essential processing is the segmenting principle: People learn more
deeply when a multimedia message is presented in learner-paced segments rather than as a continuous
unit. The rationale is that segmenting allows people to fully process one step in the process before
having to move onto the next one. This principle was supported in 7 out of 7 experimental tests, yielding
a median effect size of 0.67. For example, Mayer and Chandler (2001) found that students performed
better on a transfer test if a narrated animation about lightning was broken into 16 10-second segments
in which students could press a “continue” button to go on to the next segment.
Some potential boundary conditions are that segmenting may have stronger effects when the material is
complex for the learner and when the lesson is otherwise fast-paced (Mayer & Fiorella, in press-b).
148
The Pre-training Principle
The second technique for managing essential processing is the pre-training principle: People learn more
deeply from a multimedia message when they have learned the names and characteristics of the main
concepts. The rationale is that pre-training allows students to focus on the causal connections in the
multimedia explanation because they already know the names and characteristics of the key elements.
This principle was supported in 10 out of 10 experimental tests, yielding a median effect size of 0.78. For
example, students performed better on a transfer test based on a narrated animation on how brakes
work if before the lesson they were introduced to the names and characteristics of key components
mentioned in the lesson such as the piston and brake shoe (Mayer, Mathias, & Wetzell, 2002). However,
some important boundary conditions are that the pre-training principle may not apply to high prior
knowledge learners (perhaps because they are less likely to experience essential overload), when the
material is simple for the learner, and when the lesson is slow-paced or under learner control (Mayer,
2021; Mayer & Fiorella, in press-b).
The Modality Principle
The third technique for managing essential processing is the modality principle: People learn more
deeply from a multimedia message when the words are spoken rather than printed. The rationale is that
the modality principle allows learners to off-load some of the processing in the visual channel (i.e., the
printed captions) onto the verbal channel, thereby freeing more capacity in the visual channel for
processing the graphics. This principle was supported in 18 out of 19 experimental tests, yielding a
median effect size of 1.00. For example, Moreno and Mayer (1999) found that students performed
better on a transfer test after receiving a narrated animation on lightning formation than after receiving
the same animation with on-screen captions that contained the same words as the narration. Similar
results are reported in reviews and meta-analyses covering the broader literature, subject to boundary
conditions (Castro-Alonso & Sweller, in press; Ginns, 2005; Reinwein, 2012). As the most studied
principle in the list, research shows that the modality principle should not be taken to mean that spoken
words are better than printed words in all situations. Some important boundary conditions reported in
some studies are that printed words may be effective when the verbal material contains technical
terms, is in the learner’s second language, or when the lesson is learner-paced; whereas spoken words
may be effective when the lesson is fast-paced, is presented in segments that are too large to be held in
the learner’s working memory, and contains words that are familiar to the learner (Jiang & Sweller, in
press; Mayer, 2021; Mayer & Fiorella, in press-b).
Foster Generative Processing

Table 4 summarizes five principles aimed at fostering generative processing: the personalization, voice,
embodiment, image, and generative activity principles. The table specifies the number of experimental
tests in which positive results were obtained and provides the median effect size based on a meta-
analysis by Mayer (2021). Most of these principles use social cues to attempt to prime the learner’s
motivation to exert effort to make sense of the material. Social cues in a multimedia message such as
conversational style, voice, and gesture are intended to prime a sense of social presence in learners that
leads to deeper cognitive processing during learning and hence better test performance (Fiorella &
Mayer, in press-b).
149
Table 4
Five Research-Based Principles Based on Fostering Generative Processing

Personalization Put words in conversational style 13 of 15 1.00
rather than formal style
Voice Put words in human voice rather than 6 of 7 0.74
machine voice
Embodiment Have onscreen agent use human-like 16 of 17 0.58
gestures and movements
Image Do not necessarily put static image of 4 of 7 0.19
agent on the screen
Generative activity Prompt learners to engage in 37 of 44 0.71
Generative learning activities

The Personalization Principle
The first technique for fostering generative processing is the personalization principle: People learn
more deeply when the words in a multimedia presentation are in conversational style rather than
formal style. The rationale for this technique is that conversational style can prime a sense of social
presence in the learner, which causes the learner to try harder to make sense of what the instructor is
saying by engaging in appropriate cognitive processing during learning, leading to learning outcomes
that are better able to support problem-solving transfer. This principle was supported in 13 out of 15
experimental tests, yielding a median effect size of d = 1.00 (Mayer, 2021). For example, Mayer, Fennell,
Farmer and Campbell (2004) found that students performed better on a transfer test after receiving a
narrated animation on how the human respiratory system works when conversational wording was used
(e.g., “your lungs,” or “your nose”) rather than formal style (e.g., “the lungs” or “the nose”). Similar
results are reported in a meta-analysis by Ginns, Martin, and Marsh (2013). Some important boundary
conditions are that the personalization principle may not apply for experienced learners or long lessons
(Fiorella & Mayer, in press-b; Mayer, 2021).
The Voice Principle
The second technique for fostering generative processing is the voice principle: People learn more
deeply when the words in a multimedia message are spoken in an appealing human voice rather than in
a machine voice. An appealing human voice is intended to prime a sense of social presence in learners.
For example, Mayer, Sobko, and Mautone (2003) found that students performed better on a transfer
test after receiving a narrated animation on lightning that used a human voice rather than a machine
synthesized voice. This principle was supported in 6 out of 7 experimental comparisons, with a median
effect size of 0.74. More recently, Craig and Schroder (2017, 2018) found that advances in modern text-
to-speech engines can allow them to be more human-like. In a broader review of the instructional
effects on onscreen agents, Schroeder and Adesope (2014, p. 247) concluded that "voice may be critical,
whereas the image of the agent is not." Research shows that a possible boundary condition is that the
voice principle may not apply when there are negative social cues such as low embodiment (Fiorella &
Mayer, in press-b; Mayer, 2021).
150
The Embodiment Principle
The third technique for fostering generative processing is the embodiment principle: People learn more
deeply when onscreen agents display human-like gesturing, movement, eye contact, and facial
expression. Human-like action is intended to create a sense of social presence with the instructor. In 16
out of 17 experimental comparisons, people performed better on transfer tests when they learned from
a high-embodied agent than from a low-embodied agent, yielding a median effect size of 0.58. For
example, Mayer and DaPra (2012) found that students performed better on a transfer test after viewing
an online slideshow that was narrated by an onscreen agent who used human-like gesture, facial
expression, eye gaze, and movement than an onscreen agent who did not move, gesture, gaze, or show
expression. As another example, Fiorella and Mayer (2016) reported that students performed better on
a transfer test after watching a video showing an instructor drawing on a board as she lectured rather
than gesturing towards already drawn graphics. As yet another example of embodiment, Fiorella, van
Gog, Hoogerheide, and Mayer (2017) found that students learned better from a video demonstration
filmed from a first-person perspective (i.e., with camera over the instructor's shoulder) rather than a
third-person perspective (i.e., with the camera across from the instructor). A meta-analysis involving the
broader literature determined that students learn better from multimedia lessons when onscreen
agents use pointing gestures (Davis, 2018). A possible boundary condition is that the embodiment
principle may not apply when there are negative social cues such as machine voice (Fiorella & Mayer, in
press-b; Mayer, 2021).
The Image Principle
The fourth technique in Table 4 is the image principle: People do not necessarily learn more deeply from
a multimedia presentation when the speaker's image is on the screen rather than not on the screen.
Having a static image may cause distraction that detracts from any social benefits. For example, Mayer,
Dow, and Mayer (2003) found that adding the image of an onscreen character did not improve learning
much (d = 0.19) from a narrated animation on how electric motors work. This principle is based on 7
experimental tests in which 3 produced negative effects, yielding a median effect size of 0.19. In a
review of research on onscreen agents, Schroeder and Adesope (2014) also concluded that a static
image of the agent on the screen does not boost learning. Concerning boundary conditions, the image
principle applies for low-embodiment onscreen agents, that is, onscreen agents that do not move
(Fiorella & Mayer, in press-b; Mayer, 2021).
The Generative Activity Principle
Finally, another way to foster generative processing during learning is to prompt and guide students in
carrying out generative activities during learning, consistent with generative learning theory (Fiorella &
Mayer, 2015, in press-c; Wittrock, 1989). Generative learning theory holds that deep learning occurs
when learners mentally construct new knowledge through the processes of reorganizing incoming
information and integrating it with relevant prior knowledge. The generative activity principle states:
People learn more deeply when they are prompted and guided in carrying out generative learning
activities during learning. Our research has shown that students perform better on transfer tests when
they are prompted to carry out generative learning activities during learning from a lesson, such as
producing a verbal summary or a concept map, making or imagining an illustration corresponding to
presented text, taking a practice test on the material, explaining the material to oneself or others, or
physically moving objects to act out the lesson. In 37 out of 44 experimental comparisons, students who
were prompted to engage in these generative learning activities achieved higher transfer test scores
than those who were not, yielding a median effect size of 0.71. In reviews of the broader literature,
Fiorella and Mayer (2015) found similar positive effects on learning outcomes when students are
151
prompted and guided to engage in summarizing (d = 0.50 based on 30 tests), mapping (d = 0.62 based
on 39 tests), drawing (d = 0.40 based on 28 tests), imagining (d = 0.65 based on 22 tests), self-testing (d
= 0.57 based on 76 tests), self-explaining (d = 0.61 based on 54 tests), teaching (d = 0.77 based on 19
tests), and enacting (d = 0.51 based on 49 tests). Concerning boundary conditions, generative learning
activities are most powerful when learners have appropriate training and guidance in how to use them,
and when the activities are properly scaffolded so as to not be too cognitively demanding (Fiorella &
Mayer, in press-c; Mayer, 2021).
Practical Application of Design Principles for Multimedia Instruction

The principles summarized in this chapter are based on research and grounded in cognitive theory, but
more work is needed to better delineate the boundary conditions under which the principles apply and
to extend the domain of study. In particular, most of the supporting research involves short-term
laboratory studies, so it is useful to determine the degree to which the principles apply in more
authentic learning situations such as in schools, online courses, or workplace training (Clark, in press;
Clark & Mayer, 2016). For example, a promising step in this direction involves a finding by Issa et al.
(2013) showing that redesigning a medical school classroom slideshow lecture on shock based on
multimedia principles resulted in improvements on an immediate transfer test (d = 0.76) and a delayed
transfer test (d = 1.17).
The principles presented in this chapter are intended to apply to a range of instructional scenarios
ranging from textbooks to face-to-face slide show presentations to computer-based lessons to
interactive games and simulations to online courses to learning in immersive virtual reality. Within a
classroom, these principles apply to the design of classroom printed materials, computer-based
exercises and simulations, and face-to-face instruction including slideshow presentations.
For example, suppose you wished to apply research-based multimedia design principles to improve a
short, narrated slideshow on how a solar cell works for presentation in an environmental science class.
• First, in deference to the coherence principle, you might decide to prune interesting but
irrelevant material that you had downloaded from the Internet, including photos of a solar cell
installation in a desert in southern California and a short video clip you found in which Al Gore
envisions the coming environmental disaster. In short, you work to weed out images and words
that are not essential to explaining how a solar cell works, including eliminating entire slides or
parts of slides.
• Based on the pre-training principle you begin with a slide that that depicts the key elements
(such as the “positive side” and “negative side” of the solar cell) along with a verbal label next to
each key element, perhaps connected with a line.
• In deference to the signaling principle, you place a heading at the top of each of the remaining
10 slides, which succinctly describes the step in the process being depicted. In addition, you use
arrows to show the movement of electrons within the solar cell, corresponding to the
description in the narration.
• In line with the segmenting principle, you break the illustration on each slide into segments that
you progressively add in sync with your narration.
• In line with the modality principle, you mainly use narration to convey the verbal material.
• Corresponding to the redundancy principle, you present only short, printed phrases on the
slides, rather than captions that duplicate what you are saying. In this case, the strictest
152
interpretation modality principle and the redundancy principle can be modified by including a
minimal number of words on the screen—mainly, to help highlight the main points and to
concretize technical or unfamiliar terms.
• In line with the spatial contiguity principle, you place the short text phrases describing the action
next to the corresponding part of the graphic (such as putting “Electrons move across the
barrier” next to an arrow from one side of the barrier to another) rather than at the bottom of
the graphic.
• In line with the personalization principle, you use first- and second- person style (such as saying,
“Let’s see how a solar cell works.” rather than “This lesson tells how a solar cell works.”).
• Consistent with the embodiment principle, you stand next to the slide as you talk, pointing out
what you are talking about and maintaining a smile and eye contact with the audience.
• Based on temporal contiguity, you are careful to choreograph your speech so that it
corresponds to what you are pointing to in the graphic.
• In line with the voice principle, you practice to make sure you use a pleasant, flowing voice that
exudes positive affect, confidence, and warmth.
• In line with the image principle, you remove a distracting logo from each slide which shows Al
Gore’s face looking down along with the slogan: “LIVE GREEN.”
• To bolster the generative activity principle, you ask students to write a one-sentence summary
of each slide in their own words.
All in all, this example demonstrates how to accomplish your goal of removing material that primes
extraneous processing, helping learners understand a complex system by using techniques such as
segmenting and pre-training, and fostering deeper processing by creating a feeling of social
conversation and priming generative learning activities. I will consider this chapter to be a success to the
extent that instructors and instructional designers are able to improve student learning by adapting the
principles presented in this chapter and other evidence-based principles (see Clark & Mayer, 2016;
Mayer & Fiorella, in press-a).
Author Note
Preparation of this chapter was supported by a grant from the Office of Naval Research.
This chapter is an updated version of Mayer (2014) and is based on Mayer (2021).
References
Adesope, O. O., & Nesbit, J. C. (2012). Verbal redundancy in multimedia learning environments: A meta-
analysis. Journal of Educational Psychology, 104(1), 250-263. https://doi.org/10.1037/a0026147
Ayres, P., & Sweller, J. (in press). The split-attention principle in multimedia learning. In R. E. Mayer & L.
Fiorella (Eds.), The Cambridge handbook of multimedia learning (3rd ed). Cambridge University
Press.
Baddeley, A. D. (1999). Human memory. Allyn & Bacon.
Castro-Alonso, J. C., & Sweller, J. (in press). The modality principle in multimedia learning. In R. E. Mayer
& L. Fiorella (Eds.), The Cambridge handbook of multimedia learning (3rd ed). Cambridge
University Press.
153
Clark, R. C. (in press). Multimedia-learning in e-courses. In R. E. Mayer & L. Fiorella (Eds.), The Cambridge
handbook of multimedia learning (3rd ed). Cambridge University Press.
Clark, R. C., & Mayer, R. E. (2016). e-Learning and the science of instruction (4th ed). Wiley.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Erlbaum.
Craig, S. D., & Schroeder, N. L. (2017). Reconsidering the voice effect when learning from a virtual
human. Computers & Education, 114, 193-205. https://doi.org/10.1016/j.compedu.2017.07.003
Craig, S. D., & Schroeder, N. L. (2018). Text-to-speech software and learning: Investigating the relevancy
of the voice effect. Journal of Educational Computing Research, 57(6), 1534-1548.
Davis, R. O. (2018). The impact of pedagogical agent gesturing in multimedia learning environments: A
meta-analysis. Educational Psychology Review, 24, 193-209.
http://dx.doi.org/10.1016/j.edurev.2018.05.002
Fiorella, L., & Mayer, R. E. (2016). Effects of observing the instructor draw diagrams on learning from
multimedia messages. Journal of Educational Psychology, 108(4), 528-546.
Fiorella, L., & Mayer, R. E. (in press-a). Principles for reducing extraneous processing in multimedia
learning: Coherence, signaling, redundancy, spatial contiguity, and temporal contiguity
principles. In R. E. Mayer & L. Fiorella (Eds.), The Cambridge handbook of multimedia learning
(3rd ed). Cambridge University Press.
Fiorella, L., & Mayer, R. E. (in press-b). Principles based on social cues in multimedia learning:
Personalization, voice, embodiment, and image principles. In R. E. Mayer & L. Fiorella (Eds.), The
Cambridge handbook of multimedia learning (3rd ed). Cambridge University Press.
Fiorella, L., & Mayer, R. E. (in press-c). The generative activity principle in multimedia learning. In R. E.
Mayer & L. Fiorella (Eds.), The Cambridge handbook of multimedia learning (3rd ed). Cambridge
University Press.
Fiorella, L., van Gog, T., Hoogerheide, V., & Mayer, R. E. (2017). It’s all a matter of perspective: Viewing
first-person video modeling examples promotes learning of an assembly task. Journal of
Educational Psychology, 109(5), 653-665. https://doi.org/10.1037/edu0000161
Ginns, P. (2005). Meta-analysis of the modality effect. Learning and Instruction, 15(4), 313-332.
Ginns, P. (2006). Integrating information: A meta-analysis of spatial contiguity and temporal contiguity
effects. Learning and Instruction, 16(6), 511-525.
Ginns, P., Martin, A.J., & Marsh, H.M. (2013). Designing instructional text in a conversational style: A
meta-analysis. Educational Psychology Review, 25, 445-472. https://doi.org/10.1007/s10648-
013-9228-0
Hattie, J. (2009). Visible learning. Routledge.
Issa, N., Mayer, R. E., Schuller, M., Wang, E., Shapiro, M. B., & DaRosa, D. A. (2013). Teaching for
understanding in medical classrooms using multimedia design principles. Medical Education, 47,
388-396. https://doi.org/10.1111/medu.12127
154
Jiang, D., & Sweller, J. (in press). The transient information principle in multimedia learning. In R. E.
University Press.
Kalyuga, S., & Sweller, J. (in press). The redundancy principle in multimedia learning. In R. E. Mayer & L.
Press.
Mautone, P. D. & Mayer, R. E. (2001). Signaling as a cognitive guide in multimedia learning. Journal of
Educational Psychology, 93(2), 377-389. https://doi.org/10.1037/0022-0663.93.2.377
Mayer, R. E. (2014). Research-based principles for designing multimedia instruction. In V. A. Benassi, C.
E. Overson, & C. M. Hakala (Eds.), Applying the science of learning in education: Infusing
psychological science into the curriculum. (59-70). Society for the Teaching of Psychology,
Mayer, R. E. (2021). Multimedia learning (3rd ed). Cambridge University Press.
Mayer, R. E. (in press-a). Multimedia principle. In In R. E. Mayer & L. Fiorella (Eds.), The Cambridge
handbook of multimedia learning (3rd ed). Cambridge University Press.
Mayer, R. E. (in press-b). Cognitive theory of multimedia learning. In R. E. Mayer & L. Fiorella (Eds.), The
Cambridge handbook of multimedia learning (3rd ed). Cambridge University Press.
Mayer, R. E., & Anderson, R. B. (1991). Animations need narrations: An experimental test of a dual-
coding hypothesis. Journal of Educational Psychology, 83(4), 484-490.
Mayer, R. E., & Chandler, P. (2001). When learning is just a click away: Does simple user interaction
foster deeper understanding of multimedia messages? Journal of Educational Psychology, 93(2),
390-397. https://doi.org/10.1037/0022-0663.93.2.390
Mayer, R. E., & DaPra, C. S. (2012). An embodiment effect in computer-based learning with animated
pedagogical agent. Journal of Experimental Psychology: Applied, 18(3), 239-252.
Mayer, R. E., Dow, G., & Mayer, R. E. (2003). Multimedia learning in an interactive self-explaining
environment: What works in the design of agent-based microworlds? Journal of Educational
Psychology, 95(4), 806-813. https://doi.org/10.1037/0022-0663.95.4.806
Mayer, R. E., Fennell, S., Farmer, L., & Campbell, J. (2004). A personalization effect in multimedia
learning: Students learn better when words are in conversational style rather than formal style.
Journal of Educational Psychology, 96(2), 389-395. https://doi.org/10.1037/0022-0663.96.2.389
Mayer, R. E. & L. Fiorella (Eds.). (in press-a). The Cambridge handbook of multimedia learning (3rd ed).
Cambridge University Press.
Mayer, R. E., & Fiorella, L. (in press-b). Principles for managing essential processing in multimedia
learning: Segmenting, pre-training, and modality principles. In R. E. Mayer & L. Fiorella (Eds.),
The Cambridge handbook of multimedia learning (3rd ed). Cambridge University Press.
Mayer, R. E., Griffith, E., Jurkowitz, I. T. N., & Rothman, D. (2008). Increased interestingness of
extraneous details in multimedia science presentation leads to decreased learning. Journal of
Experimental Psychology: Applied, 14(4), 329-339. https://doi.org/10.1037/a0013835
155
Mayer, R. E., Heiser, H., & Lonn, S. (2001). Cognitive constraints on multimedia learning: When
presenting more material results in less understanding. Journal of Educational Psychology, 93(1),
187-198. https://doi.org/10.1037/0022-0663.93.1.187
Mayer, R. E., Mathias, A., & Wetzell, K. (2002). Fostering understanding of multimedia messages through
pre-training: Evidence for a two-stage theory of mental model construction. Journal of
Experimental Psychology: Applied, 8(3), 147-154. https://doi.org/10.1037/1076-898X.8.3.147
Mayer, R. E., Sobko, K., & Mautone, P. D. (2003). Social cues in multimedia learning: Role of speaker's
voice. Journal of Educational Psychology, 95(2), 419-425. https://doi.org/10.1037/0022-
0663.95.2.419
Moreno, R., & Mayer, R. E. (1999). Cognitive principles of multimedia learning: The role of modality and
contiguity. Journal of Educational Psychology, 91(2), 358-368. https://doi.org/10.1037/0022-
0663.91.2.358
Moreno, R., & Mayer, R. E. (2002). Learning science in virtual reality multimedia environments: Role of
methods and media. Journal of Educational Psychology, 94(3), 598-610.
Paivio, A. (1986). Mental representations: A dual coding approach. Oxford University Press.
Paas, F., & Sweller, J. (in press). Implications of cognitive load theory for multimedia learning. In R. E.
University Press.
Reinwein, J. (2012). Does the modality effect exist? And is so, which modality effect? Journal of
Psycholinguistic Research, 41, 1-32. https://doi.org/10.1007/s10936-011-9180-4
Rey, G. D. (2012). A review and a meta-analysis of the seductive detail effect. Educational Research
Review, 7(3), 216-237. https://doi.org/10.1016/j.edurev.2012.05.003
Richter, J., Scheiter, K., & Eitel, A. (2016). Signaling text-picture relations in multimedia learning: A
comprehensive meta-analysis. Educational Psychology Review, 17, 19-36.
Schneider, S., Beege, M., Nebel, S., & Rey, G. D. (2018). A meta-analysis of how signalling affects learning
with media. Educational Psychology Review, 23, 1-14.
Schroeder, N. L., & Adesope, O. O. (2014) A systematic review of pedagogical agents’ persona,
motivation, and cognitive load implications for learners. Journal of Research on Technology in
Education, 46(3), 229-251. https://doi.org/10.1080/15391523.2014.888265
Schroeder, N. L., & Cenkci, A. T. (2018). Spatial contiguity and spatial split-attention effects in
multimedia learning environments: A meta-analysis. Educational Psychology Review, 30, 679-
Sundararajan, N., & Adesope, O. (2020). Keep it coherent: A meta-analysis of the seductive details
effect. Educational Psychology Review, 32, 707-734. https://doi.org/10.1007/s10648-020-09522-
4
156
Van Gog, T. (in press). The signaling (or cueing) principle in multimedia learning. In R. E. Mayer & L.
Press.
Wittrock, M. C. (1989). Generative processes of comprehension. Educational Psychologist, 24(4), 345-
376. https://doi.org/10.1207/s15326985ep2404_2
157
In Their Own Words Learning by Teaching

Logan Fiorella
You to Know About Why and How University of Georgia
Editors
Lauren L. Kordonowy
Victor A. Benassi
Overview
The best way to learn something new is to teach it to someone else. You have probably heard this
proclamation before, and perhaps you have made it yourself. The idea that teaching others enhances
one’s own learning is far from new. In Letters to Lucilius, the first-century Stoic philosopher Seneca
wrote, “while we teach, we learn.” Over two thousand years later, many prevalent educational practices
incorporate elements of learning by teaching (Duran & Topping, 2017), including peer tutoring programs
(Leung, 2015; Roscoe & Chi, 2007), collaborative learning (Nokes-Malach et al., 2015), and small group
discussions (van Blankenstein et al. 2011; Webb, 1982). In this chapter, I take a closer look at the
evidence for learning by teaching, focusing on the benefits and boundary conditions associated with
specific elements of teaching. Then I provide recommendations for how to implement teaching activities
to enhance student understanding.
The term ‘learning by teaching’ suggests that explaining a concept to someone else will improve one’s
own understanding of the concept. However, I suspect this is not the full picture of what most people
mean when they say that we learn best by teaching. Rather, they are often referring to how multiple
elements of the teaching process benefit learning, such as by motivating students to process the
learning material more deeply and to engage in meaningful interactions with others. In practice,
learning by teaching often translates to, ‘when I am expected to teach others, I need to make sure that I
understand it myself,’ or ‘that my explanation is coherent,’ or ‘that I can respond to students’
questions.’ As I will describe in this chapter, there are three key elements of teaching—preparing,
explaining, and interacting—each of which can provide unique cognitive, metacognitive, and
motivational benefits.
Theoretical Framework
Learning by teaching can be classified as a generative learning activity because it encourages learners to
actively make sense of what they are learning. According to generative learning theory (Fiorella &
Mayer, 2015, 2016), making sense involves selecting key information from the learning material,
organizing this information into a coherent structure in working memory, and integrating it with existing
knowledge from long-term memory. Organizing and integrating are generative processes because
learners must generate inferences to build appropriate relationships among the key ideas of the
material. Generative processes are primarily driven by one’s existing domain knowledge, strategic
knowledge, and motivation to make sense of the learning material.
When learners are prompted with the prospect of teaching others, it may motivate them to engage in
generative processing while preparing to teach, generating an explanation, and interacting with others.
However, learners must have sufficient domain and strategic knowledge to make sense of the learning
158
material, generate a coherent and comprehensive explanation, and effectively respond to questions. In
other words, merely prompting learners to engage in a generative learning activity like teaching does
not guarantee students will spontaneously engage in generative processing. Learners need instructional
support to generate high-quality explanations for others that result in deep understanding (Roscoe &
Chi, 2007). This chapter will identify learner and instructional characteristics that are most important for
maximizing the benefits of learning by teaching.
Generative processing is closely connected to what Chi and Wylie (2014) refer to as the constructive
mode of engagement in their ICAP (Interactive, Constructive, Active, Passive) framework, wherein
teaching others may encourage students to monitor their understanding, generate inferences, and
repair gaps or misconceptions in their existing knowledge (Chi & Boucher, this volume). The ICAP further
classifies the interactive mode as the highest mode of engagement, in which learners actively engage in
dialogue to ask questions or to clarify and elaborate on each other’s ideas. Thus, when learning by
teaching fosters productive interactions among students, it has the potential to enhance learning
beyond other common learning activities that target lower levels of engagement, such as summarizing
(active mode) or self-explaining (constructive mode).
Description and Discussion

Much of the prior research on learning by teaching involves evaluating peer tutoring programs, such as
cross-age tutoring (in which more advanced students teach less advanced students) and reciprocal
teaching (in which students take turns teaching each other) compared to more conventional forms of
instruction (Duran & Topping, 2017; Leung, 2015; Roscoe & Chi, 2007). Although valuable, these studies
are limited in their ability to isolate the specific ‘active ingredients’ of teaching that are responsible for
learning. According to Bargh and Schul (1980), learning by teaching consists of three distinct stages,
which may uniquely contribute to learning: preparing to teach, generating an explanation for others,
and interacting with others. In the following sections, I focus on research that isolates the unique and
additive benefits at each stage of the teaching process.
Benefits and Boundaries of Preparing to Teach
Learners typically approach learning tasks with the goal of supporting their own learning, such as when
studying for an upcoming test. Do students benefit more from studying with the expectation to teach
others? One hypothesis is that the social responsibility of teaching others may motivate learners to
study in a more effortful and effective way. An early study by Benware and Deci (1984) found that
students who expected to teach the contents of an article on brain functioning were more intrinsically
motivated to learn and achieved better conceptual understanding of the material than students who
expected to take a test. They concluded that preparing to teach promotes a more active ‘motivational
set’ than the way in which students typically study. In other words, the prospect of teaching others can
shift students’ mindset toward actively making sense of the learning material.
More recent research has helped to uncover the specific cognitive and metacognitive mechanisms
underlying preparing to teach. In a study by Nestojko and colleagues (2014), college students studied
text passages with the expectation of either teaching or taking a test. Then both groups of students
completed a free recall test and answered targeted questions about the content of the texts. Students
who expected to teach generated more organized and complete recall of the passages and performed
better on questions targeting the main ideas of the passage than students who expected a test. Similar
findings were reported in a study by Muis and colleagues (2016) involving elementary students in math.
Students who prepared to teach developed more organized representations of math problems, engaged
in more metacognitive processing strategies, and exhibited higher problem-solving achievement than
159
students who prepared normally for a test. Taken together, these studies suggest preparing to teach
may help students better organize the learning material around key ideas, resulting in better learning
outcomes.
Despite the benefits of preparing to teach, research also suggests potential boundary conditions. Some
studies report either no benefits or inconsistent effects of learning by expecting to teach compared to
expecting a test (e.g., Ehly et al., 1987; Hoogerheide et al., 2014; Renkl, 1995). For example, Renkl (1995)
found students did not benefit from preparing to teach probability worked examples, likely because
students reported experiencing high levels of anxiety at the prospect of teaching. Thus, although
teaching can be motivating, under some conditions it can be stressful and potentially interfere with
learning.
Another explanation for the mixed findings is that the expectation of teaching encourages, but does not
guarantee, that students will apply appropriate strategies for making sense of the material. In other
words, higher levels of motivation or effort do not necessarily support learning if they do not lead to
productive generative processing. A study by Fukaya (2013) suggests students have different
metacognitive knowledge about what constitutes a quality explanation. Some students are more
oriented toward preparing elaborative explanations that connect the learning material with their
existing knowledge, whereas others take a summarization approach that does not incorporate their
prior knowledge. In terms of the ICAP framework, preparing to teach may reach the ‘constructive’ mode
of engagement for some, but for others it may only reach the ‘active’ mode. As expected, Fukaya (2013)
only found benefits of preparing to teach among students oriented toward preparing elaborative
explanations.
There is also some inconsistency in the literature about whether preparing to teach supports long-term
learning. In a series of studies by Fiorella and Mayer (2013, 2014), undergraduate students who
prepared to teach a lesson on the Doppler effect performed better on an immediate comprehension
test, but not a delayed comprehension test, compared to students who prepared normally for a test. In
contrast, a recent study by Guerrero and Wiley (2021) found a teaching expectancy effect for immediate
and delayed comprehension. Further research is needed to understand the conditions under which
preparing to teach (without actually teaching) leads to long-term learning. As discussed later, long-term
outcomes may be strongest when students have the opportunity to prepare and generate an
explanation for others.
Overall, prompting learners to prepare to teach offers an easy-to-implement technique that can
motivate learners to engage in generative processing. A recent meta-analysis by Kobayashi (2019a)
found a small-to-medium average effect size (d = .30 to .40) across 28 studies favoring preparing to
teach over preparing for a test. The evidence reviewed in this section suggests this effect may be
moderated by the extent to which learners experience excessive anxiety at the prospect of teaching, the
beliefs learners have about their ability to generate quality explanations, and the quality of strategies
learners use when preparing to teach. Thus, instructors may consider using a relatively low-stakes
teaching expectancy prompt and instructing students on the elements of quality explanations and
strategies for preparing explanations for others.
Benefits and Boundaries of Explaining to Others
The prospect of teaching is a simple way to leverage students’ expectations to support learning. Are
there added benefits of actually teaching the material to someone else? An early study by Annis (1983)
found that students better understood the contents of a history text when they prepared to teach and
then actually taught a peer compared to when they only prepared to teach. This study was somewhat
160
limited, however, because students who taught also interacted with their peers, such as by responding
to questions. Thus, the study cannot isolate the effects solely due to explaining to others.
To overcome this limitation, more recent work has isolated the effects of asking students to generate
non-interactive explanations for fictitious peers, such as by recording a video lecture or composing a
written explanation. In a series of experiments by Fiorella and Mayer (2013, 2014), college students
learned about the Doppler effect with the expectation of teaching or taking a test. Of those expecting to
teach, some students actually taught by creating a video lecture intended to teach a fictitious student
who does not know about the learning material. On immediate comprehension tests, students who
prepared to teach performed similarly, regardless of whether they actually taught on video. However,
students who actually taught performed best on a delayed comprehension test. There was also evidence
that the added benefit of teaching was strongest on a delayed test when students prepared to teach
(rather than prepared for a test). Hoogerheide and colleagues (2014) reported a similar pattern of
results in which actually teaching on video led to better understanding than only preparing to teach.
Why does generating an explanation for others enhance learning? One explanation is that it involves
retrieving the learning material from memory, which strengthens the memory of that information and
makes it more accessible over time. To test this idea, Koh and colleagues (2018), using the same
materials from Fiorella and Mayer (2013, 2014), asked students to teach the Doppler effect lesson on
video either from memory or with teaching notes available to them. Other students did not teach and
were asked to only recall the information from memory. Results indicated that students asked to
retrieve the information from memory (either via teaching or via a recall test) outperformed students on
a delayed comprehension test who taught while having access to teaching notes. This finding suggests
learning by teaching is most effective when used as a retrieval activity. It is important to note, however,
that the benefits of explaining to others are not solely explained by retrieval processes. Other studies
have found that explaining as a retrieval activity is more effective than merely taking a free-recall test
(Jacob et al., 2020; Lachner et al., 2021), likely because explaining better primes students to organize
and integrate the material with their existing knowledge (Fiorella & Mayer, 2016). Furthermore,
teaching involves adapting an explanation for one’s audience, which may also play a unique and
important role in supporting learning (Rittle-Johnson et al., 2008).
Another consideration is whether students should generate written or oral explanations when teaching
others. Hoogerheide and colleagues (2016) found that orally teaching syllogistic reasoning problems on
video was more effective than restudying the learning material, but teaching via written explanations
was not more effective than restudying. There was not a significant difference in performance between
the oral and written explanation groups, so this study could not draw strong conclusions about the role
of explanation modality. Lachner and colleagues (2018) took a closer look at differences in the quality of
students’ written and oral explanations and their relationship with different learning outcomes. Written
explanations tended to be more organized and contributed to better conceptual understanding; in
contrast, oral explanations tended to be more elaborative and contributed to better transfer test
performance. Interestingly, the studies by Hoogerheide and colleagues (2016) and Lachner and
colleagues (2018) both found that students’ oral explanations contained more audience-directed
speech, in which students used first- or second-person pronouns like ‘I’ and ‘you’. This finding suggests
the benefits of orally explaining may be in part due to greater feelings of social presence, or the sense
that one is in communication with an audience.
A recent study by Jacob and colleagues (2020) suggests the benefits of orally explaining may also
depend on the nature of the learning materials: orally explaining was more effective when students
learned from texts higher in complexity but not lower in complexity. Further, the benefits of orally
explaining complex texts were mediated by higher levels of audience-directed speech in students’
161
explanations and more comprehensive explanations. Overall, oral explanations appear to create higher
levels of social presence and make it easier for students to elaborate on the learning material,
facilitating sense making of more complex learning materials.
The research described above suggests that explaining to others is most effective when students
prepare with the expectation of teaching, explain as a retrieval activity, and generate oral explanations.
However, without additional support, many students may still struggle to generate high-quality
explanations that support deep understanding. According to Roscoe (Roscoe, 2014; Roscoe & Chi, 2007,
2008), students exhibit a ‘knowledge-telling bias’ when prompted to explain to others, in which they
simply restate or summarize the learning material while making minimal inferences. The goal is for
students to engage in ‘knowledge-building’ – similar to generative or constructive processing – in which
they actively monitor their understanding and use their existing knowledge to elaborate on the learning
material.
One simple way to foster knowledge building in learner-generated explanations is to incorporate the use
of instructional visualizations (e.g., Ainsworth & Loizou, 2003; Butcher, 2006; Cromley et al., 2010). In a
recent study, Fiorella and Kuhlmann (2020) explored whether creating drawings helped learners
generate higher-quality explanations during learning by teaching. College students studied a lesson on
the human respiratory system and then recorded a video lecture to a fictitious student. Students either
orally explained (explain-only), created drawings (draw-only), or orally explained while simultaneously
creating drawings (explain-and-draw). A control group of students restudied the learning material
without teaching. Results indicated all teaching conditions significantly outperformed the control group
on a series of learning outcome measures (including a transfer test) administered one week later.
Furthermore, the explain-and-draw group significantly outperformed the explain-only and draw-only
groups, which did not differ from each other. Analyses of students’ explanations indicated that students
who created drawings while explaining generated more elaborative explanations (an indicator of
knowledge building) than students who only explained. This study suggests that drawing facilitates
explaining, which leads to better understanding. An open question is whether learner-generated
drawings and instructor-provided visuals might differentially impact learning by teaching (Fiorella,
2021). Related research suggests learners may benefit from a combination of generated and provided
visuals (Zhang & Fiorella, 2019, 2021).
Overall, generating explanations for others enhances learning beyond only preparing to teach and is
generally more effective than restudying or engaging in retrieval tasks like free recall tests. The review
by Kobayashi (2019a) reports a medium average effect size (d = .5 to .6) across 28 studies when students
both prepare to teach and actually teach others. For explaining to be most effective, learners need
guidance to facilitate knowledge building, such as the use of provided visualizations (e.g., Ainsworth &
Loizou, 2003), focused prompts to explain or draw key elements of the learning material (e.g., Berthold,
Eysink, & Renkl, 2009; Fiorella & Mayer, 2017), and explicit training and practice on how to generate
quality explanations (e.g., McNamara, 2004).
Benefits and Boundaries of Interacting with Others
According to the ICAP framework (Chi & Wylie, 2014), the highest mode of cognitive engagement is the
interactive mode, in which learners work together to co-construct knowledge of the learning material.
Thus, learning by teaching should have the highest potential to support deep learning when it involves
preparing to teach, generating an explanation, and engaging in quality interactions with one’s peers,
such as asking and responding to questions.
In a recent review, Kobayashi (2019b) distinguished two approaches for examining the effects of
interactivity in learning by teaching. The first approach is to compare learning by teaching a live
162
audience (higher interactivity) to learning by self-explaining (lower interactivity). Thus, this approach
allows researchers to isolate the effects of explaining to an audience. Coleman and colleagues (1997)
found that students learned better from preparing for and actually explaining the material to others
compared to preparing for and explaining the material to themselves. Similarly, Rittle-Johnson and
colleagues (2008) found young children benefited more from explaining math problems to their mothers
than generating self-explanations. In contrast, some studies have found either no differences or an
advantage of self-explaining over teaching (Bargh & Schul, 1980; Pi et al., 2021; Roscoe & Chi, 2008). Yet
in these studies, participants were not informed prior to the study period that they would later explain
the material to others. Thus, the social interactivity associated with both preparing and actually
explaining to a live audience may offer unique benefits beyond self-explaining, which itself is an
effective learning strategy (Bisra et al., 2018).
Recent studies suggest another way in which the benefits of explaining to others may depend on
interactivity. Lachner and colleagues (2021) found that students learned better from writing
explanations for oneself than for teaching others. As discussed above, written explanations may not be
the most suitable implementation of learning by teaching because they do not incorporate high degrees
of interactivity or feelings of social presence. A follow-up study by Jacob and colleagues (2021) found
that trying to increase the level of social presence in written explanations (by typing explanations into a
chat box) was not sufficient to boost learning. Thus, explaining orally to an audience may be an
important component of interactivity, although further research is needed to test other ways in which
written explanations might create higher levels of social presence.
The second approach for studying the effects of interactivity is to compare teaching with or without the
opportunity to subsequently engage in dialogue with others, such as asking and responding to
questions. Thus, this approach allows researchers to isolate the effects of interactive dialogue during
teaching. Surprisingly few studies have made this direct comparison (Kobayashi, 2019b). Roscoe and Chi
(2008) found that college students learned better from interactive teaching, which involved answering
questions from a tutee, than from non-interactive teaching, which involved recording a video lecture.
Similarly, a recent study by Kobayashi (2021) found that teaching with a subsequent question-and-
answer period benefited learning beyond only teaching without interacting. Thus, there is some
empirical evidence that opportunities for student interaction can provide additive benefits to learning
by teaching.
Why might interacting with others boost learning? One explanation is that asking and responding to
questions encourages tutors to engage in knowledge building, in which they reflect on their
understanding and attempt to clarify or revise their understanding of the learning material. In the study
by Roscoe and Chi (2008), tutors engaged in more knowledge building when tutees asked questions that
required generating inferences. Similarly, Roscoe (2014) found that the number of conceptual questions
posed by tutees was predictive of knowledge building among tutors which, in turn, supported deeper
understanding. However, analyses of tutor-tutee interactions reveal that tutees rarely ask deep,
conceptual questions that provoke knowledge building in tutors. A recent study by Wang and colleagues
(2021) further indicates that merely having the opportunity to interact with a peer does not guarantee
that the interaction will lead to knowledge building. In that study, imagined teaching, in which students
only imagined the process of teaching, was actually more effective than interactive teaching.
Taken together, opportunities to ask and respond to questions can benefit learning, but students need
support to engage in effective questioning activities during learning. One remedy may be to take
advantage of the preparation phase of teaching, as preparing to teach may help students better ask,
anticipate, and respond to questions from peers. Related research suggests that collaborative learning is
most effective when students have the opportunity to first prepare individually (Mende et al., 2021).
163
Other potential solutions include providing explicit training in explaining and questioning strategies or
providing prompts or scaffolds to structure peer interactions (e.g., Fuchs et al., 1994; King, 1994; King et
al., 1998), so that students do not have to fully generate explanations and questions themselves. In
short, students need support to pose provocative questions that lead to reflective, elaborative, and
accurate explanations.
Recommendations
The available research evidence generally supports our intuitions – teaching others can indeed be a
highly effective way to learn. However, simply prompting students to teach others will not necessarily
result in better understanding. The research reviewed in this chapter provides a set of basic guidelines
for when and how to implement learning by teaching activities most effectively.
Learner Characteristics
Instructors should first consider individual differences in learners’ levels of background domain
knowledge, strategic knowledge, and motivation. First, learners need sufficient domain knowledge to
generate appropriate inferences from the learning material (Roscoe, 2014). In other words, the learning
materials need to be ‘knowledge-appropriate.’ Learners without the requisite prior knowledge may
need pre-training before engaging in learning by teaching. Second, learners need sufficient strategic
knowledge of what constitutes quality explanations and interactions with others. Although some
students may already be oriented toward generating elaborative explanations, many learners approach
the tasks of explaining and interacting with others passively and rely on knowledge-telling. Instructors
can help students develop strategic knowledge by using examples to explicitly model effective
explaining and questioning in their particular disciplines. Third, learners need sufficient motivation to
engage in learning by teaching. Although the prospect of teaching others can be motivating, in some
situations it may be stressful. Thus, learners may need the opportunity to prepare and practice in a
supportive environment to experience the benefits of learning by teaching. Instructors may also want to
use learning by teaching primarily as low-stakes learning activities and formative assessments.
Learning Materials
Instructors should also consider the nature of the learning materials when implementing learning by
teaching. Learning by teaching is most suitable for conceptual learning materials, such as understanding
how a system works or underlying problem-solving principles. It may not be as effective for
remembering relatively isolated factual information, such as foreign language vocabulary words. Prior
research has mostly involved explaining complex concepts in math and science, such as how the Doppler
effect works, how electric circuits work, how the human retina works, or how to apply principles of
probability (e.g., Fiorella & Mayer, 2013; Hoogerheide et al., 2019a; Roscoe, 2014). The format of these
learning materials has included text passages, visualizations, and worked examples. Although prior
research has focused primarily on mathematics and science concepts, instructors could likely adapt
learning by teaching activities to fit conceptual materials across other disciplines, such as the humanities
(e.g., Pi et al., 2021).
Applications of Learning by Teaching
There are many potential applications of learning by teaching that extend beyond conventional
classroom activities. For example, students may be asked to create brief instructional videos as a
homework assignment, rather than typical homework assignments, such as reading or answering
practice questions. Hoogerheide and colleagues (2019b) found that recording an instructional video as
homework was more effective and enjoyable for students than conventional homework activities.
164
Similarly, students can learn by teaching within remote classrooms by either pre-recording videos or
engaging in live video interactions with other students. Virtual tools, such as screen sharing or the ability
to construct visualizations, may also help facilitate knowledge building. Other technology-based
applications include the use of interactive computer-based avatars (or ‘pedagogical agents,’ e.g., Biswas
et al., 2005), virtual reality (Klingenberg et al., 2020), and educational games (Fiorella et al., 2019).
Research suggests that students can benefit from teaching and interacting with intelligent virtual agents
or by explaining content presented in a virtual classroom, as long as these environments do not impose
excessive extraneous cognitive load (e.g., Fiorella et al., 2019; Ashman & Sweller, this volume). Overall,
technology-based tools offer promising ways to provide adaptive guidance to support quality student
explanations and deeper understanding.
Implementation
Instructors may choose to incorporate one or more of the following key elements of learning by
teaching into course activities: preparing, explaining, and interacting. First, preparing to teach offers a
simple way to shift students’ expectations and orient them toward making sense of the learning
material. For example, students could be instructed to read and take notes on a textbook chapter as if
they were preparing to teach another student who has not read the chapter. Second, actually
generating an explanation can encourage students to monitor their understanding and generate
inferences. Explaining may be most effective when students prepare with the expectation of teaching,
explain orally to a live audience, and explain as a retrieval activity. For example, at the beginning of a
lesson, instructors can inform students that they will later teach the key concepts to a peer. After the
lesson, instructors can provide students time to prepare their explanations before actually explaining
the concepts to a peer, without access to their notes or the learning material. Third, providing students
with the opportunity to interact with each other can further encourage students to monitor their
understanding, clarify, and elaborate on the learning material. Instructors can encourage students to ask
and respond to questions that require inference generation or apply knowledge to a new situation.
Instructors may also incorporate reciprocal teaching (e.g., Palincsar & Brown, 1984), in which students
take turns explaining concepts to each other and asking for clarification or elaboration questions.
At each stage of the teaching process, students will likely need instructional support. The literature on
generative learning strategies suggests three primary modes of guidance: modeling, scaffolding, and
feedback (Fiorella & Zhang, 2018). At the preparing stage, students need to know what constitutes a
quality explanation and how to translate the provided explanations into a coherent explanation for
others. Instructors can model the process of preparing to teach, provide scaffolds such as outlines or
visualizations to facilitate organization, and provide feedback on the quality of students’ lesson plans. At
the explaining phase, students need support to generate explanations that involve knowledge building.
Instructors can use examples to model the (meta)cognitive processes of monitoring and inference
generation, provide students with focused prompts to explain key relationships or to create
visualizations, or provide feedback on the coherence, accuracy, and completeness of students’
explanations. Finally, at the interacting phase, students need support on how to generate conceptual
questions that encourage knowledge building. Instructors can provide examples of inference questions
to pose, provide partial scripts to guide students toward quality interactions (e.g., ‘Can you provide a
concrete example of X?’ ‘How would X apply in Y situation?’) or provide feedback on how to shift
questions targeting knowledge telling more toward knowledge building. Overall, when students receive
adequate support at each stage of the process, learning by teaching can serve as a highly effective
strategy for supporting student understanding.
165
Author Note
Correspondence for this chapter should be directed to Logan Fiorella at [email protected]. Preparation
of this chapter was supported by a grant from the National Science Foundation (1955348) and a
National Academy of Education/Spencer Postdoctoral Fellowship.
References
Ainsworth, S., & Loizou, A. T. (2003). The effects of self-explaining when learning with text or diagrams.
Cognitive Science, 27(4), 669-681. https://doi.org/10.1207/s15516709cog2704_5
Annis, L. F. (1983). The processes and effects of peer tutoring. Human Learning: Journal of Practical
Research & Applications, 2(1), 39–47.
Bargh, J. A., & Schul, Y. (1980). On the cognitive benefits of teaching. Journal of Educational
Psychology, 72(5), 593-604. https://doi.org/10.1037/0022-0663.72.5.593
Benware, C. A., & Deci, E. L. (1984). Quality of learning with an active versus passive motivational set.
American Educational Research Journal, 21, 755-765.
Berthold, K., Eysink, T. H., & Renkl, A. (2009). Assisting self-explanation prompts are more effective than
open prompts when learning with multiple representations. Instructional Science, 37, 345-363.
https://doi.org/10.1007/s11251-008-9051-z
Bisra, K., Liu, Q., Nesbit, J. C., Salimi, F., & Winne, P. H. (2018). Inducing self-explanation: A meta-
analysis. Educational Psychology Review, 30, 703-725. https://doi.org/10.1007/s10648-018-
9434-x
Biswas, G., Leelawong, K., Schwartz, D., Vye, N., & The Teachable Agents Group at Vanderbilt. (2005).
Learning by teaching: A new agent paradigm for educational software. Applied Artificial
Intelligence, 19(3-4), 363-392. https://doi.org/10.1080/08839510590910200
Butcher, K. R. (2006). Learning from text with diagrams: Promoting mental model development and
inference generation. Journal of Educational Psychology, 98(1), 182-197.
Chi, M. T., & Wylie, R. (2014). The ICAP framework: Linking cognitive engagement to active learning
outcomes. Educational Psychologist, 49(4), 219-243.
Coleman, E. B., Brown, A. L., & Rivkin, I. D. (1997). The effect of instructional explanations on learning
from scientific texts. The Journal of the Learning Sciences, 6(4), 347-365.
Cromley, J. G., Snyder-Hogan, L. E., & Luciw-Dubas, U. A. (2010). Cognitive activities in complex science
text and diagrams. Contemporary Educational Psychology, 35(1), 59-74.
https://doi.org/10.1016/j.cedpsych.2009.10.002
Duran, D., & Topping, K. (2017). Learning by teaching: Evidence-based strategies to enhance learning in
the classroom. Routledge.
166
Ehly, S., Keith, T. Z., & Bratton, B. (1987). The benefits of tutoring: An exploration of expectancy and
outcomes. Contemporary Educational Psychology, 12(2), 131-134.
https://doi.org/10.1016/S0361-476X(87)80046-2
Fiorella, L. (2021). Fostering knowledge building in learning by teaching: A test of the drawing‐facilitates‐
explaining hypothesis. Applied Cognitive Psychology, 35(2), 548-558.
Fiorella, L., & Kuhlmann, S. (2020). Creating drawings enhances learning by teaching. Journal of
Educational Psychology, 112(4), 811-822. https://doi.org/10.1037/edu0000392
Fiorella, L., Kuhlmann, S., & Vogel-Walcutt, J. J. (2019). Effects of playing an educational math game that
incorporates learning by teaching. Journal of Educational Computing Research, 57(6), 1495-
Fiorella, L., & Mayer, R. E. (2013). The relative benefits of learning by teaching and teaching expectancy.
Contemporary Educational Psychology, 38(4), 281-288.
Fiorella, L., & Mayer, R. E. (2014). Role of expectations and explanations in learning by teaching.
Fiorella, L., & Mayer, R. E. (2016). Eight ways to promote generative learning. Educational Psychology
Review, 28(4), 717-741. https://doi.org/10.1007/s10648-015-9348-9
Fiorella, L., & Mayer, R. E. (2017). Spontaneous spatial strategy use in learning from scientific
text. Contemporary Educational Psychology, 49, 66-79.
Fiorella, L., & Zhang, Q. (2018). Drawing boundary conditions for learning by drawing. Educational
Psychology Review, 30(3), 1115-1137. https://doi.org/10.1007/s10648-018-9444-8
Fuchs, L., Fuchs, D., Bentz, J., Phillips, N., & Hamlett, C. (1994). The nature of students’ interactions
during peer tutoring with and without prior training and experience. American Educational
Research Journal, 31(1), 75-103. https://doi.org/10.3102/00028312031001075
Fukaya, T. (2013). Does metacognitive knowledge about explanation moderate the effect of explanation
expectancy? Psychologia, 56(4), 246-258. https://doi.org/10.2117/psysoc.2013.246
Guerrero, T. A., & Wiley, J. (2021). Expecting to teach affects learning during study of expository
texts. Journal of Educational Psychology, 113(7), 1281–
1303. https://doi.org/10.1037/edu0000657
Hoogerheide, V., Deijkers, L., Loyens, S. M., Heijltjes, A., & van Gog, T. (2016). Gaining from explaining:
Learning improves from explaining to fictitious others on video, not from writing to
them. Contemporary Educational Psychology, 44-45, 95-106.
Hoogerheide, V., Loyens, S. M., & van Gog, T. (2014). Effects of creating video-based modeling examples
on learning and transfer. Learning and Instruction, 33, 108-119.
167
Hoogerheide, V., Renkl, A., Fiorella, L., Paas, F., & Van Gog, T. (2019a). Enhancing example-based
learning: Teaching on video increases arousal and improves problem-solving performance.
Journal of Educational Psychology, 111(1), 45-56. https://doi.org/10.1037/edu0000272
Hooherheide, V., Visee, J., Lachner, A., & van Gog, T. (2019b). Generating an instructional video as
homework activity is both effective and enjoyable. Learning and Instruction, 64, 101226.
https://doi.org/10.1016/j.learninstruc.2019.101226
Jacob, L., Lachner, A., & Scheiter, K. (2020). Learning by explaining orally or in written form? Text
complexity matters. Learning and Instruction, 68, 101344.
Jacob, L., Lachner, A., & Scheiter, K. (2021). Does increasing social presence enhance the effectiveness of
writing explanations? PloS one, 16(4), e0250406.
https://doi.org/10.1371%2Fjournal.pone.0250406
King, A. (1994). Guiding knowledge construction in the classroom: Effects of teaching children how to
question and how to explain. American Educational Research Journal, 29(2), 303-323.
King, A., Staffieri, A., & Adelgais, A. (1998). Mutual peer tutoring: Effects of structuring tutorial
interaction to scaffold peer learning. Journal of Educational Psychology, 90(1), 134.
Klingenberg, S., Jørgensen, M. L., Dandanell, G., Skriver, K., Mottelson, A., & Makransky, G. (2020).
Investigating the effect of teaching as a generative learning strategy when learning through
desktop and immersive VR: A media and methods experiment. British Journal of Educational
Technology, 51(6), 2115-2138. https://doi.org/10.1111/bjet.13029
Kobayashi, K. (2019a). Learning by preparing‐to‐teach and teaching: A meta‐analysis. Japanese
Psychological Research, 61(3), 192-203. https://doi.org/10.1111/jpr.12221
Kobayashi, K. (2019b). Interactivity: A potential determinant of learning by preparing to teach and
teaching. Frontiers in psychology, 9, 2755. https://doi.org/10.3389%2Ffpsyg.2018.02755
Kobayashi, K. (2021). Learning by teaching face-to-face: the contributions of preparing-to-teach, initial-
explanation, and interaction phases. European Journal of Psychology of Education, 37, 551-566.
http://dx.doi.org/10.1007/s10212-021-00547-z
Koh, A. W. L., Lee, S. C., & Lim, S. W. H. (2018). The learning benefits of teaching: A retrieval practice
hypothesis. Applied Cognitive Psychology, 32(3), 401-410. https://doi.org/10.1002/acp.3410
Lachner, A., Jacob, L., & Hoogerheide, V. (2021). Learning by writing explanations: Is explaining to a
fictitious student more effective than self-explaining? Learning and Instruction, 74, 101438.
Lachner, A., Ly, K. T., & Nückles, M. (2018). Providing written or oral explanations? Differential effects of
the modality of explaining on students' conceptual learning and transfer. The Journal of
Experimental Education, 86(3), 344-361. https://doi.org/10.1080/00220973.2017.1363691
Leung, K. C. (2015). Preliminary empirical model of crucial determinants of best practice for peer
tutoring on academic achievement. Journal of Educational Psychology, 107(2), 558-579.
McNamara, D. S. (2004). SERT: Self-explanation reading training. Discourse Processes, 38(1), 1-30.
http://dx.doi.org/10.1207/s15326950dp3801_1
168
Mende, S., Proske, A., & Narciss, S. (2021). Individual preparation for collaborative learning: Systematic
review and synthesis. Educational Psychologist, 56(1), 29-53.
Muis, K. R., Psaradellis, C., Chevrier, M., Di Leo, I., & Lajoie, S. P. (2016). Learning by preparing to teach:
Fostering self-regulatory processes and achievement during complex mathematics problem
solving. Journal of Educational Psychology, 108(4), 474-492.
Nestojko, J. F., Bui, D. C., Kornell, N., & Bjork, E. L. (2014). Expecting to teach enhances learning and
organization of knowledge in free recall of text passages. Memory & Cognition, 42(7), 1038-
1048. https://doi.org/10.3758/s13421-014-0416-z
Nokes-Malach, T. J., Richey, J. E., & Gadgil, S. (2015). When is it better to learn together? Insights from
research on collaborative learning. Educational Psychology Review, 27(4), 645-656.
Palincsar, A. S., & Brown, A. L. (1984). Reciprocal teaching of comprehension-fostering and
comprehension-monitoring activities. Cognition and Instruction, 1(2), 117-175.
Pi, Z., Zhang, Y., Zhou, W., Xu, K., Chen, Y., Yang, J., & Zhao, Q. (2021). Learning by explaining to oneself
and a peer enhances learners’ theta and alpha oscillations while watching video lectures. British
Journal of Educational Technology, 52(2), 659-679. https://doi.org/10.1111/bjet.13048
Renkl, A. (1995). Learning for later teaching: An exploration of mediational links between teaching
expectancy and learning results. Learning and Instruction, 5(1), 21-36.
https://doi.org/10.1016/0959-4752(94)00015-H
Rittle-Johnson, B., Saylor, M., & Swygert, K. E. (2008). Learning from explaining: Does it matter if mom is
listening? Journal of Experimental Child Psychology, 100(3), 215-224.
https://doi.org/10.1016/j.jecp.2007.10.002
Roscoe, R. D. (2014). Self-monitoring and knowledge-building in learning by teaching. Instructional
Science, 42(3), 327-351. https://doi.org/10.1007/s11251-013-9283-4
Roscoe, R. D., & Chi, M. T. (2007). Understanding tutor learning: Knowledge-building and knowledge-
telling in peer tutors’ explanations and questions. Review of Educational Research, 77(4), 534-
Roscoe, R. D., & Chi, M. T. (2008). Tutor learning: The role of explaining and responding to
questions. Instructional Science, 36(4), 321-350. https://doi.org/10.1007/s11251-007-9034-5
van Blankenstein, F. M., Dolmans, D. H., van der Vleuten, C. P., & Schmidt, H. G. (2011). Which cognitive
processes support learning during small-group discussion? The role of providing explanations
and listening to others. Instructional Science, 39(2), 189-204. https://doi.org/10.1007/s11251-
009-9124-7
Wang, Y., Lin, L., & Chen, O. (2021). The benefits of teaching on comprehension, motivation, and
perceived difficulty: Empirical evidence of teaching expectancy and the interactivity of
teaching. British Journal of Educational Psychology, 91(4), 1275-1290.
https://doi.org/10.1111/bjep.12416
Webb, N. M. (1982). Student interaction and learning in small groups. Review of Educational Research,
52(3), 421-445. https://doi.org/10.3102/00346543052003421
169
Zhang, Q., & Fiorella, L. (2019). Role of generated and provided visuals in supporting learning from
scientific text. Contemporary Educational Psychology, 59, 101808.
https://doi.org/10.1016/j.cedpsych.2019.101808
Zhang, Q., & Fiorella., L. (2021). Learning by drawing: When is it worth the time and effort?
Contemporary Educational Psychology, 66, 101990.
https://doi.org/10.1016/j.cedpsych.2021.101990
170
In Their Own Words The Region of Proximal Learning and Curiosity

You to Know About Why and How Janet Metcalfe
Your Academic Setting Columbia University
Editors
Lauren L. Kordonowy
Victor A. Benassi
Overview
This chapter reviews the Region of Proximal Learning (RPL) perspective, a viewpoint that focuses on the
effects and nuances of titrating the difficulty of the to-be-learned materials to the individual level of the
learner, so that the challenges the learner faces are neither so ostensibly trivial and inconsequential that
boredom ensues nor so difficult and insurmountable that the learner is overwhelmed and gives up (see,
Kornell & Metcalfe, 2006; Metcalfe, 2002; Metcalfe & Kornell, 2003, 2005). If people perceive that they
already know the answers, their curiosity is not incited, and no learning occurs. But materials that are
too difficult and far removed from people's schemata—resulting in their being unable to understand or
appreciate their significance—will also fail to evoke their curiosity. I propose that people are most
curious and engaged, and that they learn best, when they are in their RPL which entails feeling that the
materials under study are 'almost known.'
The RPL is different for different individuals as a function of their expertise and knowledge in the
particular domain at issue. It is also different for the same individual in different domains and at
different times: the sweet spot where learning and curiosity are maximized is, thus, in constant flux as
the learner gains concepts, skills, understanding and knowledge. RPL, then, is a dynamic concept that
refers, at each point in time, to the difficulty level that productively induces the learner's curiosity and
focuses attention and learning effort. The boundary between 'knowing' and 'almost knowing' is crucial,
and it can be both shifting and fuzzy. Pedagogically important cases can arise in which people perceive
that they 'know' whereas really, they only 'almost know.'
This feeling of 'almost knowing' has been described by curiosity theorists as a perception of there being
a 'gap' in a person’s knowledge (Berlyne, 1950, 1954; Fastrich, et al., 2018; Kang et al., 2009;
Loewenstein, 1994, 2007; Marvin & Shohamy, 2016; Metcalfe et al., 2021). Resolving the gap results in
enhanced learning and, often, in a feeling of satisfaction, though it can sometimes be less rewarding
than was anticipated (Loewenstein, 2007). Once that knowledge has been attained and a sense of
closure is evoked, further curiosity and learning does not occur, at least in the moment. With forgetting,
as is shown by the blue arrows in Figure 1, known material may regress back into the RPL, and once
again become a prime candidate for curiosity and learning. The feeling that the material is learned can
sometimes be illusory, though, and can occur even though full learning has not yet been achieved. Such
illusions can have adverse consequences.
Taking on tasks that are well outside the individual's current RPL because they are either too easy or too
difficult may lead not only to momentary disengagement and a lack of learning, but also to a failure to
build up the necessary skills, knowledge, and capabilities needed to establish a knowledge base that will
eventually allow those more difficult tasks to themselves enter the RPL and become an enjoyable
171
challenge rather than an insurmountable obstacle. Getting the order right is paramount. Thus, the RPL
has consequences for the experience of pleasure and accomplishment associated with learning,
motivation, and epistemic curiosity (Litman, 2005).
Figure 1
Illustration of the Relation Between the Feeling of Having No Knowledge, Partial Knowledge, and Full
Knowledge, and Curiosity and Learning
Note: Subjective 'feeling of knowing,' on the x axis, ranges from unknown to known; Curiosity, which
ranges from low to high, is on the y axis. FOK / TOT = Feeling of Knowing / Tip of the Tongue.
In his seminal work on curiosity, Daniel Berlyne (1954) posed the following question: "Why, out of the
infinite range of knowable items in the universe, are certain pieces of knowledge more ardently sought
and more readily retained than others?" (p. 180) This is the core question of epistemic curiosity. The
answer posed by the RPL framework is that the individual is drawn to and epistemically curious about
those items that are in their RPL (although, of course, there are also survival-specific contents that may
have special access, Nairn et al., 2007). Many theorists have pointed out that people drill down when
they become aware that there is a 'gap' in their own knowledge base. The dominant theory of curiosity
by Loewenstein (1994, 2007), for instance, centers on such gaps. He notes: "The information-gap theory
views curiosity as arising when attention becomes focused on a gap in one's knowledge. Such
information gaps produce the feeling of deprivation labeled curiosity. The curious individual is motivated
to obtain the missing information to reduce or eliminate the feeling of deprivation" (p. 87) William
James, famously, noted that the tip-of-the-tongue state (TOT) is associated with gaps. Such states are
also associated with palpably high metacognitive feelings of 'almost knowing' and are quintessential
exemplars of items in the person's RPL. James (1893) wrote about TOTs:
The state of our consciousness is peculiar. There is a gap therein; but no mere gap. It is a gap
that is intensely active. A sort of wraith of the name is in it, beckoning us in a given direction,
making us at moments tingle with the sense of our closeness and then letting us sink back
without the longed-for term. If wrong names are proposed to us, this singularly definite gap acts
immediately so as to negate them. They do not fit into its mold. And the gap of one word does
not feel like the gap of another, all empty of content as both might seem necessarily to be when
described as gaps. (p. 251)
Brown and McNeill (1966) noted that a person in a TOT state appears to have been “seized” by the
state: "The signs of it were unmistakable; he would appear to be in mild torment, something like the
brink of a sneeze, and if he found the word his relief was considerable" (p. 326). A metacognitively
noticeable gap, operationalized as the feeling that one almost but not quite knows, is also at the core of
172
the RPL framework (Metcalfe et al., 2020). This feeling of imminent knowing underpins a state of
malleability where people are both motivated to find out what they do not yet quite know and where
they can succeed in learning it when they find out. Upon closure of the gap, curiosity (and the attendant
learning) plummets, as is shown in Figure 1.
As is also indicated in Figure 1, the boundary between the feeling that one 'almost knows' and that one
'knows' is thin, and the consequences of placement on the wrong side of the boundary dramatic. It turns
out that it is easy to unwittingly induce an illusion of knowing, with detrimental consequences for
learning. An instructor who is clever enough to thwart this metacognitive illusion of knowing can have
an outsize effect on the students' curiosity and their learning.

Learning and the Region of Proximal Learning
Almost knowing
The finding that people prefer to study material in their own RPL has been demonstrated with both
children and adults. For instance, in a series of studies that varied the expertise of subjects along with
the difficulty of the materials, it was found that when people are experts, and they are allowed to
allocate their own study time to materials that are easy, of medium difficulty, and difficult, they tend to
choose the more difficult materials. In contrast, novices tend to devote their time to the easy materials.
People tend to gravitate, then, to the level of difficulty that is appropriate for them (Metcalfe, 2002;
Metcalfe & Kornell, 2003).
Figure 2
Allocation of Study Time to Easy (Black), Medium-Difficulty (Blue), and Difficult (Pink) Items, Depending
on Total Study Time Available
Note. Data are from Metcalfe, 2002.

The results show that novice children tend to allocate their time to the easy items (left panel), whereas
adult experts allocate their time to the more difficult items (right panel). Furthermore, when people are
explicitly given a choice of whether to study items that are those they think they have almost learned--
as given by their own high judgments of learning—versus those that they have learned less well—as
given by low judgments of learning—they prefer the 'almost learned' items, that is, those that are more
likely to be in their own RPL (Kornell & Metcalfe, 2005).
173
Items that are on the tip-of-the-tongue (TOT), as noted above, are quintessential RPL items. People
claim that they 'almost know' the answer when they are in a TOT state. Schwartz (2016) proposed that
this state is universal: all 54 language groups that he investigated have an expression for the TOT state
(including American sign language, though it refers to the fingers not the tongue). And even in Q’eqchi’,
a language that does not have such an expression (Brennen et al.,2007), people, when queried,
expressed that that they experienced a distinctive state when they felt that recall was imminent. As well
as feeling like they almost have the answer, when they are in a TOT state, people also actually do have a
great deal of partial information and will devote considerable time and effort to attaining the answer
(Brown, & McNeill, 1966; Schwartz, 2016).
To directly investigate the consequences of this particular feeling of 'almost knowing,' Metcalfe, et al.,
(2017) showed college students a series of general information questions such as: "In addition to the
Kentucky Derby and the Belmont Stakes, what horse race completes the triple crown?" or "According to
the FDA, what is the only fish that produces real caviar?" Immediately upon seeing each question they
were asked (a) whether or not they were in a TOT state, and (b) how much they wanted to know the
answer. When they were in a TOT state, people were approximately twice as curious to know the
answer as compared to when they were not in a TOT state. Interestingly, even when they produced the
correct answer, they were sometimes in a TOT state. Apparently, they were not always certain about
the correctness of their own answers and could feel that they only 'almost knew' even when they were
correct. When they actually produced the answer but felt themselves to be below the threshold
indicating closure, they were highly curious. Furthermore, Bloom et al. (2018) showed that a particular
brain state that could be observed in electroencephalogram (EEG) tracings was also associated with
being in a TOT state and both this brain state and the TOT feeling, itself, as expressed by participants,
predicted later memory.
Mind Wandering
One of the most important implications of being in the Region of Proximal Learning is that the learner's
interest and attention is engaged. But, if being in one's RPL engages one's attention, the converse is also
true. An adverse consequence of not being in one's RPL is that attention flags, and boredom, and mind
wandering (Smallwood & Schooler, 2006) set in. It seems obvious that if a student is not paying
attention, the learning of the ignored material is going to suffer. And yet, rarely have methods of
teaching and study been systematically investigated to determine whether or not they are sufficiently
engaging to prevent mind wandering.
People engage in mind wandering during cognitive tasks at a surprisingly high rate, with some estimates
reaching up to 50% of the time (Killingsworth & Gilbert, 2010). Mind wandering is associated with lower
knowledge retention (Farley et al., 2013; Thomson et al., 2014), poorer comprehension (Smallwood,
2011; Unsworth & McMillan,2013), impaired memory for lectures (Spzunar et al., 2013), lower exam and
SAT scores (Unsworth et al., 2012), and diminished recall (Smallwood et al., 2007). Insofar as mind
wandering appears to be one of the largest deterrents to learning, understanding when and why
students engage in this behavior is an issue of paramount importance for education.
Interestingly, just as people are more likely to be on task when they are studying materials that are in
their RPL, they are more likely to mind wander when they are 'studying' materials that are outside their
RPL. To illustrate, Xu and Metcalfe (2016) presented students with Spanish-English translations, such as
those that a student learning Spanish might have to master in a language class. Their materials were
divided into pairs that were easy, of medium difficulty, or difficult. For example, Family-Familia was an
example of an easy translation; Music Hall-Vodevil was of medium difficulty; and Stain- Chafarrinada
was difficult. Given that the individual's RPL is specific to their own level of expertise, mind wandering
174
should differ depending on both the level of expertise of the subjects and the difficulty of the materials.
If one were a novice and knew nothing about Spanish, the very easy items might be in one's RPL, and
learning the small changes needed to make the easy translations would be sufficiently challenging to
engage attention. Mind wandering would be expected to set in with more difficulty. For a student with
slightly more expertise, who had taken an introductory course in Spanish, the easy items might have
already been mastered. However, figuring out a mediator that could be used to remember the medium-
difficulty items could be at just the right difficulty level to offset mind wandering. ('Vodevil' is reasonably
similar to the English word 'vaudeville,' the learner might consider, for so learning the translation for
music hall might not be so difficult after all, and, indeed, might actually even be fun). For experts, if both
the easy and medium difficulty items were mastered already, their presentation might invite mind
wandering. Perhaps only the very difficult items would engage their attention.
In Xu and Metcalfe's (2016) study, after having been presented with a block of pairs of items at one of
the three difficulty levels for between 15 and 90 seconds, the participants were then probed as to
whether they were still on task or were mind wandering. The results showed that people were much
more likely to mind wander if they were being shown items that were not in their own RPL. Everyone, of
course, mind wandered more as the session progressed, but the content of the materials themselves
had a strong effect. People who had low mastery of the materials and did poorly on the test, mind-
wandered most on the difficult materials. People who did very well on the test and had high mastery,
mind-wandered most on the easy materials.
At the Boundary Between ‘Almost Known’ and ‘Known’
One of the most distinctive aspects of the RPL framework is that there is a sharp boundary—in terms of
curiosity, attention, and learning potential—between items that are almost learned (and which garner
the greatest curiosity), and items that are known (and garner no curiosity).1 With closure, when the
1
In some cases, the boundary between 'almost known' and 'known' is best thought of as being sharp. The
switch in feeling state upon learning the answer to particular questions may often be abrupt (Berlyne, 1954),
as when one solves an insight problem (Danek et al., 2013). Nevertheless, in aggregate, rather than
empirically observing an increasing curiosity function followed by a precipitous drop at the time of 'knowing'
the boundary will, inevitably, appear to be fuzzy. Noise or variability at the boundary will contribute to its
presenting as an inverted u-shaped function rather than a cliff. First of all, as noted in the text, mere correct
responding is not enough to guarantee that the person feels a sense of closure that they now 'know' the
answer: people are sometimes still in an 'almost known' TOT state even when they have produced the
correct answer. Conversely, it is possible to be wrong, but to strongly feel that one's answer is
unimpeachable. Thus, the objective state of knowledge, while correlated with whether at any point in time
the answer falls above or below the critical boundary, is not crisp. Furthermore, metacognitive judgments are
noisy, imperfect and can be systematically biased. To give just one example, Son (2014) asked children to
provide ratings of how sure they were that they knew the answers to particular problems. Often, they would
say that they were 100% sure. This would seem to be the kind of rating that should indicate that they were
above the threshold. Interestingly though, when she asked the children whether they wanted to restudy the
Item, they frequently said that they did. 100% known apparently does not mean 100% known (and see
Fischhoff et al., 1977, for many examples of people's stated overconfidence and overuse of extreme
judgments). Our use of rating scales is inconsistent and noisy, if not systematically distorted. Thus, although,
in some cases, if the person feels that they have the answer already their curiosity may be entirely
extinguished, in practice, if curiosity were plotted against confidence, rather than a sharp peak and a vertical
cutoff being in evidence, an inverted U-shaped function peaking at the high end of confidence would
invariably occur--as has, indeed, been shown in a number of experiments (Kang, et al., 2009; Cohanpour,
2021; Metcalfe et al., 2022).
175
problem is solved or the item is known, curiosity is extinguished. Given that curiosity about the answers
to questions has been shown to be strongly related to memory (Fastrich et al, 2018; Kang et al., 2009),
whether a particular item falls on the 'almost known' side of the boundary where interest and
engagement are at the highest level, or on the 'known' side of the boundary where curiosity is sated,
could have considerable consequences for learning. We will now turn to a well-known case where the
lack of engagement appears to be due to wrongly thinking that one has already learned the answer,
when learning is not yet complete: the case of massed practice.
Massed Practice and RPL
It is well established that spaced rather than massed practice facilitates learning (Melton, 1970).
Consideration of the fragile border between 'known' and 'almost known' in the RPL model, and of the
impact that being on one or the other side of that border has on attention, may shed some light on
reasons for the difference between massed and spaced practice and, in particular, for the failure to
learn with massed practice.
Consider a hypothetical student's thought processes in a vocabulary learning experiment. Suppose the
materials are English-French translations, and they are presented at a 5-second rate. Suppose the pair
our student is currently trying to learn is "library-bibliothèque." It takes about a second to orient,
perceive, read, and internally vocalize the pair. She may then try to come up with mediators that will
help her to remember later. (This is a good idea, insofar as mediators actually do help learning, Crutcher
& Ericsson, 2000). She might think: "hm, bibliothèque sounds like bible," then "the bible is a book, and
books are in the library," and "oh, and there are books in the bible." Then she may think: " hm,
bibliothèque also sounds something like bibliographies, and those are something like card catalogues,
which, of course, are in the library." Three more seconds have passed. Then, running out of ideas, she
rotely rehearses "library-bibliothèque.... library-bibliothèque" a few times (which, by the way, has no
effect on later recall, see Craik & Watkins, 1973) and the 5 second period is over. It is time to move on to
the next pair. But what if that next pair is library-bibliothèque once again? This is the massed practice
situation. She has exhausted most of the mediators that will help, and she thinks she has learned the
pair well anyhow, especially since she has no difficulty retrieving it. So, although she rotely rehearses the
pair a few more times (a strategy that still has no effect), she is bored with this pair. She loses attention.
As Kahneman (1973) remarked, it is only possible to devote so much attention to one event. She starts
to look off into space but, fortunately, it is only 5 seconds before the next pair will appear.
Suppose, however, that the next pair is library-bibliothèque again. At this point her thoughts drift
towards the party she is planning for the weekend. She is soon completely engaged in mind wandering.
Indeed, she may no longer be focused on learning French at all, and may even miss the next pair, when
it comes up, since her party plans are so much more engrossing. This is not bad faith or a lack of
discipline on the part of our hypothetical student. If different pairs had come up that would have
allowed engagement (and especially if those pairs had been easy enough that she could show off her
virtuosity at coming up with what she thinks are terrific mediators—i.e., if the materials were in her RPL)
she would have been happy to learn. But the manner in which the practice was structured simply did
not afford the opportunity. At the same time, because she did rather well at coming up with the
mediators, and she can demonstrate to herself that she can fluently retrieve (or perhaps just read)
"bibliothèque" given "library," right now (and she has no experience of having any difficulty with it—as
might happen were she to try at a delay) she metacognitively reports that she learned very, very well,
and that massed practice is an excellent way to study.
In terms of the RPL model, the first few seconds of studying were effective but soon the item was
shuttled across the crucial boundary from the perceived 'almost known' to the perceived 'known.' The
176
student's motivation to further study that item was thereby quenched and her learning during that
second and third presentation reflected a lack of attention and effort. This is the massed practice
situation.
What would happen if, rather than presenting the same item again immediately, the experimenter had
presented something different? Our subject could have easily recruited all the good strategies to this
new content without boredom (as long, of course, as the new items were at the right level of difficulty
for our subject). Furthermore, if the first item were presented again after allowing some forgetting to
occur, the second presentation could also be beneficial. In that case, rather than being in the 'known'
state the item would once again be in the 'almost known' state—in the individual's RPL—just the right
place for long-term learning to occur. It would take less time to shuttle this item over the boundary,
though, and this would need to be taken into account in optimizing study time.
Spaced practice is advantageous, not only for item memory, but also for inductive learning, but probably
for similar reasons. In an experiment by Kornell and Bjork (2008), participants viewed many works of art
by particular artists, in order to be able to induce the style of the artist and thereby be able to recognize
new paintings as being by particular artists. The works were presented either in a massed manner, such
that many of one artist's works were presented together (as one might see at a one-person show, for
instance), or by being interleaved with works of other artists. The results showed that even in the
induction tasks, where the works were similar but not identical to one another, and the individual had to
generalize to similar but nonidentical items, spaced or interleaved practice resulted in better learning
than did massed practice. Xu and Metcalfe (2016) replicated the Kornell and Bjork (2008) finding, as is
shown in Figure 3, but they also showed that—just as in the case of massed versus space item
learning—people mind wandered more in the massed practice condition than in the spaced practice
condition.
Figure 3
The Difference in Mind Wandering (Left Panel) And Later Memory Performance (Right Panel) With
Massed and Spaced Practice.
There appear to be either only very small benefits or no benefits from immediate extended practice of
items when they are in the 'known' state, as a study by Vaughn et al. (2016) illustrates. People in their
study first learned a list of difficult foreign language translations to a criterion of 1 correct recall. Some
went on immediately in that first session to additional practice, repeating up to a criterion of 2, 3, 4, 5, 6
or 7 correct recalls—consuming a considerable amount of time and effort, of course. Participants all
went on to successive relearning sessions (see also Dunlosky et al., this volume), each one week apart. In
177
these, everyone relearned to a criterion of one correct recall. Performance, tested at a one-week delay
after the first session, hovered at around 25% correct, and was very slightly better (roughly 10%) for the
people who had relearned 7 times rather than only once on that first session. The spaced (by a week)
relearning sessions, though, had a much larger effect. After 4 spaced refresher sessions participants
reached a performance level of 80% correct. Furthermore, it no longer mattered at all whether they had
relearned once or 7 times on the first session. The gain from restudy when people 'almost knew' (i.e.,
the delayed or spaced sessions) far exceeded that of when they 'knew' (i.e., the first session repetitions).
The authors called this effect the "relearning-override effect".
People act as if the to-be-remembered item is already sufficiently learned, in the massed practice
condition, and this neglect contributes to their poor memory. There may, however, be other instances
where people have the misperception that materials are already learned. I will speculatively suggest
that students may sometimes inadvertently be exposed to what I call 'spoilers' –information that
produces the illusion that they know.
Spoilers and RPL
There are several brands of what I will call 'spoilers' that may make to-be-acquired knowledge seem like
it is already 'known' to the learner. Such spoilers harm curiosity and interest, and they are thereby
detrimental to learning. I will mention a few possible instances.
Many thousands of participants have been tested in 'generation-effect' experiments in which they had
to either generate an answer to a question or were, instead, provided with the answer (Slamecka &
Graf, 1978). Even the very slight uncertainty of having to come up with the answer to a category
exemplar probe such as Fruit-O_____ seems to be sufficient to put it on the 'almost known' side of the
boundary. However, in the read condition, being given the entire pair as Fruit-Orange may act as a
spoiler. The generation condition results in better later recall than does the read condition
(deWinstanley & Bjork, 2004).
Interestingly, if judgments of learning are made immediately upon presentation of to-be-learned cue-
target pairs, and those pairs are still available to be read by the subject, many targets that will not be
later remembered later are falsely evaluated as being learned. It seems that if the item is there, in front
of the subject (or if it is still in short-term memory; Nelson & Dunlosky, 1993), the subject cannot resist
thinking that it is known. This failure of metacognition, whereby one thinks one knows something that
one does not know when it is physically present or is in working memory, does not occur (or, at least,
not as badly) if the answers are not provided and physically present at the time of judgment (Dunlosky &
Nelson, 1992).
A similar phenomenon has been observed by Jacoby et al. (1994) who asked people to say how easy it
would be to come up with the answer to an anagram. When the answer was provided at the same time
they made the rating, they said it would be easy (i.e., it was more likely to be judged as 'known'),
whereas if they are not given the solution, they (correctly) rated it as being very difficult. The presence
of the answer, then, seems to act as a spoiler.
The benefits for memory of taking a test rather than studying (Roediger & Karpicke, 2006) may also be
due, in part, to a spoiler effect. Having access to all of the material to be studied at will, with no query to
even pique one's interest, may induce a false security of 'knowing.' While much emphasis has been
placed on the benefits of retrieval practice in this literature, little attention has been paid to the
question of why people get so little benefit from study. Why do they so clearly fail to engage and learn
in the study condition? It seems likely that because the material is so readily available to them, they feel
they have no need to study—they already know it.
178
Finally, the spoiler effect may be important in authentic educational settings as well. The learning
differences observed between the lecture format and interactive teaching formats (Freeman et al.,
2014) may result, in part, because a well-structured lecture provides the students with the illusion of
knowing. The uncertainty induced in the interactive classroom may instead result in a feeling of almost
but not quite knowing—a stance more conducive to learning.
Recommendations
The RPL framework is not a one-size-fits-all scheme for learning (Metcalfe, 2002). Indeed, it is closer to
one size fits none: the difficulty of the materials needs to be tailored to the knowledge of the individual
student if learning is to be optimized and curiosity nurtured. Although difficulty level is adjusted to the
skill of the player in karate and chess, in golf, and in computer games—all of which allow people to join
at whatever level they happen to be and to measure their progress relative to themselves rather than,
necessarily, comparing them to others—in schooling our children are usually grouped together by age
alone. According to RPL, they would be better off if there were a better fit between their knowledge and
the to-be-learned material.
One obvious way to accomplish such a fit would be for each child to have their own sensitive tutor who
deeply understood the student’s own level of achievement in each of their different domains of study,
and who adjusted the to-be-learned materials and their methods of teaching those materials to fit the
student’s ever-changing RPL level. Such a tutor would, of course, also benefit from knowing about
spoilers, and would be careful to not induce an illusion of knowing that might have inadvertent
consequences. The tutor would use spaced practice, of course, and Socratic methods—getting the
student to generate and guess whenever possible rather than passively receiving or being merely told
the answers. Such a tutor would test—not to evaluate the student but rather to enhance the student's
learning. The tests would also be used as formative assessments, to further tune the tutor's own
teaching to the RPL of the student. There are few studies investigating anything close to such a
possibility. Individual tutoring, though, even when the tutors are not necessarily informed of the wisdom
of the modern cognitive science of learning (Brown et al., 2014), has produced favorable results for
learning (Bausell et al., 1972).
The perfect human tutor may be a fantasy, at least for most students. However, a partial solution that
may become increasingly available in the near future may be found in adaptive computerized tutoring.
Of course, great care will need to be taken in the implementation.
Smaller class sizes, which would allow perceptive teachers to more adequately titrate the difficulty of
the materials to the needs of the individual students could be helpful. Although generally favorable, the
results of studies that have investigated the effects of smaller classes on student achievement are not
entirely consistent. As Glass and Smith (1979, and see, Achilles, 2012; Konstantopoulos & Chung, 2009;
c.f. Chingos, 2012) noted in discussing an early meta-analysis on the effects of class size on student
learning that, other things equal, students learn more in smaller classes. But things are not always equal.
The effectiveness of a move to smaller classes would depend on the teachers thereby being more able
to calibrate teaching to the RPLs of their students. More effective teaching to the RPL of individual
students is not an automatic consequence of smaller classes. Taking advantage of the possibility of
greater sensitivity that could enhance student learning would depend critically on the responsiveness of
the teacher (Jepsen & Rivkin, 2009).
The RPL framework also suggests that it could be productive to rethink the evaluation of students to
better reflect the individual's own learning trajectory. Learning evaluations should be intrapersonal—
how much the child has learned relative to the state of his or her own past self –rather than
179
interpersonal—pitting one child against another. As such, the dynamic RPL framework aligns with a self-
growth rather than a fixed mindset (Dweck, 2006).
Finally, it is extremely tempting for teachers who want, in good faith, to convey their knowledge to their
students, to simply tell the students the answers. However, it is all too easy to instill in people a feeling
that they know even if, demonstrably, they do not. The illusion of knowing that can come from
knowledge that is attained too easily can be pernicious. As such, it may be worthwhile to alert teachers
to the adverse consequences of spoilers.
Author Note
I would like to especially thank Judy Xu and Nate Kornell. This research was supported by NSF grant
1824193. The author alone is responsible for all interpretations herein. Please address correspondence
to: Janet Metcalfe, Department of Psychology, Columbia University, New York NY 10027;
[email protected]
References
Achilles, C. M. (2012). Class-size policy: The STAR experiment and related class-size studies. NCPEA Policy
Brief, 1(2), 1-9.
Bausell, R. B., Moody, W. B., & Walzl, F. N. (1972). A factorial study of tutoring versus classroom
instruction. American Educational Research Journal, 9(4), 591-597.
Berlyne, D. E. (1950). Novelty and curiosity as determinants of exploratory behavior. British Journal of
Psychology, 41(1-2), 68-80. https://doi.org/10.1111/j.2044-8295.1950.tb00262.x
Berlyne, D.E. (1954). A theory of human curiosity. British Journal of Psychology, 45(3), 180–91.
Bjork, R.A., & Bjork,. E. L. (1992). A new theory of disuse and an old theory of stimulus fluctuation. In A.
Healy, S. Kosslyn, & R. Shiffrin (Eds.), From learning processes to cognitive processes: Essays in
honor of William K. Estes (Vol. 2, pp. 35-67). Erlbaum.
Bloom, P.A., Friedman, D., Xu, J., Vuorre, M. & Metcalfe, J. (2018). Tip–of-the-Tongue states predict
enhanced feedback processing and subsequent memory. Consciousness and Cognition, 62, 206-
217. https://doi.org/10.1016/j.concog.2018.05.010
Brennen, T., Vikan, R., & Dybdahl, R. (2007). Are tip-of-the-tongue states universal? Evidence from an
unwritten language. Memory, 15(2), 167–176. https://doi.org/10.1080/09658210601164743
Brown, A.S. (1991). A review of the tip-of-the-tongue experience. Psychological Bulletin, 109(2), 204-
223. https://doi.org/10.1037/0033-2909.109.2.204
Brown, P. C., Roediger, H. L. III, & McDaniel, M. A. (2014). Make it stick: The science of successful
learning. Belknap Press of Harvard University Press.
Chingos, M. M. (2012). The impact of a universal class-size reduction policy: Evidence from Florida’s
statewide mandate. Economics of Education Review, 31(5), 543-562.
https://doi.org/10.1016/j.econedurev.2012.03.002
180
Craik, F. I. M., & Watkins, M. J. (1973). The role of rehearsal in short-term memory. Journal of Verbal
Learning and Verbal Behavior, 12(6), 599-607. https://doi.org/10.1016/S0022-5371(73)80039-8
Crutcher, R. J., & Ericsson, K. A. (2000). The role of mediators in memory retrieval as a function of
practice: Controlled mediation to direct access. Journal of Experimental Psychology: Learning,
Memory, and Cognition, 26(5), 1297-1317. https://doi.org/10.1037/0278-7393.26.5.1297
Danek, A. H., Fraps, T., von Müller, A., Grothe, B., & Ollinger, M. (2013). Aha! experiences leave a mark:
Facilitated recall of insight solutions. Psychological Research, 77(5), 659–669.
deWinstanley, P.A., & Bjork, E.L. (2004). Processing strategies and the generation effect: Implications for
making a better reader. Memory & Cognition. 32(6), 945–955.
https://doi.org/10.3758/BF03196872
Dunlosky, J., & Nelson, T. O. (1992). Importance of the kind of cue for judgments of learning (JOL) and
the delayed-JOL effect. Memory & Cognition, 20(4), 374-380.
Dweck, C. (2006). Mindset: The new psychology of success. Ballantine Books.
Farley, J., Risko, E. F., & Kingstone, A. (2013). Everyday attention and lecture retention: The effects of
time, fidgeting, and mind wandering. Frontiers in Psychology, 4, 1–9.
Fastrich, G. M., Kerr, T., Castel, A. D., & Murayama, K. (2018). The role of interest in memory for trivia
questions: An investigation with a large-scale database. Motivation Science, 4(3), 227–250.
https://doi.org/10.1037/mot0000087
Fischhoff, B., Slovic, P., & Lichtenstein, S. (1977). Knowing with certainty: The appropriateness of
extreme confidence. Journal of Experimental Psychology: Human Perception and Performance,
3(4), 552-564. https://doi.org/10.1037/0096-1523.3.4.552
mathematics. PNAS, 111(23), 8410-8415. https://doi.org/10.1073/pnas.1319030111
Gardiner, F. M., Craik, F. I. M., & Bleasdale, F. A. (1973). Retrieval difficulty and subsequent recall.
Memory & Cognition, 1, 213–216. https://doi.org/10.3758/BF03198098
Glass, G. V., & Smith, M. L. (1979). Meta-analysis of research on class size and achievement. Educational
Evaluation and Policy Analysis, 1(1), 2-16. https://doi.org/10.3102/01623737001001002
Jacoby, L. L., Bjork, R. A., & Kelley, C. M. (1994). Illusions of comprehension, competence, and
remembering. In D. Druckman and R. A. Bjork (Eds.), Learning, remembering, believing:
Enhancing human performance (pp.57-80). National Academy Press.
James, W. (1893). The principles of psychology. Henry Holt and Company.
Jepsen, C., & Rivkin, S. (2009). Class size reduction and student achievement: The potential tradeoff
between teacher quality and class size. Journal of Human Resources, 44(1), 223-250.
https://doi.org/10.3368/jhr.44.1.223
Kahneman D. (1973). Attention and effort. Prentice-Hall.
Kang, M. J., Hsu, M., Krajbich, I. M., Loewenstein, G., McClure, S. M., Wang, J. T., & Camerer, C. F. (2009).
The wick in the candle of learning: Epistemic curiosity activates reward circuitry and enhances
181
memory. Psychological Science, 20(8), 963–973. https://doi.org/10.1111/j.1467-
9280.2009.02402.x
Killingsworth, M. A., & Gilbert, D. T. (2010). A wandering mind is an unhappy mind. Science, 330(6006),
932–932. https://doi.org/10.1126/science.1192439
Konstantopoulos, S., & Chung, V. (2009). What are the long-term effects of small classes on the
achievement gap? Evidence from the Lasting Benefits Study. American Journal of Education,
Kornell, N., and Bjork, R. A. (2008). Learning concepts and categories: is spacing the “enemy of
induction”? Psychological Science, 19(6), 585–592. https://doi.org/10.1111/j.1467-
9280.2008.02127.x
Kornell, N., & Metcalfe, J. (2006). Study efficacy and the region of proximal learning framework. Journal
of Experimental Psychology: Learning, Memory, and Cognition, 32(3), 609-622.
Litman, J. A. (2005). Curiosity and the pleasures of learning: Wanting and liking new information.
Cognition & Emotion, 19(6), 793–814. https://doi.org/10.1080/02699930541000101
Loewenstein, G. (1994). The psychology of curiosity: A review and reinterpretation Psychological
Bulletin, 116(1), 75-98. https://doi.org/10.1037/0033-2909.116.1.75
Loewenstein, G. (2007). Exotic preferences: Behavioral economics and human motivation. Oxford
University Press
Marvin, C. B., & Shohamy, D. (2016). Curiosity and reward: Valence predicts choice and information
predictions errors enhance learning. Journal of Experimental Psychology: General, 145(3), 266-
272. https://doi.org/10.1037/xge0000140
Melton, A. W. 1970 The situation with respect to the spacing of repetitions and memory. Journal of
Verbal Learning and Verbal Behavior, 9(5), 596-606. https://doi.org/10.1016/S0022-
5371(70)80107-4
Metcalfe, J. (2002). Is study time allocated selectively to a region of proximal learning? Journal of
Experimental Psychology: General, 131(3), 349-363. https://doi.org/10.1037/0096-
3445.131.3.349
Metcalfe, J., & Kornell, N. (2003). The dynamics of learning and allocation of study time to a region of
proximal learning. Journal of Experimental Psychology: General, 132(4), 530-542.
Metcalfe, J., & Kornell, N. (2005). A region of proximal learning model of study time allocation. Journal of
Memory and Language, 52(4), 463-477. https://doi.org/10.1016/j.jml.2004.12.001
Metcalfe, J., Schwartz, B. L., & Bloom, P.A. (2017). The Tip-of-the-tongue (TOT) State and Curiosity.
Cognitive Research: Principles and Implications, 2 (article #: 31).
Metcalfe, J., Schwartz, B. L., & Eich, T.S. (2021). Epistemic curiosity in the Region of Proximal Learning.
Current Opinion in Behavioral Sciences, 35, 40-47.
https://doi.org/10.1016%2Fj.cobeha.2020.06.007
182
Metcalfe, J., Vuorre, M., Towner, E., & Eich, T. S. (2023). Curiosity: The effects of feedback and
confidence on the desire to know. Journal of Experimental Psychology: General, 152(2), 464–
482. https://doi.org/10.1037/xge0001284
Nairn, J. S., Thompson, S. R., & Pandierada, J. N. S. (2007). Adaptive memory: Survival processing
enhances retention. Journal of Experimental Psychology: Learning, Memory, and Cognition,
Nelson, T. O., & Dunlosky, J. (1993). When people's judgments of learning (JOLs) are extremely accurate
at predicting subsequent recall: The “delayed-JOL effect.” Psychological Science, 2(4), 267-271.
https://doi.org/10.1111/j.1467-9280.1991.tb001
Roediger, H. L., & Karpicke, J. D. (2006). Test-enhanced learning: Taking memory tests improves long-
term retention. Psychological science, 17(3), 249-255. https://doi.org/10.1111/j.1467-
9280.2006.01693.x
Schwartz, B. L. (1999). Sparkling at the end of the tongue: The etiology and phenomenology of tip-of-
the-tongue states. Psychonomic Bulletin & Review, 6, 379 - 393.
Schwartz, B. L. (2001). The relation of tip-of-the-tongue states and retrieval time. Memory & Cognition,
29, 117 - 126. https://doi.org/10.3758/BF03195746
Schwartz, B. L. (2008). Working memory load differentially affects tip-of-the-tongue states and feeling-
of-knowing judgment. Memory & Cognition, 36, 9 – 19. https://doi.org/10.3758/mc.36.1.9
Slamecka, N. J. & Graf, P. (1978). The generation effect: Delineation of a phenomenon. Journal of
Experimental Psychology: Human Learning and Memory, 4(6), 592-604.
Smallwood, J. (2011). Mind-wandering while reading: Attentional decoupling, mindless reading and the
cascade model of inattention. Language and Linguistics Compass, 5(2), 63–77.
https://doi.org/10.1111/j.1749-818X.2010.00263.x
Smallwood, J., & Schooler, J. W. (2006). The restless mind. Psychological Bulletin, 132(6), 946–958.
Smallwood, J., McSpadden, M., & Schooler, J. W. (2007). The lights are on but no one’s home: Meta-
awareness and the decoupling of attention when the mind wanders. Psychonomic Bulletin &
Review, 14(3), 527–533. https://doi.org/10.3758/BF03194102
Szpunar, K. K., Moulton, S. T., & Schacter, D. L. (2013). Mind wandering and education: From the
classroom to online learning. Frontiers in Psychology, 4, 495.
Thomson, D. R., Smilek, D., & Besner, D. (2014). On the asymmetric effects of mind-wandering on levels
of processing at encoding and retrieval. Psychonomic Bulletin & Review, 21(3), 728–33.
Unsworth, N., & McMillan, B. D. (2013). Mind wandering and reading comprehension: Examining the
roles of working memory capacity, interest, motivation, and topic experience. Journal of
Experimental Psychology. Learning, Memory, and Cognition, 39(3), 832–42.
183
Unsworth, N., McMillan, B. D., Brewer, G. A., & Spillers, G. J. (2012). Everyday attention failures: An
individual differences investigation. Journal of Experimental Psychology. Learning, Memory, and
Cognition, 38(6), 1765–72. https://doi.org/10.1037/xlm0001125
Vaughn, K.E., Dunlosky, J. & Rawson, K.A. Effects of successive relearning on recall: Does relearning
override the effects of initial learning criterion? Memory & Cognition, 44, 897–909.
https://doi.org/10.3758/s13421-016-0606-y
Wellman, H. M. (1977). Tip of the tongue and feeling of knowing experiences: A developmental study of
memory monitoring. Child Development, 48(1), 13–21. https://doi.org/10.2307/1128875
Xu, J. & Metcalfe, J. (2016). Studying in the region of proximal learning reduces mind wandering.
Memory & Cognition, 44, 681-695. https://doi.org/10.3758/s13421-016-0589-8
184
In Their Own Words What Every Teacher Should Know About
Cognitive Load Theory and the Importance of
Cognitive Load to Instruction
Greg Ashman and John Sweller
Editors
University of New South Wales
Lauren L. Kordonowy
Victor A. Benassi
Our knowledge of how the human mind works has been transformed over the last few decades. Until
recently, while we knew humans could learn, think, and solve problems, our understanding of the
machinery associated with these cognitive abilities was sketchy, with most pieces of the jigsaw puzzle
missing. In particular, we were not clear how learning, thinking, and problem-solving related to each
other and to teaching, and these connections were not illuminated by any coherent theory or
supporting data. For example, how effectively do we learn during problem solving? Does additional
knowledge of an area improve general problem-solving skills or just skills specific to the area? How
should problem solving be taught, and how should learning be improved? Without data and the
framework of a theory to help answer these and similar questions, the consequences for instructional
design and education in general, were dire. We could do little more than guess how we should organize
instruction. With the advantage of hindsight, it now is clear many of those guesses were wrong.
Cognitive load theory (Garnett, 2020; Lovell, 2020; Sweller et al., 2011; Sweller et al., 2019) has used our
knowledge of human cognition to design instructional procedures that improve learning and problem-
solving performance in educational contexts. We will begin by discussing those aspects of human
cognitive architecture relevant to education. Human cognitive architecture refers to the structures and
functions that govern how we learn, think, and solve problems. It does not necessarily refer to any
physical structures in the brain but instead to a model of processes that enables us to make predictions
relevant to learning.
Categories of Information
The knowledge and skills we acquire through learning can all be thought of as different forms of
information. While the amount of available information has exploded with the advent of the internet,
knowledge and skills that are desirable and important represent only a small proportion of the total
amount of information available.
Are all knowledge and skills the same and acquired in the same way? That seems unlikely. In order to
explore these potential differences, we can assign different types of knowledge and skill to different
categories. One such categorization scheme developed by David Geary is critical to education (Geary,
2012). That scheme is based on evolutionary educational psychology.
Biologically Primary and Secondary Information
Consider the knowledge and skill required to learn one’s native language. Whatever the language,
learning to speak it is an immensely complex task, but it is a task we are not usually explicitly taught.
Other than speech therapists, we do not teach children the enormously complex skill of organizing their
lips, tongue, breath, and voice when learning to speak their native language but neither do we need to
185
because we have evolved to learn to speak a native language without tuition. Children will acquire the
knowledge and skill required to speak their native language purely by being members of their society.
Similarly, they will learn the complex task of solving a problem by looking at where they are now in the
problem, looking at where they have to go to reach the goal, trying to find the differences between
where they are now and where they need to go, and finding moves that will reduce those differences
(Newell & Simon, 1972).
The information needed for these skills is acquired without conscious effort. We have evolved over
countless generations to acquire it automatically without being explicitly taught. It is a category of
information central to our existence as humans, called biologically primary knowledge and skills.
Not all information is biologically primary. There is a second category of knowledge and skills that is
acquired in a different manner and has different characteristics. Consider a student learning how to add
two numbers, learning how to solve an algebraic problem, learning how to read and write, or learning
about the history and structures of democratic political systems. These skills are not acquired
automatically and unconsciously. They need to be explicitly taught and explicitly learned. In each case,
they co-opt and build upon biologically primary skills, such as oral language, but they are distinct from
them.
Why are they distinct? Because we have evolved the capacity to learn biologically primary knowledge
and skills but not the capacity to learn this second category of knowledge and skills. For example,
humans developed writing systems several thousands of years ago, but it was not until a little over one
hundred years ago that most people in a few societies learned to read and write. The contrast with
biologically primary knowledge that is acquired effortlessly could hardly be greater. This second
category of information is called biologically secondary knowledge. It requires explicit instruction and
mental effort by learners and should never be confused with biologically primary knowledge as,
regretfully, frequently occurs.
We invented schools, universities, and other educational institutions to teach biologically secondary
knowledge and skills because without educational institutions, most people will not acquire the skills
required by modern societies. We have not evolved a mechanism for acquiring this knowledge and
these skills through simple immersion in the way we have evolved the capacity to learn our native
language. The bulk of this chapter is concerned with biologically secondary knowledge.
Domain-General and Domain-Specific Information
As well as the biologically primary and biologically secondary classification, another way of classifying
knowledge and skills is as domain-general or domain-specific (Tricot & Sweller, 2014). A domain is any
area of knowledge such as English literature, mathematics, or a branch of mathematics such as
geometry. Domain-general skills can be applied regardless of the particular domain of knowledge to
which they are applied. Many domain-general skills overlap with biologically primary information. The
general problem-solving skill of analyzing whether any moves you make will take you closer to the goal
is an example; it is a generic-cognitive skill that is unteachable. In contrast, domain-specific skills only
apply to particular domains of knowledge and overlap heavily with biologically secondary information.
We will not learn how to manipulate an algebraic equation unless we are taught how to do so. Once we
can effectively manipulate an algebraic equation, our skill in solving algebra problems will improve, but
this will not have a general effect on our ability to solve problems that do not involve algebra. If a
chemistry problem requires the manipulation of algebra, then this skill will also aid in the solution of this
class of chemistry problems, but it will not help with solving chemistry problems that do not involve
algebra. Although this skill is specific to algebra, it not only can, but must be taught if we wish to see an
improvement in the ability to solve algebra problems. This contrasts with the generic-cognitive skill of
186
general problem solving that humans have evolved to acquire automatically without instruction because
it is more important.
The Structures and Functions of Human Cognitive Architecture When

Acquiring Biologically Secondary, Domain-Specific Knowledge
Cognitive load theory is concerned with the way that the mind deals with biologically secondary,
domain-specific knowledge and skills. That means it is concerned with how learners acquire and use
biologically secondary, domain-specific knowledge, and how we should organize instruction to best help
learners acquire this category of knowledge and skills.
Basic Principles of Human Cognitive Architecture
Human cognitive architecture can be described by five basic principles used to generate instructional
design procedures. The same principles apply to the way that biological evolution processes
information, and so the information processes governing human cognition are analogous to the
information processes of biological evolution. Those principles constitute a natural information
processing system (Sweller & Sweller, 2006). The functioning of each principle is based on biologically
primary, generic-cognitive information that we have evolved to acquire in order for us to process
biologically secondary, domain-specific information.
The first two principles are concerned with how humans acquire novel, biologically secondary
information. Novel information can either be obtained during problem solving using the randomness as
genesis principle, or from other people, using the borrowing and reorganizing principle. When we are
unable to obtain information that we need from others, we have no choice but to try to work out how
to obtain it ourselves during problem solving. Any problem-solving strategy that we use will at some
point require a random generation and test procedure to generate novel problem-solving moves
(Sweller, 1988). For example, let us assume we want to solve a maze puzzle. We know the start and the
goal, but do not, of course, know how to connect them. We know that we need to move closer to the
goal, but also know that as is common in maze problems, we will often need to move away from the
goal. Frequently, the only way in which we can decide which route to take is to randomly choose a route
and test it for effectiveness: does it get us closer to the goal or does it lead to a dead end? The same
generate-and-test process is used to solve many types of problems. In educationally relevant areas, such
as learning how to solve particular mathematics problems or answer particular types of essay questions,
we can use this generate-and-test procedure to determine which solutions work best and then use them
to solve subsequent, similar problems. Ultimately, the randomness as genesis principle is the source of
all novel, biologically secondary knowledge. As previously discussed (Sweller & Sweller, 2006), we do not
need to be taught how to randomly generate problem-solving moves and test them for effectiveness.
We have evolved to acquire the knowledge for this skill and so it is biologically primary.
While the randomness as genesis principle is essential in generating novel information, it is slow and
clumsy. If possible, it is far more efficient to obtain information from other people than to generate it
ourselves and test it for effectiveness. The borrowing and reorganizing principle describes the way that
we obtain information from other people. We obtain information from others by imitating what they do
(Bandura, 1986), listening to what they say, and reading what they write. Once information is obtained,
it is reorganized to conform with previously obtained information.
The entire education enterprise is built on this biologically primary skill. We have evolved to obtain
biologically secondary information from others and are one of the few species capable of doing so
(Thornton & Raihani, 2008). An example of the borrowing and reorganizing principle can be found
whenever information is explicitly presented to learners in a classroom, in reading material, or via
187
electronic devices. The principle is a very efficient way of obtaining novel information compared to the
randomness as genesis principle. Of course, its disadvantage is that other people or sources of
information produced by other people may not be available or available only in an unsuitable form.
When unavailable, we can always attempt to use the randomness as genesis principle.
Once obtained, biologically secondary information must be processed and stored. The narrow limits of
change principle deal with the manner in which secondary information is processed by working memory
while the information store principle deals with how the information is stored in long-term memory for
later use. Both the processing of biologically secondary, domain-specific information in working memory
and the storage of that information in long-term memory are dependent on biologically primary,
domain-general cognitive skills that we have evolved to acquire.
Working memory is used to initially process information. If that information is novel, as it normally is
when considering educationally relevant, biologically secondary information, working memory has two
critical, related characteristics. It is severely limited in capacity (Cowan, 2001; Miller, 1956) and in
duration (Peterson & Peterson, 1959). We can process no more than about 3 elements of novel
information at a time and hold that information in working memory for no more than about 20 seconds.
These limitations are central to cognitive load theory and to instructional design. If only very limited
amounts of novel information can be simultaneously processed in working memory by learners and only
held in working memory for about 20 seconds, those facts have implications for the design and
presentation of educationally relevant material that teachers and everyone else concerned with
education need to consider.
Given these limitations in our ability to process novel information, the obvious question arises, how do
we manage to learn anything of substance? The answer lies in long-term memory, a structure that is at
least equal in importance to working memory. Long-term memory depends on the information store
principle. We store what appear to be unlimited amounts of information in long-term memory that can
be retrieved for later use (De Groot & Gobet, 1996). The only differences between experts and novices
in a given area are differences in domain-specific, biologically secondary information held in long-term
memory. Differences between experts and novices in the amount of information held in long-term
memory can be very large. That information, which is both teachable and learnable, can take many
years to acquire but provides the main justification for the existence of educational institutions.
The last principle which completes human cognitive architecture is the environmental organizing and
linking principle. This principle provides the machinery for appropriate action. Based on environmental
signals, we can transfer information from long-term memory to working memory in order to use that
information to generate action that is appropriate to the environment in which we find ourselves. If we
have learned how to solve problems such as a/b = c, solve for a, whenever we see a problem of this
type, we automatically transfer the problem and its solution from long-term memory to working
memory to solve the current problem. Again, information transfer is a biologically primary skill that does
not need to be taught. Skill in algebra depends on domain-specific knowledge associated with this
category of problems, not on teachable general problem-solving skills.
Unlike obtaining novel information from the environment, there are no working memory limits when
transferring familiar information from long-term to working memory. Enormous amounts of information
can be processed by working memory when that information originates from long-term memory. It is
due to these processes of human cognitive architecture in general, and the environmental organizing
and linking principle in particular, that the machinery of human cognitive architecture transforms us and
transforms human societies. We can accomplish things that we otherwise could not dream of
accomplishing. Education is transformational to the extent that its procedures conform to this
architecture.
188
Cognitive Load Theory
Cognitive load theory uses this understanding of cognitive architecture to devise novel instructional
procedures. The concept of element interactivity is central to the theory (Sweller, 2010). Element
interactivity provides another way of categorizing information in addition to biologically primary and
secondary information, and domain-general and domain-specific information. It is discussed here
because it leads directly to most of the cognitive load theory effects and the educational implications
outlined below.
Consider the biologically secondary task of learning some of the nouns of a foreign language or the
symbols of the chemical periodic table. We can learn the translation of the word “cat” with no reference
to the word “dog”. We can learn that the symbol for “iron” is “Fe” while the symbol for “copper” is “Cu”
without reference to each other. These tasks are low in element interactivity because each of the
elements can be learned without reference to any of the other elements.
In contrast, learning the grammar of a second language or how to balance a chemical equation cannot
be accomplished without considering multiple elements at the same time. Algebraic equations similarly
require the simultaneous consideration of multiple elements. If solving a problem, such as (a + b)/c = d,
solve for a, the entire equation with all of its elements needs to be considered simultaneously because
any change to one element is likely to affect all of the others. Element interactivity is high.
What is an element? An element is anything that needs to be learned or has been learned. Based on this
definition, element interactivity is determined by a combination of the nature of the information being
considered and the knowledge of the person considering the information. Element interactivity will
change if the information changes or the knowledge base of the learner changes. For example, once
fully learned, the sub-elements of the above equation become a single element.
• Low element interactivity information may be very difficult to learn if there are many elements.
Learning many vocabulary items can be a challenging task. Nevertheless, it is a task that imposes
a low working memory load because each element can be learned in isolation. In contrast, high
element interactivity information may be difficult for an entirely different reason. If a few
elements interact and so must be processed in working memory simultaneously, the learning
task may be difficult because of the limitations of working memory.
• Due to the characteristics of human cognitive architecture, element interactivity changes with
changes in knowledge. Based on the information store principle and the environmental
organizing and linking principle, once high element interactivity information is stored in long-
term memory, it can be transferred back to working memory as a single element without
imposing a heavy cognitive load. An expert in elementary algebra, on seeing the above
equation-based problem, can effortlessly transfer the entire problem including its solution from
long-term memory to working memory. For an expert, the solution to the problem is obvious. In
contrast, someone just introduced to such problems must determine the best first move and
how to make that move as well as considering any possible later moves. For a novice algebra
student, the symbols can be treated as elements. There are 10 symbols including the problem
goal of solve for a, a number far above the 2-4 elements that working memory can
simultaneously process. Solving the problem will impose a heavy working memory load in
contrast to an expert for whom the working memory load will be minimal and probably as low
as 1 if the entire problem and its solution is held in working memory.
• There is no simple way of quantifying element interactivity as a measure of complexity because
the objective “complexity” of information depends not just on the characteristics of the
189
information but also, the knowledge of the person processing that information. Information that
is impossibly complex for a novice is very simple for an expert. We cannot accurately measure
the complexity of information without knowing the knowledge base of the learner because that
knowledge base determines what constitutes a single element. However, it is possible to
estimate element interactivity by assuming a knowledge base and counting the number of
discrete elements that a learner must deal with. Furthermore, differences in element
interactivity, such as memorizing lists of individual items versus simultaneously considering
multiple elements when solving a problem, are sufficiently large to not require precise
measures. When dealing with vast differences in element interactivity, there can be little doubt
the differences are real. Our ability to determine smaller differences awaits further advances.
Element interactivity contributes to intrinsic cognitive load which depends on the intrinsic nature of the
information. As indicated above, it will be low when learning chemical symbols or high when learning to
deal with equations. The intrinsic cognitive load of information can only be changed by changing what
needs to be learned or changing the expertise of the learners. Intrinsic cognitive load should be
optimized to match what needs to be learned, rather than solely attempting to reduce the load
When element interactivity is changed by the instructional procedures being used, then element
interactivity contributes to extraneous cognitive load which is amenable to variation due to instructional
design. Extraneous cognitive load can be reduced by instructional design. Most of the instructional
effects, discussed next, are due to reductions in extraneous cognitive load. While extraneous cognitive
load needs to be reduced, intrinsic cognitive load needs to be optimized. An artificially low intrinsic
cognitive load may mean students are not learning as much as they can while a very high intrinsic may
exceed the limits of working memory.
Educational Implications: The Cognitive Load Effects

Cognitive load theory has been used to generate a variety of cognitive load effects. Each effect is based
on multiple, randomized, controlled trials comparing a novel instructional procedure with a currently
used procedure. The purpose of every cognitive load effect is to provide empirical evidence for a
recommended instructional procedure including its limitations and when it should and should not be
used. These educational implications provide the ultimate justification for the theory.
The Worked Example Effect
This effect occurs when learners presented worked examples to study perform better at later problem-
solving tasks than learners who are asked to solve the equivalent problems (Cooper & Sweller, 1987).
Based on the biologically primary borrowing and reorganizing principle, humans have evolved to obtain
information from other people. Worked examples provide problem-solving information from someone
else. Studying worked examples is a far more efficient way of obtaining information than problem-
solving using the randomness as genesis principle. Element interactivity is reduced by worked examples
compared to problem-solving because worked examples require learners to consider a single set of
problem-solving moves rather than the entire set of potential problem-solving moves while also
monitoring progress towards the goal. Worked examples, therefore, result in a reduction of extraneous
cognitive load. When stored in long-term memory, they increase domain-specific problem-solving skill.
Skill comes from accumulating large amounts of information in long-term memory which then can be
efficiently transferred back to working memory using the environmental organizing and linking principle.
Studying a worked example in order to enable us to remember it for subsequent use accords with
human cognitive architecture.
190
The worked example effect has been demonstrated on numerous occasions around the globe using a
large variety of curriculum areas from mathematics and science problems to English literature essays,
and a variety of learners from children attending primary schools to undergraduate university students.
The major educational implication is that showing learners how something should be done results in far
more learning than having them work it out themselves during problem solving.
The Element Interactivity Effect
Are worked examples always effective? All instructional effects have limits beyond which they are
ineffective. Commonly, those limits lead to new instructional effects. Extraneous cognitive load effects
are caused by working memory overload. If, for any reason, working memory is not overloaded,
cognitive load effects will not be observed. Whether working memory is overloaded depends on levels
of element interactivity. If the element interactivity associated with intrinsic cognitive load is low,
reducing element interactivity by reducing extraneous cognitive load may not improve learning. The
total cognitive load may not overload working memory. If so, worked examples may not be effective
when presented for information that is low in element interactivity due to intrinsic factors. In contrast,
we can expect that providing learners with worked examples of high element interactivity information
that imposes a high intrinsic cognitive load should be effective.
This result can be readily demonstrated (Chen et al., 2015). For example, if learners are presented with
several worked examples indicating the solution to geometry problems, they will learn more than if they
are required to solve the problems themselves, demonstrating the worked example effect. Geometry
problem solving is a high element interactivity task for novices. In contrast, if learners are asked to learn
and practice the names of geometrical shapes, a very low element interactivity task, once they are told
the name they will learn more by testing themselves (i.e., solving a problem) rather than being shown
the name repeatedly (i.e., studying a worked example). These differential results provide an example of
the element interactivity effect. Cognitive load effects like the worked example effect only occur under
high element interactivity conditions. When the information being dealt with is low in element
interactivity, providing learners with a series of worked examples is unnecessary and may be
counterproductive. Solving problems may be superior to studying worked examples for low element
interactivity information. Accordingly, the major educational implication is that when dealing with low
element interactivity information, instructional procedures should incorporate problem solving. In
contrast, for high element interactivity information, studying worked examples should be emphasized.
The Expertise Reversal Effect
As was indicated previously, whether information is low or high in element interactivity does not
depend only on the characteristics of the information. It also depends on the knowledge of the learner.
Information which is high in element interactivity for novices may be low in element interactivity for
experts. As a consequence, a version of the element interactivity effect can occur by comparing novices
and experts. For example, a heavy emphasis on worked examples may be important when novices are
learning a new area because the intrinsic cognitive load may be high. With increasing expertise, the
advantage of worked examples over problem solving reduces and at some point, the advantage may
reverse with practice at problem solving being superior to studying worked examples. As expertise
increases, the element interactivity of the information reduces to the point where studying worked
examples no longer is necessary and so is ineffective. This reversal in relative effectiveness, which also
occurs for other cognitive load theory effects, is called the expertise reversal effect (Kalyuga et al.,
2003). It is a variant of the more general element interactivity effect, discussed above (Chen et al.,
2017).
191
This effect has considerable educational implications. It suggests that while teachers should initially
provide students with many worked examples to study, as learners become more proficient, those
worked examples should be replaced with practice problems to solve.
The Redundancy Effect
This effect explains why the element interactivity and expertise reversal effects not only can eliminate
effects such as the worked example effect but can reverse them. With respect to the worked example
effect, why do reductions in element interactivity not only eliminate the advantage of worked examples
over problem solving, but sometimes reverse the advantage with problem solving proving superior to
studying worked examples (Kalyuga et al., 2001)? The redundancy effect provides the answer.
Redundant information is broadly defined as any unnecessary information. Text that unnecessarily
explains a clear diagram (Chandler & Sweller, 1991), a picture (such as a cartoon) added to instruction,
the same information simultaneously presented in written and spoken form during a PowerPoint
presentation, or listening to music while studying, all constitute redundancy. The effect occurs because
if limited working memory resources are devoted to elements of unnecessary information, fewer
resources are available for essential information. Accordingly, eliminating redundant information
reduces extraneous cognitive load and facilitates learning.
Explicit instruction such as the use of worked examples is designed to reduce the cognitive load imposed
by high element interactivity information. If information is low in element interactivity, either because it
is intrinsically low such as when learning new vocabulary or because lower-level elements have been
stored in long-term memory as a single, higher-level element, then explicit instruction may become
redundant. For example, when a learner merely needs to practice solving a particular class of problems,
detailed explanations of how to solve the problem within a worked example can be rendered
redundant. Accordingly, problem solving may prove to be superior to studying now redundant worked
examples.
The major educational implication of the redundancy effect is that providing learners with additional
information or requiring them to process unnecessary information has a cost. The increased cognitive
load will reduce learning. That adverse result can be avoided by eliminating redundant information.
The Guidance Fading Effect
The guidance fading effect indicates how the previous effects should be applied in real classrooms
(Renkl & Atkinson, 2003). Complex, high element interactivity information should be studied via multiple
worked examples. With increasing expertise, those worked examples should be faded out and replaced,
firstly by completion problems (van Merrienboer & Krammer, 1987) that provide a partial solution with
the remaining solution provided by learners, and subsequently by full problems to be entirely solved by
learners. In this manner, guidance is gradually faded out until learners can confidently and accurately
solve problems on their own. Of course, when dealing with relatively simple information, low in intrinsic
cognitive load, explicit instruction via practice using worked examples is unnecessary and may be
counterproductive. Based on randomized, controlled trials, the recommendation of gradual fading of
instructional guidance provides a major educational implication. It also follows that high levels of
guidance should be provided to learners when initially dealing with an area with problem solving
practice only introduced subsequently (Ashman et al., 2020).
Other Cognitive Load Effects
The above five related effects are the central effects of cognitive load theory. There are many other
instructional procedures that flow from the theory. Evidence for those procedures come from additional
192
cognitive load effects. A full list of the current cognitive load effects may be found in Table 1 of Sweller
et al. (2019). Some of the more prominent of those effects are the split-attention effect (Tarmizi &
Sweller, 1988) according to which related information that is presented in physically separated form
such as a diagram and related textual information imposes a heavier extraneous cognitive load than the
same information presented in physically integrated form; the modality effect in which, for example, a
diagram and related written text impose a heavier working memory load than the same diagram and
oral text, due to the fact that use of both the visual and auditory information channels increases working
memory capacity over the use of the visual only channel; the collective working memory effect
(Kirschner et al., 2018) in which collaboration between individuals reduces the cognitive load of those
individuals; the transient information effect (Leahy & Sweller, 2011) according to which transient
information such as oral presentations or videos increase working memory load in comparison to
permanent presentations such as written text or static diagrams; the human movement effect (Paas &
Sweller, 2012) which suggests that humans have evolved to readily learn to detect human movement
despite its transient nature; and the spacing effect, a well-known effect according to which providing
learners with frequent rest periods between learning episodes increases performance by allowing
working memory to recover after being depleted by excessive use (Chen et al., 2018). Each of these
effects flows from cognitive load theory, and each provides an instructional recommendation with
educational implications.
Conclusions
Knowledge of human cognitive architecture and evolutionary psychology has advanced substantially.
That knowledge provides a secure base for instructional design that previously was absent. In light of
that knowledge, many of our assumptions concerning instructional procedures appear to be at odds
with current knowledge, with the decades of emphasis on using problem solving to learn complex skills
being a prominent example. Cognitive load theory has used human cognition to generate hypotheses
that can be tested using randomized, controlled trials. Where successful, those trials provide
instructional effects that can be used as prescriptions for instructional procedures. The ultimate
validation of the theory lies in its ability to successfully generate instructional effects.
References
Ashman, G., Kalyuga, S., & Sweller, J. (2020). Problem solving or explicit instruction: Which should go
first when element interactivity is high? Educational Psychology Review, 32(1), 229-247.
Bandura, A. (1986). Social foundations of thought and action: A social cognitive theory. Prentice-Hall.
Chandler, P., & Sweller, J. (1991). Cognitive load theory and the format of instruction. Cognition and
Instruction, 8(4), 293-332. https://doi.org/10.1007/s10648-019-09500-5
Chen, O., Castro-Alonso, J. C., Paas, F., & Sweller, J. (2018). Extending cognitive load theory to
incorporate working memory resource depletion: Evidence from the spacing effect. Educational
Chen, O., Kalyuga, S., & Sweller, J. (2015). The worked example effect, the generation effect, and
element interactivity. Journal of Educational Psychology, 107(3), 689-704.
193
Chen, O., Kalyuga, S., & Sweller, J. (2017). The expertise reversal effect is a variant of the more general
element interactivity effect. Educational Psychology Review, 29(2), 393-405.
Cooper, G., & Sweller, J. (1987). Effects of schema acquisition and rule automation on mathematical
problem-solving transfer. Journal of Educational Psychology, 79(4), 347-362.
Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage
capacity. Behavioral and Brain Sciences, 24(1), 87-114.
https://doi.org/10.1017/S0140525X01003922
De Groot, A., & Gobet, F. (1996). Perception and memory in chess: Heuristics of the professional eye. Van
Gorcum & Co.
Garnett, S. (2020). Cognitive load theory: A handbook for teachers. Crown House Publishing.
Geary, D. (2012). Evolutionary Educational Psychology. In K. Harris, S. Graham, & T. Urdan (Eds.), APA
Educational Psychology Handbook (Vol. 1, pp. 597-621). American Psychological Association.
Kalyuga, S., Chandler, P., Tuovinen, J., & Sweller, J. (2001). When problem solving is superior to studying
worked examples. Journal of Educational Psychology, 93(3), 579-588.
Kirschner, P., Sweller, J., Kirschner, F., & Zambrano, J. (2018). From cognitive load theory to collaborative
cognitive load theory. International Journal of Computer-Supported Collaborative Learning,
Leahy, W., & Sweller, J. (2011). Cognitive load theory, modality of presentation and the transient
information effect. Applied Cognitive Psychology, 25(6), 943-951.
Lovell, O. (2020). Sweller's cognitive load theory in action. John Catt Educational.
Miller, G. A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for
processing information. Psychological Review, 63(2), 81-97. https://doi.org/10.1037/h0043158
Newell, A., & Simon, H. A. (1972). Human problem solving. Prentice Hall.
Paas, F., & Sweller, J. (2012). An evolutionary upgrade of cognitive load theory: Using the human motor
system and collaboration to support the learning of complex cognitive tasks. Educational
Peterson, L., & Peterson, M. J. (1959). Short-term retention of individual verbal items. Journal of
Experimental Psychology, 58(3), 193-198. https://doi.org/10.1037/h0049234
Renkl, A., & Atkinson, R. (2003). Structuring the transition from example study to problem solving in
cognitive skills acquisition: A cognitive load perspective. Educational Psychologist, 38(1), 15-22.
https://doi.org/10.1207/S15326985EP3801_3
Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2),
257-285. https://doi.org/10.1207/S15326985EP3801_3
194
Sweller, J. (2010). Element interactivity and intrinsic, extraneous and germane cognitive load.
Educational Psychology Review, 22(2), 123-138. https://doi.org/10.1007/s10648-010-9128-5
Sweller, J., & Sweller, S. (2006). Natural information processing systems. Evolutionary Psychology, 4(1),
434-458. http://dx.doi.org/10.1177/147470490600400135
Sweller, J., van Merriënboer, J., & Paas, F. (2019). Cognitive architecture and instructional design: 20
years later. Educational Psychology Review, 31(2), 261-292. https://doi.org/10.1007/s10648-
019-09465-5
Tarmizi, R. A., & Sweller, J. (1988). Guidance during mathematical problem solving. Journal of
Thornton, A., & Raihani, N. (2008). The evolution of teaching. Animal Behaviour, 75(6), 1823-1836.
https://doi.org/10.1016/j.anbehav.2007.12.014
Tricot, A., & Sweller, J. (2014). Domain-specific knowledge and why teaching generic skills does not
work. Educational Psychology Review, 26(2), 265-283. https://doi.org/10.1007/s10648-013-
9243-1
van Merrienboer, J., & Krammer, H. P. (1987). Instructional strategies and tactics for the design of
introductory computer programming courses in high school. Instructional Science, 16(3), 251-
285. https://doi.org/10.1007/BF00120253
195
In Their Own Words Productive Failure

Manu Kapur
You to Know About Why and How ETH Zurich
Editors
Ido Roll
Lauren L. Kordonowy
Technion - Israel Institute of Technology
Victor A. Benassi
Introduction
Suppose one wants to teach students a math concept that is novel to them, say variability (and the
procedure of calculating standard deviation, SD). The traditional, prevailing method is to first teach
students the concept and procedures of SD and then have them solve problems requiring the concept
and procedures. This sequence of instruction followed by problem solving is commonly referred to as
direct instruction (Kirschner et al., 2006). Advocates argue that students should be given the formal
target knowledge (such as equations or concepts) before being expected to apply it (Sweller, 2010).
A contrasting method is one that reverses the sequence, that is, one that engages students in problem-
solving first, and only then teaches them the concept and procedures (problem-solving followed by
instruction, or PS-I, Loibl et al., 2017). One successful approach for problem-solving followed by
instruction is called Productive Failure (Kapur, 2010, 2012, 2014; Chowrira et al., 2019). By ‘failure,’ we
simply mean that students will typically not be able to generate or discover the desired solution(s) by
themselves. However, to the extent that students are able to use their prior knowledge to generate
suboptimal or even incorrect solutions to the problem, the process can prepare students to learn better
from the subsequent instruction (Kapur & Bielaczyc, 2012; Schwartz & Martin, 2004). Not every PS-I
design is Productive Failure. Productive Failure is a subset of PS-I designs where the PS and I phases are
designed based on the principles of Productive Failure we outline in this chapter. Productive Failure
facilitates students’ understanding as they explore the task at hand and its structure, features, and
requirements, before being handed the tools to solve it. In this manner, productive failure combines the
benefits of exploratory problem-solving with those of explicit instruction, thereby mitigating the risk
that students do not discover the correct concepts and procedures on their own (Kapur & Rummel,
2012; Loibl et al., 2017).
In this chapter, we report on a program of research on Productive Failure and its implications for
designing instruction (see also Kapur, 2016). We start with a brief description of what Productive Failure
is, describing its underlying design and principles. We then discuss the research basis examining the
effectiveness of Productive Failure. Here we illustrate the effect by describing one of our classroom-
based experimental studies, followed by summarizing key findings from our meta-analysis of fifteen
years of research on productive failure. We end by discussing the educational implications of our
research.
Productive Failure (PF)

Productive Failure (PF) is a learning design that affords students opportunities to generate
representations and solutions to a novel problem that targets a concept they have not yet learned,
which is followed by consolidation and knowledge assembly, where they receive instruction on the
196
targeted concept. Because learners have not learned the concept, the problem-solving process usually
leads to failure. By failure, we mean that while students make progress, they are typically unable to
generate or discover the target solution(s) by themselves. However, as they struggle to construct valid
solutions, students gain understanding of the challenge, as well as its relationship with their existing
relevant knowledge, in a manner that prepares them to learn better from the subsequent instruction
(Kapur & Bielaczyc, 2012; Loibl et al., 2017 ; Schwartz & Martin, 2004;).
By encouraging students to analyze and attempt to solve relevant problems ahead of instruction, PF
addresses two challenges of direct instruction. First, in direct instruction, students often gain limited
understanding of the target concepts. This is because instruction uses language and representations that
are foreign to them. Instead, PF lets students apply their current understandings and thus differentiate
the required knowledge (e.g., Kapur & Bielaczyc, 2012; Schwartz & Bransford, 1998; Schwartz & Martin,
2004). Second, direct instruction presents the target concept as a monolithic entity, the rationale of
which students often fail to understand (Chi et al., 1988; Schwartz & Bransford, 1998). By recognizing
the requirements of the target solution during the initial problem-solving phase, students are more able
to understand the rationale behind the target procedures (Loibl et al., 2017).
It is important to emphasize that we do not advocate that any type of ill-designed problem-solving
activities prior to instruction suffices. Rather, we put forward an effective set of principles for design of
such activities. The PF process is designed around four key principles:
1. Activation and differentiation of prior knowledge in relation to the targeted concepts (that is,
understanding what existing knowledge is relevant, what it achieves, and what are its
shortcomings).
2. Attention to critical features of the targeted concepts (e.g., understanding the idea that
variability needs to account for the number of points, or is a property of distance).
3. Explanation and elaboration of these features, including their mathematical properties (e.g., for
the concept of variability, division by N helps control for sample size; subtraction is a measure of
distance).
4. Organization and assembly of the critical conceptual features into the targeted concepts.
PF has two main phases. During the initial problem solving (or Generation phase), students are given
opportunities to explore the affordances and constraints of different approaches towards the solution.
These approaches are based on familiar procedures and representations that students may already
know. During the subsequent Consolidation phase, students are given opportunities to assemble these
features and update them based on explicit instruction about the typical solutions which use the target
knowledge to be taught. As stated above, it is important to emphasize that not every failure is
productive, and not every task is suitable for exploration. The following design features support the
above-mentioned principles:
1. The problem-solving context should accommodate various solution approaches. Students
should be able to make progress in these, even if they do not solve them entirely. While
students should be challenged, they should not be frustrated (see Metcalfe, this volume, on the
region of proximal learning).
2. These challenges should invite students to explain and elaborate on their existing knowledge as
well as task requirements.
3. While no explicit, external feedback is provided, students should be able to extract situational
feedback by evaluating their solution approaches using their intuition and interpretation of their
outcomes (Nathan, 1998; Roll et al., 2014). Such self-feedback helps students better understand
197
the properties of their approaches. Students later should contrast their own solutions with the
taught target procedure.
Evidentiary Basis of PF
Here we illustrate with an example of one such study (Kapur, 2012) in a real classroom setting. Kapur
(2012) compared learning from PF and Direct Instruction (DI) through a pre-posttest, quasi-experimental
study with 133 ninth-grade mathematics students (14–15-year-olds) from a public school in Singapore.
Students had no instructional experience with the targeted concept prior to the study. All students, in
their intact classes, participated in four, 50-minute periods of instruction on the concept as appropriate
to their assigned condition. The same teacher taught both the PF and DI conditions.
In the PF condition, students spent forty minutes to solve the following data analysis problem on their
own:
The PF problem solving task
Mr. Fergusson, Mr. Merino, and Mr. Eriksson are the mangers of the Supreme Football Club.
They are on the lookout for a new striker, and after a long search, they short-listed three
potential players: Mike Arwen, Dave Backhand, and Ivan Right. All strikers asked for the same
salary, so the managers agreed that they should base their decisions on the players’
performance in the Premier League for the last 20 years. Table 1 shows the number of goals
that each striker had scored between 1988 and 2007.
Table 1: Number of goals scored by three strikers in the Premier League.
Year Mike Arwen Dave Backhand Ivan Right
1988 14 13 13
1989 9 9 18
1990 14 16 15
1991 10 14 10
1992 15 10 16
1993 11 11 10
1994 15 13 17
1995 11 14 10
1996 16 15 12
1997 12 19 14
1998 16 14 19
1999 12 12 14
2000 17 15 18
2001 13 14 9
2002 17 17 10
198
2003 13 13 18
2004 18 14 11
2005 14 18 10
2006 19 14 18
2007 14 15 18
The managers agreed that the player they hire should be a consistent performer. They
decided that they should approach this decision mathematically and would want a formula for
calculating the consistency of performance for each player. This formula should apply to all
players and help provide a fair comparison. The managers decided to get your help. Please
come up with a formula for consistency and show which player is the most consistent striker.
Show all working and calculations on the paper provided.
The data analysis problem presented a distribution of goals scored each year by three soccer players
over a twenty-year period. Students were asked to design a quantitative index to determine the most
consistent player. The choice of consistency was to help students harness their intuitive understanding
of the concept to their mathematical generation of indexes. During this generation phase, no explicit
guidance was provided. After this, the teacher first consolidated by comparing and contrasting student-
generated solutions with each other, and then modeled and worked through the canonical solution.
Finally, students solved three data analysis problems for practice, and the teacher discussed the
solutions with the class.
In the DI condition, the teacher explained the canonical formulation of the concept of variance using a
series of worked examples. After each worked example, students solved a comparable problem, then
their errors, misconceptions, and critical features of the concept were discussed with the class as a
whole. Having learned the concept, students then solved the same problem that the PF students solved
in their generation phase, following which the teacher discussed the solutions with the class. The DI
cycle ended with a final set of three data analysis problems for practice (the same problems given to the
PF students), and the teacher discussed the solutions with the class.
Process findings suggested that PF groups generated on average six solutions to the problem. These
student-generated solutions have been described in greater detail elsewhere (see Kapur, 2012, 2013,
2014). Notably, none of the PF groups were able to generate the canonical formulation of SD.
Furthermore, none of the PF groups generated a consistent and effective mathematical expression that
captures consistency. In contrast, analysis of DI students’ classroom work revealed that students relied
only on the canonical formulation to solve data analysis problems. The solutions generated by PF
students suggested that not only were students’ prior knowledge activated (central tendencies,
graphing, differences, etc.) but that students were able to assemble them into different ways of
measuring consistency. Therefore, the more solutions students can generate, the more it can be argued
that they are able to conceptualize the targeted concept in different ways; that is, their prior knowledge
is not only activated but also differentiated in the process of generation. In other words, these solutions
can be seen as a measure, albeit indirect, of knowledge activation and differentiation; the greater the
number of such solutions, the greater the knowledge activation and differentiation.
On the day immediately following the intervention, all students took a posttest comprising three types
of items: procedural fluency, conceptual understanding, and transfer (for the items, see Kapur, 2012).
Items that evaluated procedural fluency required students to calculate the value of SD and interpret it in
199
a given context. Items that evaluated conceptual understanding required students to notice missing
features in suboptimal solutions and correct them, articulate why SD is formulated the way it is, and
apply its mathematical properties. Transfer items required students to flexibly adapt their knowledge of
SD to solve problems on normalization, a concept that was not taught nor prompted during instruction.
Analysis of pre-post performance suggested that PF students significantly outperformed their DI
counterparts on conceptual understanding and transfer without compromising procedural fluency.
Further analyses revealed that the number of solutions generated by PF students was a significant
predictor of how much they learned from PF. That is, the more solutions they generated, the better they
performed on the procedural fluency, conceptual understanding, and transfer items on the posttest.
These findings are consistent with the earlier studies on productive failure (Kapur, 2008; Kapur & Kinzer,
2009), as well as with additional studies that show the benefits of having students invent (often faulty)
solutions to novel problems as a way of preparing them for future instruction (Chowrira et al., 2019;
Schwartz & Bransford, 1998; Schwartz & Martin, 2004). Notably, evidence for learning from PF comes
not only from quasi-experimental studies conducted in the real ecologies of classrooms across topics
and age groups (e.g., Kapur, 2012, 2013; Schwartz & Bransford, 1998; Schwartz & Martin, 2004), but also
from controlled experimental studies (e.g., DeCaro & Rittle-Johnson, 2012; Kapur, 2014; Loibl &
Rummel, 2014; Roll et al., 2011; Schmidt & Bjork, 1992; Schwartz et al., 2011; Chowrira et al., 2019).
These findings are also consistent with the math education literature that emphasizes the role of
struggle in learning (e.g., Hiebert & Grouws, 2007). More broadly, these findings can also be seen to be
consistent with some forms of Problem-Based learning (PBL) environments in which students are given
just-in-time instruction after they have engaged in problem-solving first (Capon & Kuhn, 2004; Hmelo-
Silver et al., 2007). More broadly, these findings demonstrate the value of constructivist instruction
(Tobias & Duffy, 2009). Proponents of DI often point out sparse data to support instruction that is
minimally guided (Kirschner et al., 2006; Klahr, 2010; Sweller, 2010). The PF approach consistently
demonstrates that, done right, students’ sense making based on their prior knowledge can be effective
vehicles to promote more robust learning. However, it is important to emphasize that these are
effective learning tools when followed by DI. That is, rather than a single-phase instruction that either
provides or withholds information, the PF effect demonstrates the value of combining them by delaying
information.
To determine the effectiveness of PF across all the studies, we carried out a meta-analysis of relevant
research from the past two decades. Our meta-analysis of more than 12,000 participants in 166
experimental comparisons (Sinha & Kapur, 2021) found that:
1. Productive Failure students significantly outperformed their counterparts in the traditional
instruction-first classrooms in conceptual understanding and transfer (Cohen’s d = 0.36 [95%
confidence interval (CI) 0.20, 0.51]), without compromising procedural knowledge; simply put,
students exposed to the PF design developed a better understanding of domain-specific ideas,
why these ideas work for certain problems, and whether and how these ideas can be applied to
solve novel (but related) problems in the future.
2. The higher the fidelity to the design principles of PF, the stronger the effect (with Cohen’s d up
to 0.58). To put this in context, this effect is about three times the effect a good teacher has
on student learning in one year.
The PF effect is far from magic. It is the result of carefully designed instruction that invokes evidence-
based learning mechanisms. It is worth emphasizing again that not every problem-solving activity
200
followed by instruction sequence equals PF. The meta-analysis (Sinha & Kapur, 2021) clearly showed
that fidelity to PF principles is key. We first discuss the fidelity implications of PF, followed by more
general implications for orchestrating PF in academic courses.
Fidelity to PF Principles
Fidelity to PF comes from making sure the design of the problem-solving tasks and activities is in line
with the PF principles, as is the context and the social surround or classroom culture within which PF is
used.
1. Features of PF tasks. Task development is key. To be effective, tasks should include the following
features. These often take 2-3 iterations to develop:
• Contrasts. Tasks should include contrasts that can be intuitively, and even better if
perceptually, evaluated. These allow learners to evaluate their own solutions. While no
explicit feedback is given, learners should be able to evaluate their own generated
methods.
• Variant-invariant relations. Varying certain features of the task and keeping others
invariant affords students the opportunities to focus on the critical features. For
example, in the PF task earlier, we kept the central tendencies and the range the same.
Once students noticed this, it afforded them and even implicitly nudged them to focus
on the distribution of the data. There is no one way of doing this. Alternatively, variant-
invariant relations can be designed on structure and stories(e.g., keeping structure
constant but changing the stories, or vice versa). The same could be done with multiple
representations. All of these must be carefully designed to keep some features constant
and others variable.
• Affective draw. To encourage learners to persist in these challenging problems the tasks
should be engaging. Using intuitive language and inviting intuitive and informal ways of
reasoning motivates students to engage with the tasks.
• Room for baby steps. Correct solutions should not require insights. Instead, learners
should be able to make gradual progress towards meaningful approaches to solving the
challenges. While overall students’ solutions are typically erroneous, they often include
valid features and ideas.
• Invite multiple solutions and representations. To facilitate students’ understanding of
the target concept and its relationship with neighboring concepts, tasks should invite a
variety of solutions or representations.
• Low computational load. Students should focus on designing ideas and exploring their
properties, rather than on calculating or applying solutions.
• Be problem oriented. While the goal and benefits of PF are conceptual, PF tasks ask
students to compute different attributes by designing methods. By contextualizing the
challenge, learners explore more aspects of the problem and are given tools to evaluate
their progress.
• Working alone or together. Although the initial design of PF advocated for collaborative
problem solving, thus far the evidence shows that PF is effective in both configurations,
whether learners work individually or in small groups (e.g., Mazziotti et al., 2019).
2. Classroom context and culture: Managing student expectations and norms in line with PF design
is key. Students should be encouraged to explore and take risks. The expectation should be on
201
the effort and generating ideas even if they are incorrect or suboptimal. The guiding principle is
that every answer that can be explained is a good answer. Answers that provide fuller
explanations are better. Students should be told that they may struggle a little and feel
frustrated, yet it is this struggle and dealing with frustration that is normal and even seen as an
indicator of learning. These expectations and norms need to be emphasized repeatedly over the
course of the academic term so that students understand the new didactic contract being put in
place. In many ways, one could conceive the unit of design and change not to be a pedagogy or
teaching method, but classroom culture.
General Considerations and Implications

Having worked with hundreds of teachers, schools, and even education systems at large in a variety of
applied contexts, we now outline some general considerations for using PF.
• Let students grapple with new concepts: In our daily lives, most effective learning comes by first
encountering new topics. Also in schools, novices should be invited to meaningfully explore new
concepts before being taught target procedures. This exploration helps them build relevant
experiences with which they can later encode instruction.
• Type of knowledge: PF is effective when learning new concepts, and it shows effects on
conceptual learning and preparation for future learning (PFL). For example, it helps students
understand how certain concepts should be applied, what they mean, and how they can be used
to assemble more complex ideas. Alas, compared to DI, PF is not more effective for procedural
fluency, for example, calculating mean, manipulating algebraic variables, and so on.
• Budget your use of PF: It follows from point 2 that we do not advocate the use of PF for all types
of learning goals. Do not use PF in every lesson. Instead, target 3-5 key concepts and big ideas
over the course of a semester. Big ideas are those that are foundational to multiple topics, such
as ratios (Koedinger & Roll, 2012). We have found that designing for PF does not require a major
overhaul in the curriculum content or time. Even a small change prior to key concept lectures
can result in positive effects.
• Use PF in your STEM lessons: Our meta-analysis showed that PF was found to work well across
STEM topics (Loibl et al., 2017; Sinha & Kapur, 2021). However, so far, it was not found to be as
effective for domain-general skills, and there is only scarce evidence for non-STEM fields. While
absence of evidence is not evidence of absence, much more research is needed in domain-
general skills and non-STEM fields before we can make an evidence-based claim about using PF.
• Learner profile: PF is an effective tool in diverse classrooms. It has shown to work with all ability
groups provided the design fidelity is good. Age-wise, meta-analysis shows better effects for
older students (secondary school onwards), possibly because they have better problem-solving
skills. Older students are arguably better at analyzing their own solutions and extract situational
feedback that allows them to learn from their challenges. The fidelity of PF studies with the
younger children also tended to be low. Once again, more research is needed here.
• Technology: Some evidence suggests that technology can be used to support PF (Chase et al.,
2019; Roll et al., 2010; Shalala et al., 2021). Blended learning, gaming, and Virtual- and Mixed-
Reality (VR and MR) environments are all relevant contexts for PF. For example, we are engaging
undergraduate students in playing complex ethical scenario games in their research contexts
prior to learning ethics in their traditional lectures. Likewise, we are designing embodied
interaction activities in VR and MR environments in calculus, graph theory, and linear algebra as
pre-instructional activities. Similar ideas are being applied in medical education using clinical
diagnosis and reasoning simulations.
202
• Assessment of effectiveness: PF shows benefits on a variety of measures, such as student
production, classroom interaction, performance on tests provided they measure conceptual
understanding and transfer. As the PF process requires complex problem solving and
independent learning, it provides an opportunity to assess key competencies and attributes,
such as persistence, creativity, collaboration, and interest. Alas, with few exceptions, these have
not been studied sufficiently in the literature (c.f. Belenky & Nokes-Malach 2013; Massey-Allard
et al., 2019; Roll et al., 2012;)
• Teacher Professional Development: Teachers should be supported in learning to design PF
instruction. There is a need for a development runway for iterations and experimentation.
Failure to design for this often results in premature failure of PF.
Conclusion
Contrary to the commonly held belief that there is little efficacy in having learners solve novel problems
that target concepts they have not yet learned, our work suggests that there is indeed such an efficacy
even if learners do not formally know the underlying concepts needed to solve the problems, and even
if such problem-solving leads to failure initially. A wealth of research has identified several design
guidelines that support learning by attempting to generate solutions to novel problems. Applying these
principles to key concepts can achieve a large effect on students’ knowledge and motivation. Further, PF
is not a monolithic design and other designs may facilitate similar effects. Notably, PF is only one way to
orchestrate known mechanisms that are effective also in other forms of instruction. For example,
creating a safe classroom culture, encouraging students to activate their prior knowledge, or providing
room for students to make sense of their solutions have all been shown to be effective.
Author Note
This work was supported by several Ministry of Education, Singapore grants to the first author, and also
by ETH Zurich, Switzerland. This work was also supported by the Gordon and Betty Moore Foundation
and by the Israel Science Foundation (ISF) Grant No 1573/21 to the second author. We thank Dr. Janan
Saba for her inputs. Any opinions are those of the authors and do not represent the views of the funding
agencies.
References
Belenky, D. M., & Nokes-Malach, T. J. (2013). Mastery-approach goals and knowledge transfer: An
investigation into the effects of task structure and framing instructions. Learning and Individual
Differences, 25, 21–34. https://doi.org/10.1016/j.lindif.2013.02.004
Capon, N., & Kuhn, D. (2004). What’s so good about problem-based learning? Cognition and
Instruction, 22(1), 61–79. https://doi.org/10.1207/s1532690Xci2201_3
Chase, C. C., Connolly, H., Lamnina, M., & Aleven, V. (2019). Problematizing helps! A classroom study of
computer-based guidance for invention activities. International Journal of Artificial Intelligence
in Education, 29(2), 283-316. https://doi.org/10.1007/s40593-019-00178-y
Chi, M. T. H., Glaser, R., & Farr, M. J. (1988). The nature of expertise. Erlbaum.
Chowrira, S. G., Smith, K. M., Dubois, P. J., & Roll, I. (2019). DIY productive failure: boosting performance
in a large undergraduate biology course. NPJ Science of Learning, 4(1), 1-8.
203
DeCaro, M. S., & Rittle-Johnson, B. (2012). Exploring mathematics problems prepares children to learn
from instruction. Journal of Experimental Child Psychology, 113(4), 552-568.
https://doi.org/10.1016/j.jecp.2012.06.009
Hiebert, J., & Grouws, D. A. (2007). The effects of classroom mathematics teaching on students’
learning. In F. Lester (Ed.), Second Handbook of Research on Mathematics Teaching and
Learning (pp. 371-404). Information Age.
Hmelo-Silver, C. E., Duncan, R. G., & Chinn, C. A. (2007). Scaffolding and achievement in problem-based
and inquiry learning: a response to Kirschner, Sweller, and Clark (2006). Educational
Holmes, N. G., Day, J., Park, A. H. K., Bonn, D. A., & Roll, I. (2014). Making the failure more productive:
scaffolding the invention process to improve inquiry behaviors and outcomes in invention
activities. Instructional Science, 42(4), 523–538. https://doi.org/10.1007/s11251-013-9300-7
Kapur, M. (2008). Productive failure. Cognition and Instruction, 26(3), 379-424.
Kapur, M. (2010). Productive failure in mathematical problem solving. Instructional Science, 38(6), 523-
550 https://doi.org/10.1007/s11251-009-9093-x
Kapur, M. (2012). Productive failure in learning the concept of variance. Instructional Science. 40(4), 651-
Kapur, M. (2013). Comparing learning from productive failure and vicarious failure. The Journal of the
Learning Sciences, 23(4), 651-677. https://doi.org/10.1080/10508406.2013.819000
Kapur, M. (2014). Productive failure in learning math. Cognitive Science, 38(5), 1008-
1022. https://doi.org/10.1111/cogs.12107
Kapur, M. (2016). Examining Productive Failure, Productive Success, Unproductive Failure, and
Unproductive Success in Learning. Educational Psychologist, 51(2), 289-299.
Kapur, M., & Bielaczyc, K. (2012). Designing for productive failure. The Journal of the Learning Sciences,
21(1), 45-83. https://doi.org/10.1080/10508406.2011.591717
Kapur, M., & Kinzer, C. (2009). Productive failure in CSCL groups. International Journal of Computer-
Supported Collaborative Learning (ijCSCL), 4(1), 21-46. https://doi.org/10.1007/s11412-008-
9059-z
Kapur, M., & Rummel, N. (2012). Productive failure in learning from generation and invention activities.
Instructional Science, 40(4), 645-650. https://doi.org/10.1007/s11251-012-9235-4
Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why minimal guidance during instruction does not
work. Educational Psychologist, 41(2), 75-86. https://doi.org/10.1207/s15326985ep4102_1
Klahr, David (2018): Coming Up for Air: But Is It Oxygen or Phlogiston? A Response to Taber's Review of
Constructivist Instruction: Success Or Failure? Carnegie Mellon University. Journal contribution.
https://doi.org/10.1184/R1/6614000.v1
Koedinger, K. R., & Roll, I. (2012). Learning to think: Cognitive mechanisms of knowledge transfer. In K. J.
Holyoak & R. G. Morrison (Eds.), The Oxford handbook of thinking and reasoning (pp. 789–806).
Oxford University Press.
204
Loibl, K., Roll, I., & Rummel, N. (2017). Towards a theory of when and how problem solving followed by
instruction supports learning. Educational Psychology Review, 29(4), 693-715.
Loibl, K., & Rummel, N. (2014). Knowing what you don't know makes failure productive. Learning and
Massey-Allard, J., Roll, I., & Ives, J. (2019). Supporting productive exploration in invention activities: are
simulations too challenging? Paper presented at Physics Education Research Conference 2019,
Provo, UT.
https://www.compadre.org/Repository/document/ServeFile.cfm?ID=15302&DocID=5214
Mazziotti, C., Rummel, N., Deiglmayr, A., & Loibl, K. (2019). Probing boundary conditions of Productive
Failure and analyzing the role of young students’ collaboration. NPJ science of learning, 4(1), 1-9.
Nathan, M. J. (1998). Knowledge and situational feedback in a learning environment for algebra story
problem solving. Interactive Learning Environments, 5(1), 135-159.
Roll, I., Aleven, V., & Koedinger, K. R. (2010). The invention lab: Using a hybrid of model tracing and
constraint-based modeling to offer intelligent support in inquiry environments. In V. Aleven, J.
Kay, & J. Mostow (Eds.), Proceedings of the International Conference on Intelligent tutoring
systems (pp. 115-124). Springer Verlag.
Roll, I., Aleven, V., & Koedinger, K. (2011). Outcomes and mechanisms of transfer in invention activities.
Proceedings of the Annual Meeting of the Cognitive Science Society, 33. Retrieved from
https://escholarship.org/uc/item/4rx036z0
Roll, I., Holmes, N. G., Day, J., & Bonn, D. A. (2012). Evaluating metacognitive scaffolding in Guided
Invention Activities. Instructional Science, 40(4), 691–710. https://doi.org/10.1007/s11251-012-
9208-7
Schmidt, R. A., & Bjork, R. A. (1992). New conceptualizations of practice: Common principles in three
paradigms suggest new concepts for training. Psychological Science, 3(4), 207-217.
http://dx.doi.org/10.1111/j.1467-9280.1992.tb00029.x
Schwartz, D. L., & Bransford, J. D. (1998). A time for telling. Cognition and Instruction, 16(4), 475-522.
Schwartz, D. L., Chase, C. C., Oppezzo, M. A., & Chin, D. B. (2011). Practicing versus inventing with
contrasting cases: The effects of telling first on learning and transfer. Journal of Educational
Psychology, 103(4), 759–775. https://doi.org/10.1037/a0025140
Schwartz, D. L., & Martin, T. (2004). Inventing to prepare for future learning: The hidden efficiency of
encouraging original student production in statistics instruction. Cognition and Instruction,
22(2), 129-184. https://doi.org/10.1207/s1532690xci2202_1
Shalala, R., Amir, O., & Roll, I. (2021). Towards asynchronous data science invention activities at scale. In
Hmelo-Silver, C. E., De Wever, B., & Oshima, J. (Eds.), Proceedings of the 14th International
Conference on Computer-Supported Collaborative Learning - CSCL 2021 (pp. 43-50).
International Society of the Learning Sciences.
205
Sinha, T., & Kapur, M. (2021). When problem solving followed by instruction works: Evidence for
productive failure. Review of Educational Research, 91(5), 761–798.
Sweller, J. (2009). What human cognitive architecture tells us about constructivism. In S. Tobias & T. M.
Duffy (Eds.), Constructivist instruction: Success or failure? (pp. 127–143). Routledge/Taylor &
Francis Group.
Tobias, S., & Duffy, T. M. (Eds.). (2009). Constructivist instruction: Success or failure? Routledge/Taylor &
Francis Group.
206
In Their Own Words Using Worked Examples for Ill-Structured
Learning Content
to Apply the Science of Learning in Alexander Renkl
University of Freiburg
Editors
Lauren L. Kordonowy
Victor A. Benassi
Worked Examples for Well-Structured and for Ill-Structured Learning

Content
Worked examples show how a problem is solved. In many cases, they contain both the problem
formulation and the final solution as well as the solution steps leading to the final solution. It is probably
fair so say that most readers' first thoughts when reading about worked examples are associated with
learning content from mathematics or physics. A typical worked example from such learning content
demonstrates a mathematical solution algorithm resulting in the final solution. Actually, the worked
example effect (i.e., the superior initial acquisition of cognitive skills when studying examples as
compared to solving problems) was first established with such well-structured learning content (Cooper
& Sweller, 1987; Sweller & Cooper, 1985; Tarmizi & Sweller, 1988). Figure 1 illustrates a worked example
from well-structured learning content (Berthold & Renkl, 2009).
Figure 1
A Worked Example for Well-Structured Content (Probability)
207
Note. From "Instructional aids to support a conceptual understanding of multiple representations," by K.
Berthold and A. Renkl, 2009, Journal of Educational Psychology, 101(1), p. 71
(http://dx.doi.org/10.1037/a0013247 ). Copyright by the American Psychological Association. Reprinted
with permission.
As the worked example effect turned out to be robust for learning to solve problems with algorithmic
solution procedures (e.g., Hattie, 2009; Renkl, 2014; Sweller et al., 2019), researchers tested whether
worked examples can also be used to foster initial skill acquisition when the learning content is ill-
structured.1 Ill-structured means in this case that a problem cannot be solved by an algorithmic solution
procedure that, if applied correctly, leads automatically to the correct final solution. Instead, principles
have to be applied that are rules of thumb (i.e., heuristics). Such heuristics help us find a solution but do
not guarantee success. For example, certain moves in interdisciplinary cooperation are helpful for
solving a medical-psychological diagnostic problem (e.g., Rummel et al., 2009), or certain moves help to
develop a proof in mathematics (Hilbert et al., 2008). These moves, however, do not guarantee success.
Figure 2 shows an excerpt from a worked example for learning to argue—another ill-structured learning
content (Hefter et al., 2019).
Figure 2
Excerpt From an Argumentation Worked Example With Two Persons Discussing the Resettling of the Lynx
in a German Area
Note. Counter-argumentation is exemplified. Depending on the learning condition in Hefter et al. (2019),
this excerpt was part of a written text, of a graphic novel, or of a video. From "Effective and efficient
acquisition of argumentation knowledge by self-explaining examples: Videos, texts, or graphic novels?"
by M. Hefter et al., 2019, Journal of Educational Psychology, 111(8), p. 1400
(https://doi.org/10.1037/edu0000350). Copyright by the American Psychological Association. Reprinted
with Permission
1
In this chapter, well-structured learning content and ill-structured learning content are differentiated.
Generally, it is more common to differentiate between well-structured and ill-structured domains.
However, the term domain is frequently used for such areas as mathematics, physics, history, social
studies etc. Note that such domains and their respective learning content do not tend to be generally
well-structured or ill-structured. For example, most learning content from school mathematics is well-
structured. However, finding proofs is ill-structured learning content. Hence, it makes more sense to
differentiate between well-structured and ill-structure learning content in the present context.
208
Ill-structured learning contents, for which worked examples have been tried and tested, can be assigned
to four categories:
• Ill-structured contents within domains considered as (primarily) well-structured: For
example, arguing and proving in mathematics (e.g., Hilbert, Renkl, Kessler, & Reiss, 2008;
Reiss et al., 2008), hands-on experimenting in chemistry (e.g., Koenen et al., 2017), learning
by modelling in biology (e.g., Mulder et al., 2016);
• Content from ill-structured domains: For example, analyzing legal cases (e.g., Nievelstein et
al., 2013), diagnostic competence in medicine (Stark et al., 2011), essay writing in English
literature (Kyun et al., 2013), negotiating (e.g., Gentner et al., 2003), recognizing designer
styles (Rourke & Sweller, 2009);
• Cross-domain skills: For example, arguing scientifically (e.g., Hefter et al., 2014; Schworm &
Renkl 2007), interdisciplinary cooperation (e.g., Rummel et al, 2009), concept mapping as a
learning strategy (e.g., Hilbert & Renkl, 2008), writing learning journals (e.g., Hübner et al.,
2010; Roelle et al., 2012);
• Ill-structured content in teacher education: For example, teachers' decision making for
managing classrooms (Cevik & Andre, 2012), teachers' assessment of students' learning
strategies (e.g., Glogger-Frey et al., 2015), teachers' design of learning materials (Hilbert et
al., 2008), implementing instructional design theory (e.g., Hoogveld et al., 2005), noticing
and interpreting important classroom events (professional vision, e.g., Martin et al., 2020).
Theoretical Explanation of the Worked Example Effect

Why is studying worked examples more effective than solving problems in the initial acquisition of
cognitive skills (e.g., learning to solve certain types of physics problems or learning to use concept
mapping as a learning strategy)? The worked-example effect is typically explained as follows. If learners
have just begun learning about certain rules in mathematics (e.g., the multiplication rule and the
addition rule in probability), they typically have a fairly restricted understanding of these rules (often
called principles in this context). If the learners then try to solve corresponding problems, they cannot
rely on a genuine understanding of the relevant principles. Learning by problem solving is then usually
slow and error prone (Salden et al., 2010). As learners are often cognitively overwhelmed, they might
experience cognitive overload (Ashman & Sweller, this volume; Sweller et al., 2019). In their effort to
find a problem solution, the learners might rely on superficial strategies (e.g., copying the solution of an
earlier probability problem and adapting the numbers). Such strategies sometimes lead to correct
solutions, but they do not enhance the learners' understanding of the principles or of how to apply them
in problem solving (Renkl, 2014). Thus, applying such strategies can be regarded as inducing extraneous
(i.e., unproductive) cognitive load. Hence, learners should first study worked examples to reach an
understanding of underlying principles. Not until they do so should the learners solve problems to
refine and (partly) automate their skills.
This explanation does not just apply to worked examples for well-structured contents. Imagine that
learners have just heard about the principles of two-sided argumentation and should then immediately
argue in that way about a complex topic; in this case also, the learners might be overwhelmed, and
learning is slow and error prone. The same applies to using concept mapping as a learning strategy: if
learners have just heard about the principles of effective concept mapping and they are then supposed
to immediately construct a concept map about a complex topic, they might also be overwhelmed.
Hence, also in the case of ill-structured content, it makes sense that the learners first gain an
209
understanding of the principles and of how to apply them before engaging in problem solving (Renkl,
2014).
Relating Worked Examples to Principles: Self-Explanations

In the classical form of learning from worked examples, the to-be-learned principles are introduced up-
front (e.g., Renkl, 2014), for example, by a short multimedia presentation or text (about 10 min).
Sometimes, some test questions check whether the students have learned the basics about the
principles; if the learners cannot provide correct answers, they are given additional explanations to close
their knowledge gaps (e.g., Renkl, 1997). Subsequently, the learners study the worked examples, in
which applying the principles is demonstrated. The learners can then relate the worked examples or
parts thereof (e.g., certain utterances in an argument between two persons) to a principle (e.g., a
rebuttal). Such relating is usually termed self-explanation (Chi et al., 1989; Renkl & Eitel, 2019; Rittle-
Johnson et al., 2017), as the learners explain certain example features to themselves by referring to
underlying principles.
However, many learners do not spontaneously self-explain (e.g., Renkl, 1997). Hence, to fully exploit the
potential of example-based learning, it is advisable to foster the learners' self-explanations (Renkl, 2014;
2021). A tried-and-tested method for this purpose is to use self-explanation prompts, which are often
questions or instructions to interrelate principles and worked examples. Figure 3 shows an example of a
self-explanation prompt for argumentation examples (Hefter et al., 2014).
Figure 3
Self-Explanation Prompt for Video-Based Argumentation Examples
Note. From "Effects of a training intervention to foster argumentation skills while processing conflicting
scientific positions," by M. Hefter et al., 2014, Instructional Science, 42(6), p. 936
(https://doi.org/10.1007/s11251-014-9320-y). Copyright by Springer Nature. Reprinted with Permission.
210
Important Specifics of Worked Examples for Ill-Structured Content
There are two typical characteristics of worked examples for ill-structured content that differ from
examples for well-structured content. These characteristics should be considered when implementing
learning from examples for ill-structured content.
Two Content Levels
In the vast majority of cases, worked examples for well-structured content have only one content level,
which is related to the structural domain features (e.g., probability: multiplication rule and addition
rule). The problem formulations, for example, in mathematics or physics word problems typically consist
of simple cover stories that correspond to the examples' surface features. In the case of worked
examples for ill-structured content, there are typically two "substantial" content levels that learners
have to process. For example, when students learn about two-sided argumentation, they are given
examples showing argumentation about a complex and controversial topic such as global warming and
forest dieback (e.g., Hefter et al., 2014). Another example is illustrated by student teachers learning how
to notice and interpret classroom events such as student-centered teacher behaviors in a classroom
video; the instruction presented in the video is about learning content, as specified in the curriculum,
such as the human circulatory system (Martin et al., 2020). Renkl et al. (2009) differentiated these levels
into the learning domain (e.g., how to structure an argument; how to notice and interpret certain types
of teacher behaviors) and the exemplifying domain (i.e., global warming and forest dieback; human
circulatory system). The learners have to integrate information from both content levels. For example,
they will not notice that a teacher in a video example is reacting to a student's misconception if they do
not closely follow the conversation about the human circulatory system.
This distinction between content levels loosely relates to the differentiation between structural features
(i.e., solution-relevant features) and surface features (e.g., cover story, objects, numbers) typically used
in research on worked examples for well-structured content. For example, in the case of mathematical
word problems the surface features correspond to a “lean description of some situation” (Greer, 1997,
p. 297). Although, some learners might also have problems understanding the description of a relatively
simple situation, the difficulty in understanding exemplifying domains in the case of worked examples
for ill-structures content is usually much greater, and learners also need to understand the exemplifying
domain as just illustrated by the example case of a teacher reacting to a student's misconception. In
other words, comprehension difficulties in the exemplifying domain can hinder learners from seeing
how the learning domain's principles are instantiated.
The need to process two complex content levels has two important implications when designing worked
examples for ill-structured content. First, the exemplifying domain should not be difficult, otherwise
some learners might be overwhelmed by the demand to process both content levels. In fact, learners
with low prior knowledge in the exemplifying domain were found to obtain poor learning outcomes in
the learning domain (Hilbert et al., 2004; Hilbert, Renkl, Kessler, & Reiss, 2008). If, for some reason, a
difficult and challenging exemplifying domain is chosen, pre-training those contents might be advisable,
that is, teaching the exemplifying domain's basics before providing the worked examples to teach the
learning domain (e.g., Clarke et al., 2005). Second, the processing support should focus the learners'
attention primarily on the learning domain, not on the exemplifying domain. For example, when
learning about two-sided argumentation, prompts directing the learners' attention primarily to the
exemplifying domain (e.g., climate change and forest dieback) hamper knowledge acquisition in the
learning domain (e.g., Schworm & Renkl, 2007). Similarly, requiring learners to fill-in gaps in geometric
expressions (exemplifying domain) was detrimental for learning to prove (learning domain; Hilbert et al.,
2008).
211
Complex Skills as Learning Goals
Typical worked examples from mathematics teach circumscribed domain-specific skills, for example,
learning to apply the complementary rule, the multiplication rule, and the addition rule to probability
problems (e.g., Atkinson et al., 2003) or determining angles in triangle geometry (Ginns et al., 2016).
Worked examples for ill-structured learning content usually teach more complex skills that develop over
longer periods such as learning to prove in mathematics (e.g., Reiss et al., 2008), learning to argue in a
two-sided way (e.g., Hefter et al., 2014; Schworm & Renkl, 2007), or analyzing legal cases (e.g.,
Nievelstein et al., 2013). Hence, the example-based lessons tried and tested in corresponding studies
could in most cases "just" be regarded as kick-off interventions on the way to acquiring such complex
skills. Hence, they show practitioners how a good starting point might look. However, the skills will have
to be further expanded upon in subsequent lessons.
Three Illustrative Applications of Worked Examples for Ill-Structured

Learning Content
To flesh out the preceding discussion of learning ill-structured contents from worked examples, three
rather heterogeneous cases will be presented in some depth: Two-sided argumentation (cross-domain
skill), mathematical proof and argumentation (ill-structured content from a domain usually regarded as
well-structured), and teachers' professional vision (teacher education). These cases are discussed in a
parallel form to clarify how the key elements of learning ill-structured content from worked examples
appeared in the corresponding studies.
Two-Sided Argumentation
Berthold, Hefter, Renkl and colleagues conducted several studies in which video-based worked-
examples were used to foster two-sided argumentation (Hefter et al., 2014, 2018, 2019; Schworm &
Renkl, 2007). The students should learn not only to consider one perspective when reflecting about
complex issues such as climate change or gene-modified food, but also to ponder the pros and cons of
different perspectives.
Learning Domain
The goal was to foster the skills of high-school students and, in one study of student teachers, in two-
sided argumentation as a key to developing deep understanding of complex topics, in particular, in
science (e.g., Kuhn, 2001, 2010). The authors largely followed the argumentation model by Kuhn (1991).
The first (heuristic) step in arguing well is to support one's own position (e.g., perspective or theory)
with strong arguments (e.g., by referring to scientific evidence rather than to an example from one’s
own life). The second step comes up with an alternative theory that is also supported by some evidence.
The third step generates counterarguments against one’s own position. The fourth step is a synthesis
based on carefully evaluating the strengths of all the arguments and counterarguments stated (which
somewhat deviates from Kuhn's model; e.g., Kuhn, 1991). In some studies (e.g., Hefter et al., 2019), six
(heuristic) steps were differentiated (i.e., theory, genuine evidence, alternative theory,
counterargument, rebuttal, and synthesis). Note that the focus was not on fostering argumentation
when debating with others, but rather on argumentative thinking (i.e., when thinking about complex
topics: supporting one's own position, considering an alternative position or theory, etc.).
Fostering such a cross-domain skill is a demanding challenge for educators, and it must be pursued as a
long term-goal. Such a skill is, as already mentioned, important for learning about complex topics (e.g.,
Iordanou et al., 2019). Beyond this aspect, such skills have probably become even more necessary in
recent years, as we seem to have entered a post-truth era in which there are demands for educational
212
responses to equip students with the skills to critically evaluate misinformation, "alternative truths," or
conspiracy theories (e.g., Barzilai & Chinn, 2020).
Exemplifying Domains
Berthold, Hefter, Renkl and colleagues have used various exemplifying domains in which the worked
examples revealed two-sided argumentation: stem-cell research, achievement differences between girls
and boys in mathematics and science (Schworm & Renkl, 2007), resettling the lynx in a specific forest
area, or global warming and forest dieback (Hefter et al., 2014, 2018, 2019).
Introducing the Principles
The four or six argumentation steps described above were the principles to be exemplified in the videos.
They were introduced by a short text-based presentation on a computer monitor (e.g., Hefter et al.,
2014, 2019). To introduce the steps, an initial example was already used (e.g., the extinction of the
dinosaurs).
The Worked Examples
The worked examples consisted usually of videos showing two students arguing about complex topics,
such as resettling the lynx in a specific forest area or global warming and forest dieback (Hefter et al.,
2014, 2018). The two students held conflicting positions (for an excerpt, see Figure 2). Hefter et al.
(2019) also successfully applied text-based worked examples (see Figure 2) as well as graphic-novel
versions of the videos. That is, screenshots from the videos were taken and bubbles with the text were
added (see Figure 4).
Figure 4
Screenshot from a Graphic Novel Version of a Worked Examples on Argumentation
213
Note. From "Effective and efficient acquisition of argumentation knowledge by self-explaining examples:
Videos, texts, or graphic novels?" by M. Hefter et al., 2019, Journal of Educational Psychology, 111(8), p.
1401 (https://doi.org/10.1037/edu0000350). Copyright by the American Psychological Association.
Reprinted with permission.
Eliciting Self-Explanations
Self-explanation prompts were employed to foster the processing of the worked examples in terms of
the to-be-learned argumentation steps (Schworm & Renkl, 2007; Hefter et al., 2014, 2018, 2019). Such a
prompt is illustrated in Figure 3. The prompt supports the learners to attend not merely to the
exemplifying domain (e.g., resettling the lynx), which may be more salient when listening to dialogs, but
primarily to the underlying argumentative structure.
Further Practical Considerations
Video-based worked-out examples might appear more authentic at first glance. However, Hefter et al.
(2019) detected no advantage for such examples in terms of learning outcomes, as compared to
examples in a text format or a graphic-novel format. Rather, learners required less learning time from
the written and from the graphic-novel formats, meaning that such examples were more efficient than
video examples. Another advantage is that the graphic-novel format and the written format in particular
are easier to produce.
Mathematical Argumentation and Proof
Reiss, Renkl and colleagues (e.g., Reiss & Renkl 2002; Hilbert, Renkl, Kessler, & Reiss, 2008; Reiss et al.,
2008) developed worked examples for teaching the skills of mathematical argumentation to high-school
(or first-year university) students as a prerequisite for finding proofs.
Learning Domain
The goal was to foster students' skills of mathematical argumentation as a basis for later proof finding.
More specifically, the students learned about a sequence of heuristic steps for proof finding, which were
taken from Boero (1999) in a slightly modified version. The students learned the following sequence:
exploring a problem situation to formulate an initial conjecture (production of a conjecture), formulating
the initial conjecture as a clear statement (formulation of a statement), exploring their conjecture by
identifying appropriate arguments (exploration of the conjecture), and finally selecting and combining
arguments in a coherent deductive chain (selection and combination of coherent arguments in a
deductive chain).
As proof is central to mathematics, it is a central goal of mathematics education in schools, namely that
students develop at least some basic understanding of the concept of proof as well as basic skills in
handling proofs (e.g., constructing, comprehending, validating; Sporn et al., 2021). However, many
students have fairly little understanding of proofs and deficient skills at the end of high school and when
entering university education (e.g., Sporn et al., 2021). Hence, it is important and demanding to foster
students' proving skills and their basis in mathematical argumentation.
Exemplifying Domain
The exemplifying domain was geometry. For example, students had to prove that opposing sides and
opposing angles in a parallelogram have the same size.
214
The details of the introduction differed somewhat between studies. In a classroom study with 8th
graders (Reiss et al., 2008), teachers provided their regular introductory instruction ("business as usual")
before the students were given worked examples.
The Worked Examples
The worked examples were substantially longer than typical worked examples (i.e., 10 or even more
pages). The examples included two or even three fictitious students who went through the four phases
of a mathematical argumentation process (Boero, 1999). The fictitious students discussed their ideas
and always spoke their thoughts out loud, which can be considered as a type of cognitive model. A
reason to use student models was their similarity to the high-school students, which helps the learners
to identify with fictitious students, thus enhancing their self-efficacy (e.g., Bandura, 1986, 1971; for an
example see the Appendix of Reiss et al., 2008). Such an example began as follows:
The Problem:
Nina and Tom have drawn and measured parallelograms. In doing so, they noticed that opposing
sides were always of equal length. Moreover, opposing angles were always of equal size.
Tom: ‘‘We measured so many parallelograms: We have drawn all kinds of quadrangles, and
always we recognized that the opposing sides were of equal length and opposing angles were of
equal size. I think, it has to be like this!’’
Nina: ‘‘I think you are right, but I don’t know a reason. Maybe by chance, we have only drawn
parallelograms for which the statement is correct? We cannot measure the angles and sides
exactly. Perhaps they were only approximately of the same size.’’
Tom: ‘‘So let’s try to prove our assumption like mathematicians would do!’’
Tom and Nina try to prove the following mathematical proposition:
‘‘In a parallelogram opposing sides are of equal length and opposing angles are commensurate!’’
In the following we have a look at how they solved the mathematical problem. Please read their
solution …" (Reiss et al., 2008, pp. 463-464)
Various studies on such worked examples employed somewhat different self-explanation prompts.
Hilbert, et al. (2008) used prompts that were most directly related to the sequence of heuristics to be
learned, and their prompts proved to be effective in an experimental test. When going through the
worked examples, the learners answered four two-fold self-explanation prompts. First, the learners
were asked in a multiple-choice format in which phase the two fictitious students had been in the
preceding section: (1) production of a conjecture, (2) formulation of a statement, (3) exploration of the
conjecture, and (4) selection and combination of coherent arguments in a deductive chain. Second, the
learners had to write a brief justification for their choice.
A measure to deepen the active processing of worked examples used in part of the studies (e.g., Reiss et
al., 2008) was inserted gaps in the geometry information that the learners had to filled in (completion
requirements). However, Hilbert et al. (2008) found that such gaps, especially when combined with self-
explanation prompts, have detrimental effects. Obviously, they direct the learners’ attention away from
the learning domain of proving to the exemplifying domain of geometry. Such gaps should therefore be
avoided. Recent studies combined heuristic examples with (scripted or unscripted) cooperation (e.g.,
Kollar et al., 2014; Schwaighofer et al., 2017). However, it is not yet clear that studying worked examples
215
on mathematical argumentation cooperatively rather than individually is more effective. Hence, an
individual learning approach might be advisable in this context, as it is easier to implement.
Student Teachers' Professional Vision
Farrell, Martin, and colleagues (Farrell et al., 2022; Martin et al., 2022) investigated the effects of an
example-based 90-minute seminar session as a kick-off intervention to foster student teachers' skills in
noticing and interpreting important classroom events in terms of theoretical concepts (professional
vision). More specifically, the student teachers should learn to notice and interpret example cases of
teacher-centered or student-centered tutoring behaviors and corresponding student reactions.
Learning Domain
The goal was to foster student teachers' professional visions skills in the area of tutoring situations (e.g.,
teachers working with a group of four students during a phase of group work). Professional vision (skills)
means that the student teachers notice relevant events, here in videos, and that they interpret these
events by relying on their theoretical knowledge (Santagata et al., 2007; van Es & Sherin, 2002). More
specifically, the student teachers should learn to notice and interpret teacher-centered tutoring
behaviors and student-centered tutoring behaviors, as well as the corresponding student reactions. The
student teachers should also learn that one of the major advantages of tutoring situations, that is, the
possibility to tailor instruction to students' specific learning prerequisites, is more fully exploited if the
teachers primarily show student-centered behaviors (e.g., having the students explain their potentially
misconceived concepts instead of simply teaching the correct concept).
Teaching such skills is nothing trivial. There is ample evidence that student teachers seldom interpret
teacher and student behaviors on the basis of theory. Instead, they often judge prematurely whether
the noticed events were "good" or "bad" without substantive justification (e.g., van Es & Sherin, 2002;
Seidel & Stürmer, 2014), and they oversimplify learning and teaching issues (Jacobs et al., 2010). The
development of sophisticated professional vision skills is a major challenge in teacher education (e.g.,
Stürmer et al., 2016).
Exemplifying Domain
The exemplifying domain was instruction on the human circulatory system, which is part of the biology
curriculum at the secondary level. Note that high-school students – also those depicted in the video-
based worked examples – have several misconceptions regarding the circulatory system. For example,
many students think of the circulatory system as a single-loop instead of a double-loop system.
The principles exemplified in the videos were explained in introductory texts (e.g., Martin et al., 2020).
Depending on the condition, there was an introductory text (about 1000 words) with a biology-
education perspective or with an educational psychology perspective on tutoring situations. A control
group was given "just" a text with general guidelines for observing classroom videos. We instructed the
participants to read the introductory text carefully and to remember the information for its subsequent
application in a video analysis.
The Worked Examples
The worked examples were presented in videos showing teacher-centered and student-centered
tutoring behaviors as well the corresponding student reactions. The videos lasted 7 to 10 minutes and
showed, for example, a discussion about differences between the students’ drawings of the circulatory
system and a scientific diagram. The videos were staged (scripted) and enacted by biology teachers and
216
four students who had volunteered to make such videos recordings. However, the individual tutoring
behaviors and student reactions were taken from authentic videos. The staging of videos made them
more condensed in the sense that the observing student teachers could see relatively many teacher-
centered and students-centered behaviors in a short time period (note that the videos contained both
types of behaviors, as did the authentic videos). For learning, the student teachers watched two video
scenarios.
The student teachers marked events they considered relevant for teaching and learning by pressing a
button on the keyboard (noticing). After each video, they commented on some of their marked events
(interpreting); this activity is regarded as self-explanation. In addition, the student teachers rated the
videos on professional-vision rating scales, for example, the extent to which tutors had elicited students'
misconceptions.
Martin et al. (2022) further supported self-explanations by providing a figure displaying important
concepts from the introductory text such as eliciting students' misconceptions (see Figure 5). After
watching a video or, in the case of segmented videos, after each video segment, the student teachers
could click on the concepts perceived as relevant to the previous video, and they could continue to
interpret the relevant events
Figure 5
Pre-Structured Self-Explanation Prompts Presented Within an Interactive Figure Displaying the Principles
Outlined in the Introductory Text
Note. Used in the study by Martin et al. (2022).

In Martin et al. (2022), the videos were cut into smaller segments lasting 3 to 4 minutes. Note that video
provides transient information so that student teachers can easily overlook important information when
217
distracted by irrelevant information (e.g., the teacher has a peculiar accent) or when sticking to relevant
information (e.g., thinking a bit longer about an important event). Segmenting can reduce the problem
of transient information. Noticing and interpreting worked best with segmented videos (in comparison
to unsegmented videos), especially when combined with pre-structured self-explanation prompts. A
limitation here was, however, that no transfer of this good performance occurred to new videos without
segmenting and self-explanation prompts. Longer training with segmented video and self-explanation
prompts is probably necessary (note: there was just one seminar session in the studies of Martin and
colleagues).
Guidelines for Implementing Learning From Worked Examples for Ill-

Structured Contents
Writing a comprehensive guide to implementing learning from worked examples for ill-structured
contents would take a whole book. The goal of this section is to provide some guidelines as a starter. If
practitioners (i.e., teachers or instructional designers) aim to dive deeper into this topic, they might
want to read additional reviews (e.g., Renkl, 2014, 2021; Renkl et al., 2009; Mayer, 2021; van Gog et al.,
2019) and original studies for models (see the cited studies in this chapter). In addition, the scientific
literature does not usually answer all the detailed questions that arise when implementing an
instructional model in practice (e.g., Biesta, 2007). Hence, practitioners should also rely on other
knowledge resources such as their experiential knowledge about what works in their specific context
(i.e., subject matter, grade level, school culture, their students' achievement level, classroom
composition, etc.) when implementing learning from worked examples.
Suggested Steps to Designing Learning From Worked Examples for Ill-Structured
Content:
1. Identify the To-Be-Learned Principles: The principle(s) to-be-taught should be determined. They
may be heuristic steps in solving an ill-defined problem or important features a good case
should have (e.g., a good concept map). These principles might be taken from theoretical
models (e.g., argumentation model), from empirical research (e.g., findings on effective concept
mapping), or the heuristics common to or recommended in certain subjects (e.g., steps for
solving mathematical word problems). This step is essential because it defines the learning
domain and, thereby, determines what the worked examples should primarily demonstrate.
2. Design a Short Presentation of the Principles: The goal of this short presentation (e.g., a text, a
multimedia presentation, or a classroom presentation) is not that the learners acquire full
understanding of the principles and their application—that goal should be pursued later during
example study. However, the point of the goal is that learners should acquire factual knowledge
of the principles and some rudimentary understanding. For this purpose, an initial easy example
can be included in this presentation.
As the learners should be able to recall the principles when later studying worked examples, the
practitioner might also pose questions to check whether the learners can remember the
principles. If there are knowledge gaps, re-teaching should close such gaps.
3. Design Worked Examples: As learning from worked examples involves the use of multiple
examples, several examples need to be constructed that clearly represent cases of the
application of the to-be-learned principles (learning domain). It might not be necessary to
construct fancy worked examples, for example, in form of high-quality videos (e.g., with
professional speakers). In most cases, written worked examples may suffice or even be
preferable (e.g., Hefter et al., 2019), though they might not look very inviting at first glance.
218
Note that videos also have disadvantages such as delivering transient information. Of course,
there might be good reasons to use video in some cases (e.g., presenting authentic classroom
situations to student teachers).
It is advisable to use exemplifying domains which are relatively easy to understand. Otherwise, a
pre-training session on this content will be necessary. Thereby, it is avoided that the learners
must devote so much of their cognitive capacity when trying to understand the exemplifying
domain that they become distracted from the learning domain. Another potential pitfall is to
select an exemplifying domain that is a very hot topic (e.g., mandatory COVID vaccination). Such
a topic might engage the learners both cognitively and emotionally to a dysfunctional degree. In
such a case, the exemplifying domain might also distract from the learning domain.
4. Add Self-Explanation Prompts to the Worked Examples: To fully exploit the potential of
learning from worked examples and to keep learners from primarily focusing on the
exemplifying domain, self-explanation prompts can be added. These prompts ask learners to
relate example features to the underlying principles. Self-explanation prompts vary on a
dimension from open (e.g., "Explain which principle applies here") to pre-structured (e.g.,
"Select from the menu the principle that applies here"; a menu lists potentially relevant
principles; see also Figure 5 for a pre-structured prompt). Particularly in the first worked
examples, it is probably better to use pre-structured prompts (e.g., Renkl & Eitel, 2019; Rittle-
Johnson et al., 2017); later on, more open prompts might be employed.
5. Consider Multimedia Principles: If multimedia design principles (e.g., Mayer, 2021; Mayer, this
volume) are violated, learning from worked examples might be less effective or even lose
entirely its advantage over learning by problem solving (e.g., Tarmizi & Sweller, 1988; Renkl,
2021). For example, if multiple representations such as text and diagrams are used (e.g., worked
examples for learning concept mapping contain written explanations and well-designed concept
maps) or if fleeting information is presented (e.g., in videos or animations), the processing
demands might overwhelm at least some learners. Such learners need support or materials that
are easier to process (e.g., text instead of video). It is beyond the scope of this chapter to discuss
multimedia design principles; the interested reader might refer to Clark and Mayer (2016),
Mayer (2021; this volume), or Renkl (2021).
6. Continue With the Learning Domain: As already mentioned, learners studying worked examples
for ill-structured contents are often expected to acquire complex and difficult skills. Hence,
providing an initial unit with an introduction to the principles and a set of corresponding worked
examples might be an excellent kick-off intervention on the way to acquiring such complex
skills—but it is just a start. It is necessary that practitioners continue to promote such skills in
subsequent instruction, initially via additional worked examples and later also by giving their
learners growing opportunities to practice their skills (e.g., Renkl, 2014; 2021).
A Final Word
To those who give the aforementioned approach a try: Considering my own experience as a university
teacher, the very first attempts to implement an instructional idea are rarely entirely satisfying.
However, they are an excellent starting point to determine what actually works and what needs to be
improved. Good luck with your first attempt and for fine-tuning your approach.
219
Author Note
Correspondence should be addressed to Alexander Renkl, Department of Psychology, University of
Freiburg, 79085 Freiburg, Germany; email: [email protected]
References
Atkinson, R. K., Renkl, A., & Merrill, M. M. (2003). Transitioning from studying examples to solving
problems: Combining fading with prompting fosters learning. Journal of Educational Psychology,
Bandura, A. (1971). Psychological modeling: Conflicting theories. Aldine Atherton.
Bandura, A. (1986). Social foundations of thought and action: A social cognitive theory. Prentice Hall.
Berthold, K., & Renkl, A. (2009). Instructional aids to support a conceptual understanding of multiple
representations. Journal of Educational Psychology, 101(1), 70-87.
http://dx.doi.org/10.1037/a0013247
Biesta, G. (2007). Why "what works" won't work: Evidence‐based practice and the democratic deficit in
educational research. Educational theory, 57(1), 1-22. https://doi.org/10.1111/j.1741-
5446.2006.00241.x
Barzilai, S., & Chinn, C. A. (2020). A review of educational responses to the “post-truth” condition: Four
lenses on “post-truth” problems. Educational Psychologist, 55(3), 107-119.
Boero, P. (1999). Argumentation and mathematical proof: A complex, productive, unavoidable
relationship in mathematics and mathematics education. International Newsletter on the
Teaching and Learning of Mathematical Proof, 7/8.
Cevik, Y. D., & Andre, T. (2012). Worked examples leads to better performance in analyzing and solving
real-life decision cases. Journal of Educational Technology, 9(2), 17-30.
https://doi.org/10.26634/jet.9.2.1945
Chi, M. T. H., Bassok, M., Lewis, M. W., Reimann, P., & Glaser, R. (1989). Self-explanations: How students
study and use examples in learning to solve problems. Cognitive Science, 13(2), 145-182.
Clark, R. C., & Mayer, R. E. (2016). E-Learning and the science of instruction: Proven guidelines for
consumers and designers of multimedia learning (4th ed.). Wiley
Clarke, T., Ayres, P., & Sweller, J. (2005). The impact of sequencing and prior knowledge on learning
mathematics through spreadsheet applications. Educational Technology Research and
Development, 53(3), 15-24. https://doi.org/10.1007/BF02504794
Cooper, G., & Sweller, J. (1987). Effects of schema acquisition and rule automation on mathematical
problem-solving transfer. Journal of Educational Psychology, 79(4), 347-362.
Farrell, M., Martin, M., Renkl, A., Rieß, W., Könings, K. D., van Merriënboer, J. J. G., & Seidel, T. (2022).
An epistemic network approach to teacher students' professional vision in tutoring video
analysis. Frontiers in Education, 7, 805422. https://doi.org/10.3389/feduc.2022.805422
220
Gentner, D., Loewenstein, J., & Thompson, L. (2003). Learning and transfer: A general role for analogical
encoding. Journal of Educational Psychology, 95(2), 393-408. https://doi.org/10.1037/0022-
0663.95.2.393
Ginns, P., Hu, F. T., Byrne, E., & Bobis, J. (2016). Learning by tracing worked examples. Applied Cognitive
Psychology, 30(2), 160-169. https://doi.org/10.1002/acp.3171
Glogger-Frey, I., Fleischer, C., Grüny, L., Kappich, J., & Renkl, A. (2015). Inventing a solution and studying
a worked solution prepare differently for learning from direct instruction. Learning &
Greer, B. (1997). Modeling reality in mathematics classrooms: The case of word problems. Learning &
Instruction, 7(4), 293-307. https://doi.org/10.1016/S0959-4752(97)00006-6
Hattie, J. A. C. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement.
Routledge.
Hefter, M. H., Berthold, K., Renkl, A., Riess, W., Schmid, S., & Fries, S. (2014). Effects of a training
intervention to foster argumentation skills while processing conflicting scientific positions.
Instructional Science, 42(6), 929-947. https://doi.org/10.1007/s11251-014-9320-y
Hefter, M. H., Renkl, A., Riess, W., Schmid, S., Fries, S., & Berthold, K. (2018). Training interventions to
foster skill and will of argumentative thinking. Journal of Experimental Education, 86(3), 325-
Hefter, M., ten Hagen, I., Krense, C., Berthold, K., & Renkl, A. (2019). Effective and efficient acquisition of
argumentation knowledge by self-explaining examples: Videos, texts, or graphic novels? Journal
of Educational Psychology, 111(8), 1396-1405. https://doi.org/10.1037/edu0000350
Hilbert, T. S., & Renkl, A. (2008). Concept mapping as a follow-up strategy to learning from texts: What
characterizes good and poor mappers? Instructional Science, 36(1), 53-73.
Hilbert, T. S., Renkl, A., Kessler, S., & Reiss, K. (2008). Learning to prove in geometry: Learning from
heuristic examples and how it can be supported. Learning & Instruction, 18(1), 54-65.
Hilbert, T. S., Renkl, A., Schworm, S., Kessler, S., & Reiss, K. (2008). Learning to teach with worked-out
examples: A computer-based learning environment for teachers. Journal of Computer-Assisted
Learning, 24(4), 316-332. https://doi.org/10.1111/j.1365-2729.2007.00266.x
Hilbert, T. S., Schworm, S., & Renkl, A. (2004). Learning from worked-out examples: The transition from
instructional explanations to self-explanation prompts. In P. Gerjets, J. Elen, R. Joiner, & P.
Kirschner (Eds.), Instructional design for effective and enjoyable computer-supported learning
(pp. 184-192). Knowledge Media Research Center.
Hoogveld, A. W. M., Paas, F., Jochems, W. M. G. (2005). Training higher education teachers for
instructional design of competency-based education: Product-oriented versus process-oriented
worked examples. Teaching and Teacher Education, 21(3), 287-297.
https://doi.org/10.1016/j.tate.2005.01.002
Hübner, S., & Nückles, M., & Renkl, A. (2010). Writing learning journals: Instructional support to
overcome learning-strategy deficits. Learning & Instruction, 20(1), 18-29.
221
Iordanou, K., Kuhn, D., Matos, F., Shi, Y., & Hemberger, L. (2019). Learning by arguing. Learning &
Instruction, 63, 101207. https://doi.org/10.1016/j.learninstruc.2019.05.004
Jacobs, V. R., Lamb, L. L., & Philipp, R. A. (2010). Professional noticing of children's mathematical
thinking. Journal for Research in Mathematics Education, 41(2), 169-202.
https://doi.org/10.5951/jresematheduc.41.2.0169
Koenen, J., Emden, M., & Sumfleth, E. (2017). Naturwissenschaftlich-experimentelles Arbeiten:
Potenziale des Lernens mit Lösungsbeispielen und Experimentierboxen [Working experimentally
in science: The potentials of learning from worked examples and experiment kits]. Zeitschrift für
Didaktik der Naturwissenschaften, 23(1), 81-98. https://doi.org/10.1007/s40573-017-0056-5
Kollar, I., Ufer, S., Reichersdorfer, E., Vogel, F., Fischer, F., & Reiss, K. (2014). Effects of collaboration
scripts and heuristic worked examples on the acquisition of mathematical argumentation skills
of teacher students with different levels of prior achievement. Learning & Instruction, 32, 22-36.
Kuhn, D. (1991). The skills of argument. Cambridge University Press.
https://doi.org/10.1017/CBO9780511571350
Kuhn, D. (2001). How do people know? Psychological Science, 12(1), 1-8. https://doi.org/10.1111/1467-
9280.00302
Kuhn, D. (2010). Teaching and learning science as argument. Science Education, 94(5), 810-824.
https://doi.org/10.1002/sce.20395
Kyun, S., Kalyuga, S., & Sweller, J. (2013). The effect of worked examples when learning to write essays
in English literature. The Journal of Experimental Education, 81(3), 385-408.
Martin, M., Farrell, M., Seidel, T. Rieß, W., Könings, K. D., van Merriënboer, J. J. G., & Renkl, A. (2020).
Knowing what matters: Short introductory texts support pre-service teachers' professional vision
of tutoring interactions [Manuscript submitted for publication]. Department of Psychology,
University of Freiburg (Germany).
Martin, M., Farrell, M., Seidel, T., Rieß, W., Könings, K. D., van Merriënboer, J. J. G., & Renkl, A. (2022).
Focused self-explanation prompts and segmenting foster pre-service teachers’ professional
vision – but only during training! International Journal of Educational Technology in Higher
Education, 19(34). https://doi.org/10.1186/s41239-022-00331-z
Mayer, R. E. (2021). Multimedia learning (3rd ed.). Cambridge University Press.
Mulder, Y. G., Bollen, L., de Jong, T., & Lazonder, A. W. (2016). Scaffolding learning by modelling: The
effects of partially worked‐out models. Journal of Research in Science Teaching, 53(3), 502-523.
https://doi.org/10.1002/tea.21260
Nievelstein, F., van Gog, T., van Dijck, G., & Boshuizen, H. P. A. (2013). The worked example and
expertise reversal effect in less structured tasks: Learning to reason about legal cases.
Reiss, K., Heinze, A., Renkl, A., & Groß, C. (2008). Reasoning and proof in geometry: Effects of a learning
environment based on heuristic worked-out examples. ZDM Mathematics Education, 40(3), 455-
222
Reiss, K., & Renkl, A. (2002). Learning to prove: The idea of heuristic examples. ZDM Mathematics
Education, 34(1), 29-35 https://doi.org/10.1007/BF02655690
Renkl, A. (1997). Learning from worked-out examples: A study on individual differences. Cognitive
Science, 21(1), 1-29. https://doi.org/10.1207/s15516709cog2101_1
Renkl, A. (2014). Toward an instructionally-oriented theory of example-based learning. Cognitive
Science, 38(1), 1-37. https://doi.org/10.1111/cogs.12086
Renkl, A. (2021). The worked example principle in multimedia learning. In R. E. Mayer, & L. Fiorella
(Eds.), The Cambridge handbook of multimedia learning (3rd ed., pp. 231-240). Cambridge
University Press. https://doi.org/10.1017/9781108894333.023
Renkl, A., & Eitel, A. (2019). Self-explaining: Learning about principles and their application. In J.
Dunlosky & K. Rawson (Eds.), Cambridge handbook of cognition and education (pp. 528-549).
Cambridge University Press. https://doi.org/10.1017/9781108235631.022
Renkl, A., Hilbert, T., & Schworm, S. (2009). Example-based learning in heuristic domains: A cognitive
load theory account. Educational Psychology Review, 21(2), 67-78.
Rittle-Johnson, B., Loehr, A. M., & Durkin, K. (2017). Promoting self-explanation to improve mathematics
learning: A meta-analysis and instructional design principles. ZDM Mathematics Education,
49(4), 599-611. https://doi.org/10.1007/s11858-017-0834-z
Roelle, J., Krüger, S., Jansen, C., & Berthold, K. (2012). The use of solved example problems for fostering
strategies of self-regulated learning in journal writing. Education Research International,
2012(Article ID 751625), 1-14. https://doi.org/10.1155/2012/751625
Rourke, A. J., & Sweller, J. (2009). The worked-example effect using ill-defined problems: Learning to
recognise designers' styles. Learning & Instruction, 19(2), 185-199.
Rummel, N., Spada, H., & Hauser, S. (2009). Learning to collaborate while being scripted or by observing
a model. International Journal of Computer-Supported Collaborative Learning, 4(1), 69-92.
Salden, R., Koedinger, K. R., Renkl, A., Aleven, V., & McLaren, B. M. (2010). Accounting for beneficial
effects of worked examples in tutored problem solving. Educational Psychology Review, 22(4),
Santagata, R., Zannoni, C., & Stigler, J. W. (2007). The role of lesson analysis in pre-service teacher
education: An empirical investigation of teacher learning from a virtual video-based field
experience. Journal of Mathematics Teacher Education, 10(2), 123-140.
Schwaighofer, M., Vogel, F., Kollar, I., Ufer, S., Strohmaier, A., Terwedow, I., Ottinger, S, Reiss, K., &
Fischer, F. (2017). How to combine collaboration scripts and heuristic worked examples to foster
mathematical argumentation-when working memory matters. International Journal of
Computer-Supported Collaborative Learning, 12(3), 281-305. https://doi.org/10.1007/s11412-
017-9260-z
Schworm, S., & Renkl, A. (2007). Learning argumentation skills through the use of prompts for self-
explaining examples. Journal of Educational Psychology, 99(2), 285-296.
http://dx.doi.org/10.1037/0022-0663.99.2.285
223
Seidel, T., & Stürmer, K. (2014). Modeling and measuring the structure of professional vision in
preservice teachers. American Educational Research Journal, 51(4), 739-771.
Sporn, F., Sommerhoff, D., & Heinze, A. (2021). Beginning university mathematics students' proof
understanding. In M. Inprasitha, N. Changsri, & N. Boonsena (Eds.), Proceedings of the 44nd
Conference of the International Group for the Psychology of Mathematics Education (vol. 4, pp.
102-110). PME.
Stark, R., Kopp, V., & Fischer, M.R. (2011). Case-based learning with worked examples in complex
domains: Two experimental studies in undergraduate medical education. Learning & Instruction,
21(1), 22-33. https://doi.org/10.1016/j.learninstruc.2009.10.001
Stürmer, K., Seidel, T., & Holzberger, D. (2016). Intra-individual differences in developing professional
vision: Preservice teachers' changes in the course of an innovative teacher education program.
Instructional Science, 44(3), 293-309. https://doi.org/10.1007/s11251-016-9373-1
Sweller, J., & Cooper, G. A. (1985). The use of worked examples as a substitute for problem solving in
learning algebra. Cognition & Instruction, 2(1), 59-89.
Sweller, J., van Merriënboer, J. J. G., & Paas, F. (2019). Cognitive architecture and instructional design:
20 years later. Educational Psychology Review, 31(2), 261-292. https://doi.org/10.1007/s10648-
019-09465-5
Tarmizi, R. A., & Sweller, J. (1988). Guidance during mathematical problem solving. Journal of
Van Es, E. A., & Sherin, M. G. (2002). Learning to notice: Scaffolding new teachers' interpretations of
classroom interactions. Journal of Technology and Teacher Education, 10(4), 571-596.
Van Gog, T., Rummel, N., & Renkl, A. (2019). Learning how to solve problems by studying examples. In J.
Dunlosky & K. Rawson (Eds.), Cambridge handbook on cognition and education (pp. 183-208).
Cambridge University Press. https://doi.org/10.1017/9781108235631.009
224
In Their Own Words Cognitive Strategies for Improving Students’
Conceptual Learning in Physics
to Apply the Science of Learning in Jennifer L. Docktor
University of Wisconsin – La Crosse
Editors
Lauren L. Kordonowy
José P. Mestre
Victor A. Benassi
University of Illinois at Urbana -Champaign
Overview
In this chapter, we discuss what novices “see” in the physics situations and scenarios they experience
while taking an introductory course and how what they see differs from what experts perceive. We do
so from studies using experimental methods common in cognitive psychology, which are novel in the
study of knowledge acquisition in a science domain. The results we will discuss reveal details that are
not evident in more traditional studies of novice behavior in the sciences. In addition, we will discuss
targeted instructional interventions based on the research findings to help physics novices learn more
productive ways of analyzing physics scenarios and solving problems.
Much has been learned since researchers started studying expert-novice differences in physics. One of
the first studies of expertise in physics employed a problem categorization task where physics graduate
student “experts” and undergraduate “novices” were asked to sort problems (without solving them)
according to similarity of solution (Chi et al., 1981). That study revealed that experts cued on the deep
structure of problems as their categorization criteria (i.e., the concepts and principles needed to solve a
problem); in contrast, novices focused on the literal objects, quantities, and motion in the problem, or
surface level attributes. Novices’ categorization criteria are unproductive; for example, two inclined
plane problems can be solved with entirely different approaches, depending on the principle applied.
The same study asked experts and novices to discuss the general approach that could be used to solve
introductory mechanics problems, with experts discussing concepts and procedures to be applied, and
novices discussing equations that could be used and quantities/variables in the problems.
These findings are consistent with additional studies on the problem-solving processes actually used by
experts and novices when solving introductory-level physics questions, which indicate that experts first
conduct a qualitative analysis of a problem to identify appropriate principles (Larkin, 1981; Larkin et al.,
1980; Reif & Heller, 1982). Inexperienced problem solvers use the quantities in the problem to hunt for
equations and start substituting values in a process referred to as plug and chug (Tuminaro & Redish,
2007; Walsh et al., 2007). Another common method used by novices is the memory-based approach,
where they attempt to find a similar problem and pattern-match the solution (Walsh et al., 2007).
Although expertise is often described as a dichotomy, it is a continuum with varying levels of
competency and proficiency (Chi, 2006; de Jong & Ferguson-Hessler, 1986), and expertise can also be
described in terms of efficiency, such as adaptive versus routine expertise (Schwartz et al., 2005).
Studies of expertise indicate that it takes a long time to become an expert in any domain (~10,000
hours; see Ericsson et al., 1993). Nevertheless, we will discuss instructional approaches that make the
path toward expertise in physics more efficient.
225
Since the early studies of expert-novice differences in physics, numerous studies have been performed,
providing additional insights (for a thorough review, see Docktor & Mestre, 2014). Some of these
insights include how experts and novices differ in their ability to use representations such as physics
diagrams to identify goals and sub-goals while solving problems (Kohl & Finkelstein, 2007; Kohl et al.,
2008) and how experts employ metacognitive strategies like self-explanations to monitor their progress
toward a solution and evaluate the reasonableness of their final answer (Chi et al., 1989; Kohl &
Finkelstein, 2005). The studies we are about to discuss flesh out more details about novices’ behavior in
physics.
Description of Relevant Research: How Well Do Novices “See” the

Underlying Physics in Situations
Change Blindness
A study that illustrates what novices see (or do not see in this case) in physics problems and situations
uses the notion of change blindness from visual cognition (Feil & Mestre, 2010). Change blindness refers
to individuals’ inability to perceive changes in pictures or situations when a change is made to them
(e.g., Jensen et al., 2011; Simons & Ambinder, 2005; in YouTube, search for “change blindness person
change events” for examples of change blindness). In the study by Feil and Mestre (2010), experts
(physics graduate students having taught introductory physics) and novices (undergraduates having
finished an introductory physics course) were presented with a diagram of a physics situation and asked
to formulate an explanation. Then, while the participants were distracted, the picture in the situation
was changed slightly to see if the participants noticed the change after they reexamined the diagram.
Figure 1A shows the original diagram with a curved ramp and below it the changed diagram with a
straight ramp. Importantly, the same physics principle, namely, conservation of mechanical energy,
could be used to explain both situations. If experts extract the principle needed to explain the situation
while initially deliberating, then we might predict that they would not notice and/or disregard the
change, because the same explanation fits both scenarios. On the other hand, because novices are
known to focus on surface attributes (Chi et al., 1981), we might predict that they would notice the
change in the ramp shape. The results were that none of the 31 experts or 48 novices indicated that
they noticed any change in the diagram. This lopsided result was surprising, perhaps less so for experts
than for novices given the discussion of expert-novice behavior in this paragraph.
Figure 1A
Sample Physics Question for a Study on Change Blindness
Note. The top diagram shows the original diagram with a curved ramp and the bottom diagram shows
the changed diagram with a straight ramp.
226
Because no one noticed the change, the study by Feil and Mestre (2010) followed up with another
scenario to consider—Figure 1B shows the original and changed diagrams. Unlike the situation in Figure
1A, in 1B a correct explanation of the original and changed situations differed—that is, the tension in the
hanging mass is different depending on whether or not a string is attached to the mass, M. The results
differed dramatically in this situation. Experts realized that the concepts they had applied to explain the
behavior in the original 1B diagram would be inappropriate to explain the behavior in the changed 1B
diagram. Twenty-four of the experts (77%) noticed the addition of the string (with the vast majority of
those noticing the change blaming themselves for “missing it” before they were distracted so that the
diagram could be changed). We might wonder why the remaining 23% of the experts did not notice the
important change, but recall that these experts were graduate students, not physics PhDs. In contrast,
only 13 of the novices (27.5%) noticed this important change. In fact, those novices who noticed the
added string were able to provide an expert-like explanation for why their original explanation would
not work in the changed diagram (hence, those novices noticing the change were behaving like experts).
Figure 1B
Sample Physics Question for a Study on Change Blindness
Note. The top diagram shows the original diagram, and the bottom diagram shows the changed diagram
with an additional string attached to the block on the table.
This study demonstrates that experts can, and do, extract the relevant concepts/principles needed to
explain a physics scenario, and that if a change is made in a scenario that does not affect the principles
that are used to explain it, then the change will very likely go unnoticed. Novices, on the other hand,
tend to miss a lot of important details in physics scenarios, suggesting that as part of our instructional
regimen we should help novices analyze and interpret important features of physics scenarios that
signal the application of specific physics principles.
Flicker Study: Additional Evidence That Experts Detect Physics-

Relevant Features Faster Than Novices
The change blindness study described above explores incidental changes and whether they are noticed.
Another approach for studying what experts and novices notice in physics situations is to explore
intentional changes—that is, participants are instructed to look for a change between two scenarios and
227
are timed to determine how quickly they can detect the change. To do so, the flicker paradigm is used
(Rensink et al., 1997). Here, participants are shown two alternating diagrams where one differs from the
other by a small change; their task is to find the change as quickly as possible (Morphew et al., 2015).
Figure 2 depicts how the flicker technique is used in this context. The first image is presented to a
participant for 200 ms, then a black screen is presented for 100 ms, then the changed image is
presented for 200 ms, then the black screen for 100 ms, and the cycle continues until the participant
finds the change. The effect of alternating between the two diagrams with a blank screen in between
creates a flickering of the image on the screen. Once the subject finds the change, they click on the
location of the change on the image to verify that they have, in fact, found the actual change.
Figure 2
Sample Flicker Diagram Containing a Physics-Relevant Change
Note. Original and altered images are reproduced from Physical Review Special Topics - Physics
Education Research, 2015, 11, 020104 Supplemental Materials with permission from the American
Physical Society.
Thirty pairs of diagrams were used in the flicker study (Morphew et al., 2015); half of them contained
physics-relevant changes (i.e., the change altered the underlying physics in the situation) and half
contained physics-irrelevant changes (i.e., the same underlying physics principle applied to both
situations despite the change). The experts were 19 physics graduate students who had taught
introductory physics, and the novices were 20 students from the educational psychology subject pool
who had not taken introductory physics in college. The study compared the reaction time for finding the
change between experts and novices.
Figure 3 displays the findings from Morphew et al. (2015). As expected, experts found physics-relevant
changes significantly faster than physics-irrelevant changes, whereas novices did not. In fact, the pattern
was reversed for the novices, who showed faster (but nonsignificant) response times for finding physics-
irrelevant changes than physics-relevant changes. These findings are consistent with other studies
(Shore & Klein, 2000; Takahashi & Watanabe, 2008), and suggest that the expert participants were
guided by their expertise in physics (i.e., schemas) in processing the diagrams and constructed physics-
228
relevant meaning from the diagrams as they processed them. This meaning-building of the configuration
of objects in the diagrams allows experts to detect physics-relevant changes more quickly in contrast to
“cosmetic” surface-level changes to a diagram.
Figure 3
Mean Response Times for Experts and Novices
Note. Modified from Physical Review Special Topics - Physics Education Research, 2015, 11, 020104 with
permission from the American Physical Society.
Studies using eye-tracking technology have further confirmed how physics knowledge can influence
what someone “sees” when viewing a diagram (Madsen et al., 2012). When presented with conceptual
physics problems that included a diagram, participants who answered the problem correctly spent a
higher proportion of time viewing relevant regions of the diagram. In addition, it was possible to overlay
visual cues on a diagram to influence participants’ visual attention and reasoning to be more expert-like
(Madsen et al., 2013).
Eye Tracking Study: For Physics Novices, Looking at Conceptual

Information Does Not Mean They “See” It
The studies reviewed above indicate that, lacking conceptual physics schemas, physics novices tend to
poorly prioritize portions of physics diagrams conveying important conceptual information. We have
referred to this as an inability to “see” important underlying physics principles in situations. There is also
evidence that when novices look at physics conceptual information for relatively long periods of time,
they struggle to extract the deep meaning behind the information. The relevant study (Smith et al.,
2010) was motivated by physics professors’ assumption that, because novices tend to rely heavily on
equation manipulations in problem solving, they are likely to ignore relevant conceptual knowledge
made available to them. The study presented novices (students who had finished an introductory
mechanics course for scientists and engineers) with worked-out solutions to challenging problems,
where the solution was specifically presented to highlight how conceptual knowledge was used in
solving the problems (cf. Renkl, this volume). The worked-out solutions had two columns that presented
the various numbered steps that went into solving the problem. The left column presented, in words
only, the concept/procedure/justification being applied; the right column showed how the
229
concept/procedure/justification was applied in equation form (with other relevant manipulative steps).
Participants studied four such worked-out problems while their eye-gaze patterns were being recorded
in an eye-tracker. After studying the worked-out examples in the eye-tracker, participants worked two
related problems as well as answered a memory recall assessment evaluating how well they
remembered information from either the textual or mathematical columns.
Given their bias from observing novice behavior in their classes, the experimenters expected
participants to largely ignore the column containing conceptual/procedural information and to focus on
the column containing the mathematical information. The experimenters were surprised to find that the
novices spent considerable time studying the conceptual information: 40% of the time was devoted to
those portions of the worked-out solution. Forty percent of the time is a long time to spend studying the
conceptual/procedural information used in the solution to the problems. One could easily argue that
studying the mathematical column in the solution should have taken longer in order to verify the
mathematical manipulations and fill in missing steps. Participants scored about 50% in solving the
related problems, which is not unreasonable given the challenging nature of all problems used in this
study. Perhaps the most revealing finding was that, despite spending 40% of the time studying the
conceptual information, participants retained little of that information. In the memory recall
assessment, participants’ performance was about 20% on the conceptual questions. Thus, “looking” at
the conceptual information for a significant portion of the time did not translate to their ability to
process and retain the underlying concepts and accurately convey their justification for why they can be
applied in the specific problem context. The novices did not possess the schemas necessary to interpret
and understand the conceptual information provided. The take-home message from this study is that
the poor grasp of conceptual knowledge by beginning physics students is not simply due to disregarding
conceptual knowledge in their educational journey. Rather, digesting, understanding, and making sense
of conceptual knowledge requires mental schemas that develop as one gains expertise. As we will
explore in the next major section, instructors need not be resigned to wait long periods of time before
students are ready to learn conceptual knowledge; there are instructional strategies that can be
implemented that target students’ development of a more rapid conceptual knowledge.
Recommendations
The studies described above demonstrate a relative lack of mastery of physics conceptual knowledge by
novice physics students. Equation-pushing can get students only so far. Left to their own devices,
considerable time passes prior to novices recognizing the value of conceptual knowledge in interpreting
physics scenarios and solving problems. How can we focus more of students’ attention on learning/using
conceptual knowledge? As it turns out, there are ways to do this.
Recommendation 1: Practice Identifying Principles for Solving Problems
One method for practicing principle identification (without actually solving physics problems) is printing
problem statements onto index cards and asking students to sort them based on similarity of solution
(Chi et al., 1981). This can be somewhat time consuming in a classroom setting, and the format makes it
difficult for instructors to provide meaningful feedback on students’ “piles” of cards. Another possibility
is to present students with a problem statement and ask them to identify which principle applies. A
series of studies by Dufresne, et al. (1992) used a menu-driven, computer-based Hierarchical Analysis
Tool (HAT) program to prompt students to select a primary principle that would be used to solve a
problem. Subsequent menus asked specific questions for instantiating that principle (for example,
identifying conservative or non-conservative forces acting on the system). Students who used HAT
improved in their ability to judge whether problems would be solved similarly, to use deep structure to
reason about problems, and to actually solve problems.
230
Another way to practice identifying principles is by using multiple problem statements. In a three-
problem categorization task, students are presented with a model problem and asked to select which of
two comparison problems would be solved most similarly to the model problem (Hardiman, et al.,
1992). Problems can be carefully designed to match on their surface features only, physics principle
only, both, or neither. Yet another approach is to use a two-problem categorization task format.
Students are presented with a pair of problems and asked whether they would be solved similarly and
why or why not. A study by Docktor et al. (2012) used this format to present student volunteers with 32
pairs of problems on a computer screen and asked whether they would be solved similarly. Half of the
students received sparse feedback on the correctness of their choice (i.e., simply whether they were
correct, Figure 4A), and the other half received more elaborate feedback focusing on the concepts and
principles used to solve each problem (Figure 4B). Participants provided their reasoning eight times
during the session. The two physics problems shown side-by-side have the same surface-level objects (a
pendulum swinging vertically on a string) but they would be solved using different physics principles.
After selecting Yes or No, the participant was shown a feedback screen in either a sparse format (first
line about correctness) or a more elaborate format about the concepts and principles in the problems.
Figure 4A
Sample Computer Screens Displayed to Participants in the Categorization Study
Note. Two physics problems were shown side-by-side in the initial item screen (a) and the participant
indicated whether or not they believe the two problems would be solved similarly. Modified from by
Docktor et al. (2012), Physical Review Special Topics - Physics Education Research, 2012, 8, 020102
Supplemental Materials with permission from the American Physical Society.
231
Figure 4B
Sample Computer Screens Displayed to Participants in the Categorization Study
Note. The elaborate feedback screen for a correct selection of “No” is shown; participants in the sparse
condition viewed only the first line of feedback. Modified from by Docktor et al. (2012), Physical Review
Special Topics - Physics Education Research, 2012, 8, 020102 Supplemental Materials with permission
from the American Physical Society.
Although this was a brief intervention (less than one hour), students in the elaborate feedback condition
significantly shifted their reasoning to focus more on concepts and principles during the session
compared to students in the sparse feedback condition (see Docktor et al., 2012, for the details).
Categorization is a difficult task for students; in the item shown in Figure 4 only 23% of students
correctly selected that the problems would NOT be solved similarly. Recall that novice behavior
indicates students would look to surface attributes in deciding whether problems are solved similarly,
and these two problems match on surface attributes. When examining the accuracy of students’
reasoning statements, the study also found that although it was possible to shift students towards more
principle-based criteria in their judgments, students still struggled to identify appropriate principles for
the problems. Nevertheless, shifting students’ attention toward principle-based problem-solving
approaches is highly desirable and difficult to accomplish with traditional instruction. Not surprisingly, it
takes a lot longer than one hour for novices to develop the ability to select the correct principles for
solving a problem.
Recommendation 2: Practice Identifying Principles During Problem Solving
As we discussed earlier in this chapter, inexperienced novice problem solvers typically use equation-
based approaches when solving physics problems compared to more expert-like approaches focused on
232
concepts and principles. One general instructional technique for highlighting principles during problem
solving is to require students to follow prescribed frameworks in their problem solutions (Heller et al.,
1992; Van Heuvelen 1991a; 1991b). When using frameworks, students are prompted to draw a picture
or diagram and identify the key physics principle(s) prior to writing down specific equations.
Frameworks can be utilized for written solutions or for online computer coaches for problem solving
(Ryan et al., 2016). Another possibility is to use a two-column format for problem solutions (discussed
earlier), where the conceptual information is written in words on the left column, and corresponding
equations are in the right column (Smith et al., 2010). It is important that instructors model how to use
this framework when working through example problems in class, and to reinforce the technique
through assessments.
Another way to highlight concepts is with qualitative problem-solving strategies (Leonard et al., 1996). In
this context, a strategy is a prose paragraph that contains three key pieces: identifying the Principle
relevant for the problem, Justifying why that principle or concept is appropriate based on checking
conditions of applicability, and providing a Plan or procedure for applying the principle to the specific
problem. A sample strategy and two-column solution is shown in Figure 5 (see Docktor et al., 2015 for a
study on strategy writing in high schools). Since strategy writing is complex and takes practice, varying
levels of scaffolding would be required in the classroom. An instructor could begin by modeling how to
write strategies or provide sample strategies and ask students to execute the strategy to solve the
problem. Blank or partially filled-in templates can also be used to prompt students to think about the
different conceptual/procedural steps in a strategy. It is also important for the problem strategy to be of
sufficient difficulty so that students will appreciate the value in writing out plan steps in words prior to
writing physics equations and executing mathematical procedures. We should point out that writing
strategies and executing them into problem solutions also interconnects problem features,
concepts/principles, and procedures/equations, thereby allowing students to build schemas.
Figure 5
Sample Strategy and Two-column Solution
233
Recommendation 3: Highlight the Hierarchical Structure of Physics Concepts
Instructors can utilize concept mapping to highlight the structure of physics concepts, either by asking
students to engage in the creation of maps or to provide a sample map for students to use while solving
problems. A sample hierarchical chart is provided in Van Heuvelen (1991a) to illustrate the structure of
Newtonian physics. As you progress down the chart, the application of principles becomes more
detailed by listing basic equations associated with each principle. Lindstrøm & Sharma (2009; 2011)
implemented “link maps” to illustrate key physics concepts from lecture and to demonstrate how they
are related to each other. Students who attended supplemental “map meetings” had higher retention,
gave more positive feedback about the course, and performed better on exams. Concept maps can be
displayed on a side board or wall in a classroom and continuously “built” as more concepts are added
throughout the course.
234
Recommendation 4: Implement Assessments That Promote Conceptual Understanding
If students are prompted to identify appropriate concepts and principles as part of the problem-solving
process (either as part of a written solution framework or during strategy writing), the accuracy of those
concepts can be assessed using a rubric (Docktor et al., 2016) or by using a grading scheme that awards
points for explicitly stating the physics approach. Students could also format their solutions using two
columns, where conceptual information is in the left column and equations are in the right column, and
their solutions could be assessed according to that format (Docktor et al., 2015 ; Smith et al., 2010;). We
discussed how computer tools could be used to prompt students for physics principles, such as HAT or
computer coaches, and that these systems could double as assessments if responses are administered
and recorded as part of a course (Dufresne et al., 1992; Ryan et al., 2016). We also reviewed how two-
problem and three-problem categorization tasks can be used to assess students’ ability to select
appropriate principles for problems without actually solving the problems. Additional question formats
include finding errors and equation instantiation (see Docktor et al., 2015, for examples). In a finding
errors question, students are presented with the solution to a problem which contains a conceptual
mistake which they must identify and describe. This is a very high-level, challenging task for most
students. In equation instantiation questions, students must match the quantities given in a problem to
blank spaces in a final equation/expression in the problem solution. Students must have a deeper
understanding of the principles and specific application of those principles in the problem in order to fill
in the blanks. For additional examples of alternate types of problems in physics, see the review by
Docktor and Mestre (2014).
Conclusion
The research reviewed in this chapter, which borrows methodologies from cognitive psychology to study
expert-novice individual differences in physics, adds details to already known general attributes of
expertise in physics. It takes considerable time and effort to develop expertise in a complex domain such
as physics, and novice physics students behave like novices in any other domain. In this case, being a
novice translates to not “seeing” the underlying deep physics principles in situations and problems and
to lacking schemas that can be used in solving challenging problems. Unfortunately, traditional
instruction is not very effective at helping physics novices focus on conceptual knowledge. This is not
meant as a criticism of physics instructors—they teach both concepts and procedures for solving
problems. However, typical homework assignments and exams almost exclusively require the
manipulation of equations to generate solutions. Hence, students focus on equation-pushing as a means
of achieving good grades in an introductory course. Thus, teaching conceptual knowledge does not
translate to "revealing", “actively using”, or “deeply thinking” about how conceptual knowledge applies
in physics situations. The recommendations provided in this chapter are meant to highlight the role of
conceptual knowledge in introductory physics instruction so that conceptual knowledge is developed
side-by-side with mathematical applications during problem solving.
Author Note
Send correspondence to Jennifer Docktor: [email protected]
References
Chi, M.T.H. (2006). Two approaches to the study of experts’ characteristics. In K.A. Ericsson, N. Charness,
R.R. Hoffman, & P.J. Feltovich (Eds.), The Cambridge handbook of expertise and expert
235
performance (pp. 21-30). Cambridge University Press.
https://doi.org/10.1017/CBO9780511816796.002
Chi, M.T.H., Bassok, M., Lewis, M.W., Reimann, P., and Glaser, R. (1989). Self-explanations: How
students study and use examples in learning to solve problems. Cognitive Science, 13(2), 145-
182. https://doi.org/10.1207/s15516709cog1302_1
Chi, M.T.H., Feltovich, P.J. & Glaser, R. (1981). Categorization and representation of physics problems by
experts and novices. Cognitive Science, 5(2), 121-152.
de Jong, T, & Ferguson-Hessler, M.G. (1986). Cognitive structures of good and poor novice problem
solvers in physics. Journal of Educational Psychology, 78(4), 279-288.
Docktor, J.L., Dornfeld, J., Frodermann, E., Heller, K., Hsu, L., Jackson, K.A., Mason, A., Ryan, Q.X., &
Yang, J. (2016). Assessing student written problem solutions: A problem-solving rubric with
application to introductory physics. Physical Review Physics Education Research, 12(010130), 1-
18. https://doi.org/10.1103/PhysRevPhysEducRes.12.010130
Docktor, J.L. & Mestre, J.P. (2014). Synthesis of discipline-based education research in physics. Physical
Review Special Topics: Physics Education Research, 10(020119), 1-58.
https://doi.org/10.1103/PhysRevSTPER.10.020119
Docktor, J.L, Mestre, J.P. & Ross, B.H. (2012). Impact of a short intervention on novices' categorization
criteria. Physical Review-Special Topics: Physics Education Research, 8(020102), 1-11.
Docktor, J.L., Strand, N.E., Mestre, J.P., & Ross, B.H. (2015). Conceptual problem solving in high school
physics. Physical Review Special Topics – Physics Education Research, 11(020106), 1-13.
Dufresne, R., Gerace, W.J., Hardiman, P.T. & Mestre, J.P. (1992). Constraining novices to perform expert-
like problem analyses: Effects on schema acquisition. Journal of the Learning Sciences, 2(3), 307-
331. https://doi.org/10.1207/s15327809jls0203_3
Ericsson, K.A, Krampe, R.T., & Tesch-Romer, C. (1993). The role of deliberate practice in the acquisition
of expert performance. Psychological Review, 100(3), 363-406. https://doi.org/10.1037/0033-
295X.100.3.363
Feil, A. & Mestre, J.P. (2010). Change blindness as a means of studying expertise in physics. Journal of
the Learning Sciences, 19(4), 480-505. https://doi.org/10.1080/10508406.2010.505139
Hardiman, P.T., Dufresne, R. & Mestre, J.P. (1989). The relation between problem categorization and
problem solving among novices and experts. Memory & Cognition, 17(5), 627-638.
Heller, P., Keith, R., & Anderson, S. (1992). Teaching problem solving through cooperative grouping. Part
1: Group versus individual problem solving. American Journal of Physics, 60(7), 627-636.
Jensen, M. S., Yao, R., Street, W. N., & Simons, D. J. (2011). Change blindness and inattentional
blindness, Wiley Interdisciplinary Reviews: Cognitive Science, 2(5), 529-546.
https://doi.org/10.1002/wcs.130
236
Kohl, P.B., & Finkelstein, N.D. (2005). Student representational competence and self-assessment when
solving physics problems. Physical Review Special Topics – Physics Education Research,
1(010104). https://doi.org/10.1103/PhysRevSTPER.1.010104
Kohl, P.B, & Finkelstein, N.D. (2008). Patterns of multiple representation use by experts and novices
during physics problem solving. Physical Review Special Topics – Physics Education Research,
4(010111), 1-10. http://dx.doi.org/10.1103/PhysRevSTPER.4.010111
Kohl, P.B. Rosengrant, D., & Finkelstein, N.D. (2007). Strongly and weakly directed approaches to
teaching multiple representation use in physics. Physical Review Special Topics – Physics
Education Research, 3(010108). https://doi.org/10.1103/PhysRevSTPER.3.010108
Larkin, J. (1981). Cognition of learning physics. American Journal of Physics, 49(6), 534-541.
Larkin, J.H., McDermott, J., Simon, D.P., & Simon, H.A. (1980). Models of competence in solving physics
problems. Cognitive Science, 4(4), 317-345. https://doi.org/10.1016/S0364-0213(80)80008-5
Leonard, W.J., Dufresne, R.J., & Mestre, J.P. (1996). Using qualitative problem-solving strategies to
highlight the role of conceptual knowledge in solving problems. American Journal of Physics,
64(12), 1495-1503. https://doi.org/10.1119/1.18409
Lindstrøm, C. & Sharma, M.D. (2009). Link maps and map meetings: Scaffolding student learning.
Physical Review Special Topics – Physics Education Research, 5(010102), 1-11.
Lindstrøm, C. & Sharma, M.D. (2011). Teaching physics novices at university: A case for stronger
scaffolding. Physical Review Special Topics – Physics Education Research, 7(010109), 1-14.
Madsen, A.M., Larson, A.M., Loschky, L.C., & Rebello, N.S. (2012). Differences in visual attention
between those who correctly and incorrectly answer physics problems. Physical Review Special
Topics – Physics Education Research, 8(010122), 1-13.
Madsen, A., Rouinfar, A., Larson, A.M., Loschky, L.C., & Rebello, N.S. (2013). Can short duration visual
cues influence students’ reasoning and eye movements in physics problems? Physical Review
Special Topics – Physics Education Research, 9(020104), 1-16.
Morphew, J.W., Mestre, J.P., Ross, B.H., & Strand, N.E. (2015). Do experts and novices direct attention
differently in examining physics diagrams? A study of change detection using the flicker
technique. Physical Review-Special Topic-Physics Education Research, 11(2) 020104.
Reif., F., & Heller, J.I. (1982). Knowledge structure and problem solving in physics. Educational
Rensink, R.A., O’Regan, J.K., & Clark, J.J. (1997). To see or not to see: The need for attention to perceive
changes in scenes. Psychological Science, 8(5), 368-373. https://doi.org/10.1111/j.1467-
9280.1997.tb00427.x
Ryan, Q.X., Frodermann, E., Heller, K., Hsu, L., & Mason, A. (2016). Computer problem-solving coaches
for introductory physics: Design and usability studies. Physical Review Physics Education
Research, 12(010105), 1-17. https://doi.org/10.1103/PhysRevPhysEducRes.12.010105
237
Schwartz, D.L., Bransford, J.D., & Sears, D. (2005). Efficiency and innovation in transfer. In J. Mestre (Ed.),
Transfer of learning (pp. 1-51). Information Age Publishing.
Shore, D. I., & Klein, R. M. (2000). The effects of scene inversion on change blindness. The Journal of
General Psychology, 127(1), 27-43. https://doi.org/10.1080/00221300009598569
Simons, D. J., & Ambinder, M. S. (2005). Change blindness: Theory and consequences. Current Directions
in Psychological Science, 14(1), 44-48. https://doi.org/10.1111/j.0963-7214.2005.00332.x
Smith, A.D., Mestre, J.P., & Ross, B.H. (2010). Eye-gaze patterns as students study worked-out examples
in mechanics. Physical Review Special Topics: Physics Education Research, 6(020118), 1-9.
https://doi.org/10.1103/PhysRevSTPER.6.020118 .
Takahashi, K., & Watanabe, K. (2008). Persisting effect of prior experience of change blindness.
Perception. 37(2), 324-327. https://doi.org/10.1068/p5906
Tuminaro, J., & Redish, E.F. (2007). Elements of a cognitive model of physics problem solving: Epistemic
games. Physical Review Special Topics – Physics Education Research, 3(020101), 1-22.
Van Heuvelen, A. (1991a). Learning to think like a physicist: A review of research-based instructional
strategies. American Journal of Physics, 59(10), 891-897. https://doi.org/10.1119/1.16667
Van Heuvelen, A. (1991b). Overview, case study physics. American Journal of Physics, 59(10), 898-907.
Walsh, L.N., Howard, R.G., & Bowe, B. (2007). Phenomenographic study of students’ problem solving
approaches in physics. Physical Review Special Topics – Physics Education Research, 3(020108),
1-12. https://doi.org/10.1103/PhysRevSTPER.3.020108
238
In Their Own Words Implementing Cooperative Learning in Large
Undergraduate Classes via PLTL
to Apply the Science of Learning in Regina F. Frey
University of Utah
Editors
Lauren L. Kordonowy
Scott E. Lewis
Victor A. Benassi
University of South Florida
Overview
Active learning has been shown to improve student performance in a wide variety of STEM courses
(Freeman et al., 2014; Theobald et al., 2020). While there are numerous active-learning strategies that
are effective, incorporating these strategies into large courses (e.g., 50-300 students) can be difficult
(Borda et al., 2020; Stains et al., 2018). This is especially true for introductory courses in which students
are transitioning into college, are learning how to successfully study and understand college-level
science, and are more likely to have large science class sizes. One primary active-learning approach,
which has been shown to improve student outcomes, especially for STEM courses, is having students
work together in small groups to discuss questions or solve problems (Barrasso & Spilios, 2021;
Rodriguez et al., 2020; Wilson & Varma-Nelson, 2016). However, for such cooperative-learning methods
to be effective, students often need support and structure to use these group-based strategies in a
manner that encourages discussion and therefore enhances learning and engagement. One strategy to
provide support for students in small groups is to have peers who have previously taken the course
facilitate these groups. Peer-led Team Learning (PLTL) is one such student-centered, group-oriented,
peer-supported pedagogy. PLTL was developed in the early 1990s as an approach to incorporate
structured group work into large introductory-course structures to support students as they are
transitioning into higher-education STEM courses (Gosser et al., 1996; Gosser Jr. et al., 2010). This
chapter is intended to introduce instructors and others who play a role in supporting instructors and
students in STEM courses to the PLTL approach, evidence of its effectiveness, and key criteria for
implementation.
In the PLTL approach, students work in groups that typically range from 6-16 students to solve
instructor-prepared problems and are facilitated by a peer leader who is trained in cooperative-learning
pedagogy. When working with larger student groups, the peer leader typically divides the students into
smaller groups of 3-4 during each session. The peer leader’s role is to guide the students in the group to
solve the problem together, scaffold ways for the students to equally participate (via different
cooperative-learning strategies such as round robin, scribe, and pairs [Barkley et al., 2014]), and
encourage the students to discuss their reasoning or process. The weekly problem sets, written by a
member of the instructor team, include approximately 8-10 questions and are designed to be solved
cooperatively and to promote active discussion. However, giving too many problems to complete within
a given setting leads to less robust discussions and hence less-effective sessions (see Wilson & Varma-
Nelson, 2016 for a review). The sessions are typically 90-120 minutes, although some variations use 50
to 75 minutes to fit into a single class meeting. A main criterion of PLTL is the required training of the
peer leaders to learn to facilitate a group that promotes student-student discussion and encourages
equal participation in solving the problem. Training occurs in several forms, ranging from several hours
of discussions with faculty (Gosser et al., 1996) to semester-long courses (Hockings et al., 2008;
239
Szteinberg et al., 2020; Tien et al., 2004). The semester-long courses typically include reading and
discussing literature on teaching and learning and how to apply this literature to the group sessions.
The PLTL philosophy is to encourage students to critically think about their problem-solving processes as
they work in their groups to solve problems for which correct answers are not provided, and to promote
metacognitive functioning by asking them to decide as a group whether or not their answer is correct.
Key objectives of the PLTL model are to enable students to become a community of scholars, take
responsibility for their own learning, and emerge as independent learners with skills in teamwork and
communication. The PLTL approach has several key criteria (Varma-Nelson et al., 2004; Wilson & Varma-
Nelson, 2016): 1) the program is integrated into the course; 2) course instructors are closely involved
with the program organization and peer-leader training; 3) peer leaders are trained in cooperative-
learning pedagogy and are closely supervised by a member of the instructor team; 4) materials are
written at an appropriately challenging level to encourage cooperation, active learning, and discussion;
5) the logistics (e.g., group size, session length, and space/furniture) are organized to promote group
work; and 6) the department and institution support the program via funding and recognition such as
valuing the work of the instructors in running the PLTL program in the faculty annual evaluation or giving
awards to outstanding peer leaders as departments do for outstanding undergraduate research.
PLTL is used in many STEM undergraduate disciplines (e.g., biology, chemistry, mathematics, physics,
psychology, and computer science) and in varying institution types (see Wilson & Varma-Nelson, 2016
for a review). The PLTL approach has been associated with gains in learning and promoting positive
student attitudes toward the STEM discipline (see details below). Hence, having peer leaders who are
trained in and focus on giving structured facilitation to students during cooperative-learning activities
enhances student outcomes and engagement in large courses and assists instructors in incorporating
active-learning strategies in their courses.
This chapter has been organized so that instructors, and those who support instructors and students in
STEM courses, learn about the background and effectiveness of the PLTL method and then are given key
points to consider for successful implementation. It contains the theoretical frameworks on which PLTL
is based, and research concerning the effectiveness of PLTL on student outcomes (e.g., performance,
persistence, knowledge gains, and attitudinal measures), the discourse that occurs within the groups as
students solve problems together, the impacts peer-leader behavior has on student interactions, and
the effects on the students who participate as peer leaders. The last section discusses education
implications for practitioners interested in incorporating PLTL into their courses. As will be seen
throughout the chapter, although the PLTL method is highly successful and studies have been conducted
in the above areas, our understanding of the robustness and effective implementation of PLTL would
benefit with more research in many of these areas.
Background and Research

Theoretical Framework
Multiple theoretical frameworks may explain the impact of PLTL on student learning. The most common
theoretical framework advanced in the research literature to explain the effectiveness of PLTL has been
social constructivism, and more specifically, Vygotsky’s description of the Zone of Proximal Development
(ZPD) (Baez-Galib et al., 2005; Eren-Sisman et al., 2018; Merkel & Brania, 2015; Smith et al., 2014; Tien
et al., 2002). According to Vygotsky’s model, student learning can be described as: 1) a current skill set
that they can enact independently, 2) a set of proximally advanced skills that can be enacted when
working with more capable peers or instructors, and 3) a set of skills that exists beyond the students’
current capabilities. The middle category represents the ZPD (cf. Metcalfe, this volume) and is identified
240
as an ideal target for instruction to promote student development (National Academies of Sciences,
Engineering, and Medicine, 2018). Currently, cognitive researchers are studying the region of proximal
learning (a region most open to student learning and is based upon the ZPD region) to better
understand the methods or approaches that students use or that instructors could use to most likely
produce efforts that optimize students’ learning (Metcalfe, 2009; Metcalfe & Jacobs, 2010; Metcalfe &
Kornell, 2005). PLTL advocates have argued that peer leaders can more effectively interact with students
in this region than an instructor (Baez-Galib et al., 2005; Merkel & Brania, 2015; Tien et al., 2002), and
thus the use of peer leaders is essential to making an instructional environment that effectively works
within students’ ZPD. In addition, others have further suggested that the ZPD concept informs the
training of peer leaders and the development of appropriately challenging materials (Eren-Sisman et al.,
2018; Smith et al., 2014), and thus the training and selection of materials is of primary importance for
instructing within students’ ZPD. Other PLTL literature has described a more general framework of social
constructivism, which characterizes knowledge construction as dependent on productive social
interactions related to the target construct (Frey et al., 2018; Tien et al., 2002). It should also be
acknowledged that PLTL represents a form of cooperative learning, and numerous theoretical
frameworks have been proposed to explain the effectiveness of cooperative learning. As reviewed by
Slavin (1996), cooperative learning can: 1) facilitate student debate, which can facilitate the generation
of ideas; 2) promote students generating explanations for other students, which requires a meaningful
reorganization of content in the mind of the one offering the explanation (see also Fiorella, this volume);
3) motivate students to persist as learning is seen as a shared goal as opposed to an individual goal, and
4) motivate students to learn and help each other learn out of an altruistic desire to help one another.
To the best of our knowledge, no research has been conducted to empirically test the plausibility of any
particular theoretical framework as the mechanism by which PLTL has been associated with learning
gains.
Research on PLTL
Academic Achievement
Ample research has been performed on investigating the impact of PLTL on academic achievement in
higher education in STEM. The majority of these studies have been conducted in general chemistry
courses (Baez-Galib et al., 2005; Eren-Sisman et al., 2018; Frey et al., 2018; Lewis, 2011; Lewis & Lewis,
2008; Mitchell et al., 2012; Robert et al., 2016), including as an online component (Smith et al., 2014),
organic chemistry courses (Tien et al., 2002; Wamser, 2006) and allied health chemistry courses
(Akinyele, 2010). A recent meta-analysis synthesized studies in chemistry education that compared PLTL
instruction to instruction as normal (typically lecture-based instruction) and found students with the
PLTL pedagogy scored better on measures of academic performance, with an average effect size (d) of
0.48 standard deviations, than students with the instruction as normal pedagogy (Rahman & Lewis,
2020). Studies have also been conducted in introductory biology (Batz et al., 2015; Kudish et al., 2016;
Snyder et al., 2015) and calculus courses (Merkel & Brania, 2015).
The most common methodology in these studies has been a quasi-experimental design comparing
students receiving PLTL instruction to students receiving lecture-based instruction on common exam
scores. In this comparison design, students with PLTL performed better than students with lecture-
based instruction on exams when controlling for general measures of academic achievement, such as
SAT or ACT scores (Frey et al., 2018; Lewis & Lewis, 2008; Robert et al., 2016), a diagnostic test (Shields
et al., 2012), or measuring growth over time using a pretest-posttest design (Batz et al., 2015; Eren-
Sisman et al., 2018). There are also research results finding no significant difference on test scores
between these groups (Lewis, 2011; Merkel & Brania, 2015; Mitchell et al., 2012), although some of
these instances found that PLTL instruction had a lower percent of students withdrawing during the
241
term (Lewis, 2011; Mitchell et al., 2012) than lecture-based instruction, which is an important student
outcome for success. Another metric of academic achievement common to the research literature is
student pass rate, again frequently relying on a quasi-experimental design. The most common finding is
that courses using PLTL had pass rates 9 to 15% higher than courses using lecture-based instruction
(Baez-Galib et al., 2005; Batz et al., 2015; Lewis, 2011; Mitchell et al., 2012; Robert et al., 2016; Tien et
al., 2002), although examples with larger (Akinyele, 2010; Snyder et al., 2015) or smaller (Merkel &
Brania, 2015) differences in pass rate have been observed. In-person PLTL instruction has also been
compared to online PLTL instruction finding that students in both formats earned comparable scores on
a common exam, although the in-person PLTL instruction had a higher pass rate (Smith et al., 2014). In
addition, PLTL instruction has been associated with higher student enrollment in subsequent courses
that appears to result from the higher pass rate in the target course (Lewis, 2014; Mitchell et al., 2012;
Wamser, 2006) and more students declaring a major in the field of the target course (Kudish et al.,
2016). Hence, research examining PLTL effects on student academic achievement in STEM is robust and
wide-ranging. The overall findings in a variety of courses show that PLTL has positive effects on student
performance, pass rates, and retention in course and progression into subsequent courses over
primarily lecture-based instruction.
PLTL Impact on Student Attitudes
Studies examining attitudes of students in PLTL groups encompass a range of disciplines such as biology,
chemistry, engineering, and graduate nursing as reviewed by Wilson and Varma-Nelson (2016). In
studies comparing PLTL and non-PLTL students, results from surveys concerning motivation (Liu et al.,
2018), anxiety (Eren-Sisman et al., 2018), discipline self-concept and discipline attitude (Chan & Bauer,
2014), and discipline-based expert-like attitudes (Batz et al., 2015) showed positive or neutral findings
on the impact of PLTL depending on the attitudinal variable. For example, from these studies, PLTL
students were more motivated to study, had less situational anxiety, and more expert-like attitudes at
the end of the semester; however, no difference was seen in discipline self-concept or social anxiety in
the course. Other studies found PLTL students perceived PLTL participation positively when asked to
rate its impact on cognitive skills such as test-taking skills, problem-solving ability, exam-preparation and
performance ability (Batz et al., 2015; Hockings et al., 2008; Tien et al., 2002; White et al., 2012) and on
communication skills such as student interactions within a group, discussions about reasoning, and
working as a team (Hockings et al., 2008; Tien et al., 2002; White et al., 2012). Smith and colleagues’
(2014) comparison of students using online PLTL or in-person PLTL found that online-PLTL students were
slightly less satisfied with the PLTL sessions than the in-person PLTL students. However, similar to other
in-person results above, interviews of graduate nursing students, participating in in-person PLTL, found
that students perceived working in a group allowed them to explore concepts at a deeper level, created
a safe environment for sharing ideas, and that peer leaders were most helpful when they guided the
discussion and did not reteach (White et al., 2012). From the literature, in general, PLTL shows gains in a
variety of attitudinal variables and in student perception of the usefulness of participating in PLTL.
However, because of the importance for students to actively participate in their PLTL group, a primary
gap in the PLTL literature is an understanding of the social belonging and group inclusivity that students
feel in their group, especially for students of underrepresented identities in STEM.
Student Discourse During the Session
Several studies examined the student discourse occurring among PLTL-group members via video data in
general chemistry (Kulatunga et al., 2013; Repice et al., 2016). Kulatunga and colleagues (2013) reported
that students supported their answers with data and reasoning (warrant component), but they did not
often elaborate on their answers (backing component). However, when students co-constructed
solutions, they more often elaborated on their solutions and identified and modified incorrect solutions
242
without help from their peer leader. Repice et al. (2016) analyzed the discourse across three problem
types (calculation, data analysis, and model building) and found that students used: 1) regulative
language, which promoted discussions and exchanges of information; and 2) instructional discourse,
which allowed students to practice ‘talking science’, to develop a shared knowledge of chemistry
content and vocabulary, and to discuss the problem-solving process as they moved through the
problems together.
Two studies analyzed discourse that occurred during online PLTL. Although there were no performance
differences between in-person and online PLTL general chemistry students, Smith et al. (2014) found
that the in-person PLTL students focused on answers and only discussed the problem when their
teammates had different answers, whereas the online PLTL students discussed the problems frequently,
focused on the methods used, and shared multiple perspectives. Conversely, in an organic chemistry
study concerning drawing mechanisms, Wilson and Varma-Nelson (2018) found that the online PLTL
discussion focused more on memorization strategies, whereas in-person PLTL discussion focused more
on understanding the concepts and thinking more deeply about their ideas and solutions.
Because discussion about the solution and process is important to the effectiveness of cooperative-
learning approaches, additional research is needed into the type of discourse that occurs among the
students and how to improve this discourse to include elaboration, the practice of “talking science,”
discussion of process, and sharing of multiple perspectives.
Peer-Leader Studies
Research that examines the peer leader is less extensive and comprises two broad areas: the effect of
peer-leader behavior during the PLTL session on student discussion and interactions, and the effects of
participating as a leader on the peer leaders themselves.
Peer-Leader Behavior in a Session. Few studies have focused on peer-leader interactions with
students in their group. In one general chemistry study, Sawyer et al. (2013) videotaped PLTL sessions
and, despite all peer leaders receiving identical training, found that peer leaders engaged in one of two
‘talk’ approaches: instructional talk in which the leader directs the conversation by using closed
questions and explanations, and facilitative talk in which the leader facilitates conversation by using
managerial questions, open and refocusing questions, and restating student ideas. Further, when
leaders used facilitative talk, students participated equally, focused on process, and built
upon/elaborated on others’ ideas; however, when leaders used instructional talk, students participated
unequally, worked more independently, and focused on the answer not the process (Sawyer et al.,
2013). Another study found similar patterns across multiple STEM courses, where again despite the
leaders receiving identical training, the leaders were observed adopting either a facilitative or
instructional approach (Streitwieser et al., 2010). In an online-PLTL study, researchers found that online
peer leaders used similar strategies as in-person PLTL peer leaders to encourage cohesive groups and
that a facilitative style was essential in making online-PLTL successful (Smith et al., 2014). Szteinberg et
al. (2020) examined peer leaders’ advice based on their experience as peer leaders, in which leaders
suggested: 1) creating a positive community that encourages risk taking; 2) promoting cooperation by
using group strategies, encouraging equal participation, and monitoring group interaction; and 3) using
questioning strategies to keep groups moving forward together and encouraging cooperative knowledge
building. The peer leader is essential in the success of an effective PLTL group; because few studies have
been conducted on peer-leader interactions with their students, additional research is needed to more
clearly understand the key features of peer-leader behavior that enhance participation, create robust
discussions among students, and set an inclusive learning environment within the group and the training
that can best promote these types of behaviors.
243
Participation as a Peer Leader. Most studies examining benefits to peer leaders consisted of self-
report surveys of undergraduate leaders and contained no comparison group (for a review, see Wilson
& Varma-Nelson, 2016). In these studies, peer leaders reported increased content knowledge, better
study skills, enhanced leadership and interpersonal skills, and increased self-confidence (Wilson &
Varma-Nelson, 2016). Chase et al. (2020) interviewed peer leaders several years after graduation to
examine what skills were transferrable to their current working positions. They found that peer leaders
perceived that their leadership skills, ability to cope with challenges, teamwork skills, self-confidence,
and problem-solving skills were the transferrable skills gained through experiences as a peer leader.
Hence, in addition to the peer leaders being essential to the success of the PLTL approach, participating
as a leader favorably impacts the leader beyond the PLTL experience. Fiorella (this volume) discusses a
research literature that documents the multiple benefits to students who prepare to teach (and actually
teach) other students.
Overall, the literature shows that PLTL has robust effects on academic achievement and attitudinal
outcomes for students in STEM courses. The sizable number of research studies presents a strong case
for generalizability; there is a likelihood that would-be PLTL adopters may anticipate gains in student
attitude and academic achievement across STEM disciplines and institution types. In addition, because
this approach adopts cooperative learning, students also gain experience working cooperatively in
teams, improving their oral-communication skills, discussing their problem-solving processes and
reasonings, and becoming responsible for their own learning. Last, the students who engage as peer
leaders report learning leadership and interpersonal skills (e.g., leading groups effectively, handling
group discussions, and creating supportive environments), improving their own study skills and content
knowledge, and increasing their self-confidence. For the remainder of this chapter, we discuss practical
advice for implementing PLTL by combining findings from the research literature with our own
experiences in initiating and sustaining PLTL programs.
Education Implications: Using PLTL in Your Teaching

A strength of PLTL is the flexibility in design which enables it to be implemented in a variety of classroom
settings. First, unlike many teaching techniques in the literature, PLTL is scalable to meet the demands
of varying class sizes; an instructor can recruit and train the number of peer leaders needed to maintain
a reasonable student-to-peer-leader ratio. Second, PLTL can be enacted during a weekly class session,
while the remaining class sessions continue with instruction as normal, or as a supplementary session
available in addition to regularly scheduled class sessions. Third, PLTL can support students’ engagement
with instructional materials of the instructor’s choosing; it is not reliant on a set of prescribed
instructional activities or classroom materials. Fourth, PLTL is low cost to enact and sustain. The primary
cost involves finding a means to incentivize students to take on the role of a peer leader and an
associated cost is the time of the faculty member to train the peer leaders. Some programs offer
financial compensation to the peer leaders for their time spent while working with students in the target
class. Other programs offer course credit, such as an upper-level elective course, in lieu of financial
compensation for students who take on the role of peer leader. The latter approach offers the benefit of
formally recognizing the faculty member’s efforts to train the peer leaders as the instructor of record in
the training course; however, it also encumbers the department with an additional course to cover.
Institutional costs in enacting PLTL should be considered in combination with the potential benefits
observed. The current research literature makes a compelling case that the enactment of PLTL can
improve student success in the target course. As discussed before, there are also benefits in addition to
academic performance that are evident during the enactment of PLTL. As PLTL supports students
discussing course content in a group setting, students routinely engage in communicating course
content more than they would with lecture-oriented instruction. PLTL is also unique in that it offers
244
students, who serve as peer leaders, opportunities to gain leadership experience and confidence in their
contextual knowledge. Students engaging in peer leading at the settings of the authors commonly
progress from expressing doubts about their own knowledge and skills to surprise at the usefulness of
what they have learned in past courses.
The cost-to-benefit ratio can also become substantially more favorable in instances where PLTL is used
with a course that has multiple class sections offered concurrently within one semester, as is commonly
done with introductory STEM courses. Once peer leaders are trained, they can support students from
more than one class thereby impacting more students without increasing the faculty time commitment
to training. We typically have peer leaders support at most students from two different classes
representing about 3 hours per week with students. If the number of concurrent classes is more than
two, then the number of peer leaders enrolled can be increased to match the demand. Additionally, by
supporting multiple classes, the instructional efforts associated with PLTL can also be shared across the
instructors teaching the classes. Instructors can co-teach the training sessions or divide up other
responsibilities such as preparation of student materials or engaging in peer-leader supervision. Thus, by
using PLTL to support multiple class sections within a course, the benefits can be expanded by
supporting students in more classes, while the costs to each individual instructor can be reduced by
sharing the work across the group of instructors.
To enact PLTL, our experience has found that ongoing training and supervision of peer leaders is
essential. Throughout the enactment of PLTL, we recommend that peer leaders are trained at the same
frequency with which they meet with students. One example would be a weekly schedule where peer
leaders meet for training early in the week and meet with students later in the week. Each training
session can then incorporate the content the students would see later in the week, which ensures peer-
leader readiness to facilitate group discussion of the content, the opportunity to discuss challenges
associated with that week’s particular content, and the opportunity to discuss techniques about
facilitating their group. Peer-led sessions with students should be supervised by a readily available
instructor who is familiar with the content. This instructor can redirect resources if a peer leader is tardy
or absent, help if a peer leader has questions about content or process, and can serve as a resource
when unexpected events occur.
Throughout the PLTL experience, a key feature of peer-leader training is the promotion of reflective
practice. One primary method for doing this is through observing peer leaders while they are leading the
student sessions. Peer leaders can initially be nervous with the thought of being observed. We found it
helpful to describe ahead of time the nature of the behaviors and space preparation that would be
observed. Developing and sharing an observation rubric may help to assuage their anxieties in this
regard. An example of such a rubric is shown in Table 1.
245
Table 1
Example Observation Rubric for Peer Leaders
No Infrequent or
Activity Evident
evidence incomplete use
Room is set-up to facilitate group-work
Addresses the entire group of students
Visits all groups in a timely fashion
Makes explanations visual (where appropriate)
Encourages groups to communicate with other groups

(either at the board or directly with another group)
Professionalism
After each observation, we recommend that the instructor or supervisor 1) debrief with the peer leader
regarding any observed strengths and areas for improvement and 2) follow-up with subsequent
observations to promote the skill sets on which the leader and supervisor had chosen to focus.
Observations can also be conducted by other peer leaders. They report that it is valuable to attend
sessions of other peer leaders to observe different peer-leading techniques.
Reflective practice can also be promoted by assigning peer leaders regular journal assignments where
leaders are tasked with reflecting on past sessions. Some example journal themes to promote such
reflection include: “How can you tell if a student understands a concept, and what evidence informs
your decision?” or “What steps do you take in your session to promote all students participating, and
how would you evaluate the effectiveness of these steps?” Additionally, it is essential to train peer
leaders to facilitate the group discussion rather than instructing students directly. This point can be
established at the very beginning by discussing constructivist and generative learning (Wittrock, 1992)
theories as the underlying philosophy and why it is necessary to promote active learning and discussion
when possible. This philosophy can be translated into a guiding principle that peer leaders can enact: if
students are stuck, the peer leader should identify the smallest piece of information that can help them
progress or ask them probing questions about the concept to assist them in moving forward; if students
are progressing, the peer leader should seek to challenge the students to demonstrate their
understanding.
Another essential feature to PLTL adoption is the integration of the peer-led sessions within the overall
course structure. Students are more likely to engage fully during their peer-led sessions when the
sessions are clearly seen as part of the course in which they are enrolled. To promote this integration,
course instructors can reference the peer-led sessions during the other elements of course instruction.
This referencing can be done in an anticipatory fashion by informing students they will see additional
examples of a particular concept in this week’s peer-led sessions; or the referencing can be done in a
retrospective fashion by discussing problems from a recent PLTL session in which students struggled, or
by instructors introducing new content built upon material covered during PLTL sessions.
Ensuring that the materials on which students are to work during PLTL sessions are relevant to the
course content, and that they are transferable topics that you wish to assess, is critical. Informing
students of this correspondence and demonstrating your planned integration of the peer-led sessions
into the course design is another opportunity to attain student buy-in.
246
As with any adoption of teaching techniques, potential adopters are encouraged to take a long view in
determining whether the PLTL implementation is working. The initial implementation of PLTL may face
unique challenges as care has to be given to create a system that promotes student engagement and
peer leadership and fits into the classroom culture, instructional goals, and institutional mission. It is
likely that during the first term of implementing PLTL unforeseen challenges will require adaptation in
future classes. Ultimately, to determine if sustained adoption is warranted, we encourage instructors to
reflect on the growth in facilitating PLTL across semesters as a means of avoiding overly interpreting the
highs and lows that inevitably occur from one week to the next.
Consider also during these early enactments a range of potential student outcomes, including academic
success, student engagement, students’ sense of belonging, and quality of discussions. In our
experiences, we have found PLTL to be a deeply rewarding experience via witnessing highly productive
conversations among students and between students and peer leaders, and in promoting highly
important career skills among students who take on the peer-leader role. We would encourage
instructors and those who support instructors and students in STEM courses to consider whether PLTL
can serve their own classroom settings.
Note
Corresponding Author: Regina Frey, [email protected]
References
Akinyele, A. F. (2010). Peer-Led team learning and improved performance in an allied health chemistry
course. Chemical Educator, 15, 353-360. https://doi.org/10.1021/acs.jchemed.8b00375
Baez-Galib, R., Colon-Cruz, H., Resto, W., & Rubi, M. R. (2005). Chem-2-Chem: A one-to-one supportive
learning environment for chemistry. Journal of Chemical Education, 82(12), 1859-1863.
https://doi.org/10.1021/ed082p1859
Barkley, E. F., Major, C. H., & Cross, K. P. (2014). Collaborative learning teaching: A handbook for college
faculty (2nd ed.). John Wiley & Sons.
Barrasso, A. P., & Spilios, K. E. (2021). A scoping review of literature assessing the impact of the learning
assistant model. International Journal of STEM Education, 8(1), 1-18.
Batz, Z., Olsen, B. J., Dumon, J., Dastoor, F., & Smith, M. K. (2015). Helping struggling students in
introductory biology: A peer-tutoring approach that improves performance, perception, and
retention. CBE - Life Sciences Education, 14(Summer 2015), 1-12.
https://doi.org/10.1187/cbe.14-08-0120
Borda, E., Schumacher, E., Hanley, D., Geary, E., Warren, S., Ipsen, C., & Stredicke, L. (2020). Initial
implementation of active learning strategies in large, lecture STEM courses: Lessons learned
from a multi-institutional, interdisciplinary STEM faculty development program. International
Journal of STEM Education, 7(1), 1-18. https://doi.org/10.1186/s40594-020-0203-2
Chan, J. Y. K., & Bauer, C. F. (2014). Identifying at-risk students in general chemistry via cluster analysis
of affective characteristics. Journal of Chemical Education, 91(9), 1417-1425.
https://doi.org/10.1021/ed500170x
247
Chase, A., Rao, A. S., Lakmala, P., & Varma-Nelson, P. (2020). Beyond content knowledge: transferable
skills connected to experience as a peer-leader in a PLTL program and long-term impacts.
International Journal of STEM Education, 7(article 29). https://doi.org/10.1186/s40594-020-
00228-1
Eren-Sisman, E. N., Cigdemoglu, C., & Geban, O. (2018). The effect of peer-led team learning on
undergraduate engineering students’ conceptual understanding, state anxiety, and social
anxiety. Chemistry Education Research and Practice, 19(3), 694-710.
http://dx.doi.org/10.1039/C7RP00201G
mathematics. Proceedings of the National Academies of Science, 111(23), 8410-8415.
Frey, R. F., Fink, A., Cahill, M. J., McDaniel, M. A., & Solomon, E. D. (2018). Peer-led team learning in
general chemistry I: Interactions with identity, academic preparation, and a course-based
intervention. Journal of Chemical Education, 95(12), 2103-2113.
https://doi.org/10.1021/acs.jchemed.8b00375
Gosser, D., Roth, V., Gafney, L., Kampmeier, J., Strozak, V., Varma-Nelson, P., Radel, S., & Weiner, M.
(1996). Workshop chemistry: Overcoming the barriers to student success. The Chemical
Educator, 1, 1-17. https://doi.org/10.1007/s00897960002a
Gosser Jr., D. K., Kampmeier, J. A., & Varma-Nelson, P. (2010). Peer-led team learning: 2008 James Flack
Norris Award address. Journal of Chemical Education, 87(4), 374-380.
https://doi.org/10.1021/ed800132w
Hockings, S. C., DeAngelis, K. J., & Frey, R. F. (2008). Peer-led team learning in general chemistry:
implemenation and evaluation. Journal of Chemical Education, 85(7), 990-996.
http://pubs.acs.org/doi/pdfplus/10.1021/ed085p990
Kudish, P., Shores, R., McClung, A., Smulyan, L., Vallen, E. A., & Siwicki, K. K. (2016). Active learning
outside the classroom: Implementation and outcomes of peer-led team-learning workshops in
introductory biology. CBE - Life Sciences Education, 15(article 31).
Kulatunga, U., Moog, R. S., & Lewis, J. E. (2013). Argumentation and participation patterns in general
chemistry peer-led sessions. Journal of Research in Science Teaching, 50(10), 1207-1231.
Lewis, S. E. (2011). Retention and reform: An evaluation of peer-led team learning. Journal of Chemical
Education, 88(6), 703-707. https://doi.org/10.1021/ed100689m
Lewis, S. E. (2014). Investigating the longitudinal impact of a successful reform in general chemistry on
student enrollment and academic performance. Journal of Chemical Education, 91, 2037-2044.
https://doi.org/10.1021/ed500404q
Lewis, S. E., & Lewis, J. E. (2008). Seeking effectiveness and equity in a large college chemistry course: An
HLM investigation of Peer-Led Guided Inquiry. Journal of Research in Science Teaching, 45(7),
794-811. https://doi.org/10.1002/tea.20254
248
Liu, Y., Raker, J. R., & Lewis, J. E. (2018). Evaluating student motivation in organic chemistry courses:
moving from a lecture-based to a flipped approach with peer-led team learning. Chemistry
Education Research and Practice, 19(1), 251-264. https://doi.org/10.1039/C7RP00153C
Merkel, J. C., & Brania, A. (2015). Assessment of peer-led team learning in calculus I: A five-year study.
Innovative Higher Education, 40, 415-428. https://doi.org/10.1007/s10755-015-9322-y
Metcalfe, J. (2009). Metacognitive judgments and control of study. Current Directions in Psychological
Science, 18(3), 159-163. https://doi.org/10.1111%2Fj.1467-8721.2009.01628.x
Metcalfe, J., & Jacobs, W. J. (2010). People’s study time allocation and its relation to animal foraging.
Behavioral Processes, 83(2), 213-221. https://doi.org/10.1016%2Fj.beproc.2009.12.011
Metcalfe, J., & Kornell, N. (2005). A region of proximal learning model of study time allocation. Journal of
memory and language, 52(4), 463-477. https://doi.org/10.1016/j.jml.2004.12.001
Mitchell, Y. D., Ippolito, J., & Lewis, S. E. (2012). Evaluating Peer-Led Team Learning across the two
semester General Chemistry sequence. Chemistry Education Research and Practice, 13(3), 378-
383. http://dx.doi.org/10.1039/C2RP20028G
National Academies of Sciences, Engineering, and Medicine. 2018. How people learn II: Learners,
contexts, and cultures. Washington, DC: The National Academies Press.
Rahman, T., & Lewis, S. E. (2020). Evaluating the evidence base for evidence-based instructional
practices in chemistry through meta-analysis. Journal of Research in Science Teaching, 57(5),
765-793. https://doi.org/10.1002/tea.21610
Repice, M. D., Sawyer, R. K., Hogrebe, M. C., Brown, P. J., Luesse, S. B., Gealy, D. J., & Frey, R. F. (2016).
Talking through the problems: A study of discourse in peer-led small groups. Chemistry
Education Research and Practice, 17(3), 555-568. http://dx.doi.org/10.1039/c5rp00154d
Robert, J. R., Lewis, S. E., Oueini, R., & Mapugay, A. (2016). Coordinated implementation and evaluation
of flipped classes and peer-led team learning in general chemistry. Journal of Chemical
Education, 93(12), 1993-1998. https://doi.org/10.1021/acs.jchemed.6b00395
Rodriguez, J. M. G., Hunter, K. H., Scharlott, L. J., & Becker, N. M. (2020). A review of research on process
oriented guided inquiry learning: Implications for research and practice. Journal of Chemical
Education, 97(10), 3506-3520. https://doi.org/10.1021/acs.jchemed.0c00355
Sawyer, K., Frey, R., & Brown, P. (2013). Knowledge Buidling Discourse in Peer-Led Team Learning (PLTL)
Groups in First-year General Chemistry. In D. D. Suthers, K. Lund, C. P. Rose, C. Teplovs, & N. Law
(Eds.), Productive Multivocality in the Analysis of Group Interactions (pp. 191-204). Spring
Science + Business Media.
Shields, S. P., Hogrebe, M., Spees, W., Handlin, L., Noelken, G., Riley, J., & Frey, R. F. (2012). A Transition
Program for Underprepared Students in General Chemistry: Diagnosis, Implementation, and
Evaluation. Journal of Chemical Education, 89(8), 995-1000. https://doi.org/10.1021/ed100410j
Slavin, R. E. (1996). Research on Cooperative Learning and Achievement: What We Know, What We
Need to Know. Contemporary Educational Psychology, 21(1), 43-69.
https://doi.org/10.1006/ceps.1996.0004
Smith, J., Wilson, S. B., Banks, J., Zhu, L., & Varma-Nelson, P. (2014). Replicating Peer-Led Team Learning
in cyberspace: Research, opportunities and challenges. Journal of Research in Science Teaching,
51(6), 714-740. https://doi.org/10.1002/tea.21163
249
Snyder, J. J., Carter, B. E., & Wiles, J. R. (2015). Implementation of the Peer-Led Team-Learning
Instructional Model as a Stopgap Measure Improves Student Achievement for Students Opting
Out of Laboratory. CBE - Life Sciences Education, 14(1), 1-6. https://doi.org/10.1187/cbe.13-08-
0168
Stains, M., Harshman, J., Barker, M. K., Chasteen, S. V., Cole, R., DeChenne-Peters, S. E., Eagan, M. K., Jr,
Esson, J. M., Knight, J. K., Laski, F. A., Levis-Fitzgerald, M., Lee, C. J., Lo, S. M., McDonnell, L. M.,
McKay, T. A., Michelotti, N., Musgrove, A., Palmer, M. S., Plank, K. M., Rodela, T. M., . . . Young,
A. M. (2018). Anatomy of STEM teaching in North American universities. Science, 359(6383),
1468–1470. https://doi.org/10.1126/science.aap8892
Streitwieser, B., Light, G., & Pazos, P. (2010). Entering the Community of Practitioners: A Science
Research Workshop Model. Change, 42(3), 17-23.
Szteinberg, G., Repice, M. D., Hendrick, C., Meyerink, S., & Frey, R. F. (2020). Peer leader reflections on
promoting discussion in peer group-learning sessions: Reflective and practiced advice through
collaborative annual peer-advice books. CBE - Life Sciences Education, 19(1), ar2.
Theobald, E. J., Hill, M. J., Tran, E., Agrawal, S., Arroyo, E. N., Behling, S., Chambwe, N., Cintrón, D. L.,
Cooper, J. D., Dunster, G., Grummer, J. A., Hennessey, K., Hsiao, J., Iranon, N., Jones, L., II, Jordt,
H., Keller, M., Lacey, M. E., Littlefield, C. E., . . . Freeman, S. (2020). Active learning narrows
achievement gaps for underrepresented students in undergraduate science, technology,
engineering, and math. PNAS Proceedings of the National Academy of Sciences of the United
States of America, 117(12), 6476–6483. https://doi.org/10.1073/pnas.1916903117
Tien, L. T., Roth, V., & Kampmeier, J. A. (2002). Implementation of a Peer-Led Team Learning
Instructional Approach in an Undergraduate Organic Chemistry Course. Journal of Research in
Tien, L. T., Roth, V., & Kampmeier, J. A. (2004). A Course to Prepare Peer Leaders to Implement a
Student-Assisted Learning Method. Journal of Chemical Education, 81(9), 1313-1321.
https://doi.org/10.1021/ed081p1313
Varma-Nelson, P., Cracolice, M. S., & Gosser, D. K. (2004). Peer-Led Team Learning: A Student-Faculty
Parternship for Transforming the Learning Environment. In Invention and Impact: Building
Excellence in Undergraduate Science, Technology, Engineering and Mathematics (STEM)
Education (pp. 43-48). American Association for the Advancement of Science.
Wamser, C. C. (2006). Peer-Led Team Learning in Organic Chemistry: Effects on Student Performance,
Success, and Persistence in the Course. Journal of Chemical Education, 83(10), 1562-1566.
http://pubs.acs.org/doi/pdfplus/10.1021/ed083p1562
White, P., Rowland, A. B., & Pesis-Katz, I. (2012). Peer–Led Team Learning Model in a Graduate–Level
Nursing Course. Journal of Nursing Education 51(8), 471-475.
https://doi.org/10.3928/01484834-20120706-03
Wilson, S. B., & Varma-Nelson, P. (2016). Small Groups, Significant Impact: A Review of Peer-Led Team
Learning Research with Implications for STEM Education Researchers and Faculty. Journal of
Chemical Education, 93(10), 1686-1702. https://doi.org/10.1021/acs.jchemed.5b00862
Wilson, S. B., & Varma-Nelson, P. (2018). Characterization of First-Semester Organic Chemistry Peer-Led
Team Learning and Cyber Peer-Led Team Learning Students’ Use and Explanation of Electron-
250
Pushing Formalism. Journal of Chemical Education, 96, 25-34.
https://doi.org/10.1021/acs.jchemed.8b00387
Wittrock, M. (1992). Generative Learning Processes of the Brain. Educational Psychologist, 27(4), 531-
541. https://doi.org/10.1207/s15326985ep2704_8
251
In Their Own Words Frequent Quizzing Accelerates Classroom
Learning
You to Know About Why and How Chunliang Yang
Your Academic Setting Beijing Normal University
Editors
David R. Shanks
Lauren L. Kordonowy
Victor A. Benassi
University College London
Wenbo Zhao, Tian Fan, and Liang Luo

Beijing Normal University
Overview
Although frequent testing (i.e., practice retrieval) seems to be unwelcome to many students, a wealth of
studies has established that taking a test is one of the most effective strategies to facilitate learning.
Specifically, over the last century, hundreds of laboratory and classroom studies on practice retrieval
have documented that retrieving information from memory can more effectively facilitate learning and
consolidate long-term retention than restudying the same material, a phenomenon that goes by a range
of names, including: the testing effect, the retrieval practice effect, test-enhanced learning, test-
potentiated learning, and retrieval-enhanced learning (for reviews, see Rawson & Dunlosky, 2011;
Roediger & Butler, 2011; Yang et al., 2021). In addition, testing tends to be more beneficial than many
other elaborative strategies, such as note-taking (Heitmann et al., 2018), concept-mapping (Karpicke &
Blunt, 2011), and self-explaining (Larsen et al., 2013).
It seems compelling that learners should seek to utilize testing for optimizing their learning outcomes.
However, both learners and instructors tend to underappreciate its merits despite its broad benefits,
leading to its underemployment in educational settings. Besides lack of meta-awareness, other reasons
for its practical underemployment will be discussed below. The main point here is that, given its efficacy,
testing has not been as widely implemented in the classroom as it deserves to be. This is a concrete
example of the well-known “know-do gap” between what we learn from cognitive research and what
we implement in real situations (cf. Rowe & Hattie, this volume). The current chapter aims to help
bridge this gap by providing practitioners with a summary of empirical findings on the classroom testing
effect, providing some responses to practitioners’ potential concerns about classroom quizzing, and
offering some suggestions about how to augment the enhancing effect of classroom quizzing.
Empirical Findings on Test-Enhanced Learning

Although testing has been typically considered as an assessment of learning, it is also an effective
assessment for learning. Ample laboratory and classroom studies have demonstrated that testing not
only consolidates long-term retention of studied information, a phenomenon termed the backward
testing effect (for reviews, see Adesope et al., 2017; Roediger & Karpicke, 2006a; Roediger et al., 2011;
Rowland, 2014), but also effectively potentiates subsequent learning of new information, a
phenomenon referred to as the forward testing effect (for reviews, see Chan et al., 2018; Pastötter &
Bäuml, 2014; Yang et al., 2017).
Test-Enhanced Consolidation of Studied Information
Roediger and Karpicke (2006b) conducted a classic study demonstrating the backward testing effect.
Undergraduates studied two texts, with one text read twice and the other read once and tested once. In
252
a final test administered one week later, the tested passage (56%) was substantially better recalled than
the restudied one (42%), reflecting that testing more effectively consolidates long-term retention than
restudying.
The backward testing effect has also been documented in classroom studies. For instance, in a Brain and
Behavior course, McDaniel et al. (2007) had students from the University of New Mexico read about 40
pages of textbook materials each week. After reading the weekly course assignments, students either
reread ten facts selected from the reading materials or took a short answer test on these facts with
corrective feedback (i.e., the correct answers) provided after they answered each question. This course
lasted for 6 weeks, and students took two exams across the term, with one administered at Week 3 and
the other at Week 6. In these exams, the 60 reread and 60 tested facts were assessed in a multiple-
choice format. Exam performance for the tested facts (57%) was substantially better than that for
reread facts (47%).
In short, testing of studied information can be employed as a practical strategy to consolidate long-term
retention.
Test-Potentiated Learning of New Information
Besides the benefit that testing confers on long-term retention of studied/tested information, an
emerging body of recent studies has found that testing on previously studied information can also
promote subsequent learning of new information. The forward (prospective) benefits of testing on new
learning reliably generalize to different types of study materials, such as lecture videos, text passages,
artists’ painting styles, and foreign word translations, and the effect extends to different populations,
such as children, college students, older adults, and patients with traumatic brain injury (for reviews, see
Chan et al., 2018; Pastötter & Bäuml, 2014; Yang & Shanks, 2018).
Szpunar et al. (2013) provided an illustration. In their experiment, two (Restudy vs. Test) groups of
undergraduates watched a 4-segment Introductory Statistics lecture video. After watching each of
Segments 1-3, the Restudy group restudied the six key concepts delivered in the just-studied segment.
By contrast, the Test group was tested on these six key concepts without corrective feedback (e.g., What
is the relation between a population and a sample?). Then, both groups were tested on Segment 4 after
watching it. In brief, both groups watched the same lecture segments, and the only difference between
groups was that the Restudy group restudied, whereas the Test group was tested on Segments 1-3.
As a result, the Test group correctly answered more questions (89%) than the Restudy group (65%) in
the test on Segment 4, reflecting a forward benefit of testing (that is, interim testing on Segments 1-3,
by comparison with restudying, boosted subsequent learning of Segment 4). Szpunar and colleagues
also found that the Test group (19%) reported fewer instances of mind wandering than the Restudy
group (39%) when they were occasionally prompted to report whether they were mind wandering or
focusing on the lecture during the study phase. Furthermore, the Test group took more notes than the
Restudy group while watching the lecture video.
Underemployment of Test-Enhanced Learning and Potential Concerns

About Quizzing
Although, as discussed above, the benefits of testing are substantial, researchers have frequently
expressed dismay that learners, instructors, and policy-makers tend not to appreciate these benefits and
that test-enhanced learning has not been fully applied in the classroom to boost students’ attainment
(Roediger & Karpicke, 2006b; cf. Rowe & Hattie, this volume). For example, Tullis and Maddox (2020)
asked 84 middle and 228 high school students to report their most frequently used strategies. For these
253
students, 43% reported frequently rereading (restudying), whereas only 16% reported using retrieval
practice. Similar to students, some teachers may also lack full appreciation of the benefits of test-
enhanced learning. For instance, Morehead et al. (2016) asked 146 teachers at Colorado State University
to explain why they thought students should test themselves. Sixty-eight percent replied that students
should test themselves to figure out how well they have mastered the material, and only 19% reported
that students will learn more through testing than through rereading. These findings reflect that a
majority of these teachers regard testing primarily as an assessment tool to identify knowledge gaps
rather than an effective strategy to boost learning.
Besides metacognitive unawareness, underemployment of test-enhanced learning might derive from
many other sources. For instance, teachers may worry that frequent quizzing consumes in-class time
and/or provokes test anxiety, which in turn impairs students’ academic performance. Below, we list
several concerns that teachers may have about the potential drawbacks of quizzing and provide
responses to these concerns.
Quizzing Borrows Time from Other Teaching Activities
One obstacle for implementing classroom quizzes is that administering them is time-consuming and
scoring can sometimes be excessively demanding. Because quizzing draws time away from didactic
teaching and curricula are typically too full to spare time for quizzing, some instructors may prefer to
minimize the uses of classroom quizzing. However, a recent meta-analysis by Yang et al. (2021),
combining results from nearly 50,000 students and over 200 research reports, demonstrated that
frequent classroom quizzing enhances students’ academic achievement compared to practice-as-usual.
Such findings suggest that even though quizzing borrows time from didactic teaching, the tradeoff is
worthwhile.
To avoid the tradeoff between teaching and quizzing, teachers can consider administering quizzes
outside the classroom. Some studies have investigated the impacts of out-of-classroom quizzes on
students’ attainment (e.g., Grimstad & Grabe, 2004; Griswold, this volume; Marden et al., 2013). In
these studies, quizzes were assigned as homework or made available online, and students were allowed
to answer them at any time and place they chose. Yang et al.’s (2021) meta-analysis found that quizzes
administered out of the classroom also significantly boost students’ attainment, although not quite as
effectively as ones administered in-class.
On the bright side, with the development of new technologies, more and more modes are available to
make it easy to administer and score quizzes, such as clicker response systems, course management
systems (e.g., Blackboard), collaborative learning platforms (e.g., PeerWise), and smartphones. Apart
from these technologies, Pyc et al. (2014) suggested alternative ways to easily incorporate quizzes into
classroom instruction, such as orally asking questions and waiting for students to generate responses, or
providing students with colored index cards and asking them to select a colored flashcard and hold it up
to respond to true/false or multiple-choice questions. All these techniques are virtually cost-free and
allow teachers to easily implement immediate quizzes. Furthermore, Yang et al.’s (2021) meta-analysis
found that quiz administration mode (e.g., paper-and-pen, web-based, oral, or clicker response system)
has minimal influence on the magnitude of test-enhanced learning, implying that the enhancing effect
of classroom quizzing is independent of administration modality.
In short, although quizzing borrows time from other teaching activities, the tradeoff is worthwhile.
Does Frequent Quizzing Provoke Test Anxiety?
Another legitimate concern that some teachers may have, which makes them reluctant to incorporate
frequent quizzing into their teaching, is that quizzing may provoke test anxiety, which causes a variety of
254
learning detriments, such as difficulty concentrating and poor test performance. However, two sets of
research findings should go some way towards allaying this concern.
The first set of evidence is that low-stake (or no-stake) tests, which arouse minimal test anxiety, can
effectively promote learning. In their meta-analysis, Yang et al. (2021) assessed the moderating role of
stake-level on test-enhanced learning. Specifically, Yang and colleagues divided 295 classroom
experiments into two sub-categories according to the stake-level (i.e., whether quiz performance was
incorporated into course grades) of quizzes: high (Yes) vs. low (No). The results showed that both high-
and low-stake quizzes significantly enhanced students’ exam performance, and there was no significant
difference between high- and low-stake quizzes, revealing that even low- or no-stake quizzes can
facilitate learning (for related findings, see Davis, 2013). Indeed, some studies have even reported that
low-stake quizzes are more beneficial than high-stake ones (e.g., Schrank, 2016).
The second set of findings is that frequent testing has little impact on or even reduces (rather than
increases) test anxiety. For instance, in a large sample study (over 1,000 college participants), Yang et al.
(2020) observed that interpolating tests across a study phase has minimal influence on participants’ test
anxiety. Szpunar et al. (2013) found that frequent tests significantly relieve test anxiety (for related
findings, see Khanna, 2015). Furthermore, in a large-scale survey conducted by Agarwal et al. (2014),
72% of 1,306 middle and 102 high school students reported that frequent quizzes made them less
anxious about exams, with only 8% reporting the opposite. In another survey conducted by Sullivan
(2017), about 91% of 353 college students agreed with the statement The option to retake a quiz
reduced my test anxiety, and only 3.1% disagreed.
In education, testing is frequently criticized in the media and by parents, largely due to standardized
tests with high stake levels which can undeniably induce high test-anxiety (Pyc et al., 2014). However,
what we recommend here is frequent low-stake quizzing or testing, which is very different from high-
stake standardized tests. Furthermore, as summarized above, low-stake quizzes facilitate classroom
learning without causing test anxiety.
Does Test-Enhanced Learning Generalize to Different Test Formats, Subjects, and
Populations?
Teachers may wonder “Does test-enhanced learning work in my class, for the subject I’m teaching, for
my particular students? Are all test formats valid?” The generalizability of test-enhanced learning to
different test formats, subjects, and populations (i.e., students at different education levels) has been
repeatedly investigated in previous studies. Through integrating research results from previous studies,
Yang et al.’s (2021) meta-analysis found that all kinds of test formats (e.g., matching, fill-in-the-blank,
short answer, multiple-choice, cued recall, free recall, and so on) reliably enhance learning (for related
discussion, see Pyc et al., 2014); the classroom testing effect generalizes across all subject categories for
which sufficient data are available, including Accounting/Business/Finance, Biology, Chemistry,
Geography, Education, History, Medical/Nursing/ Physiology, Language/Reading/Vocabulary,
Psychology, and so on. Test-enhanced learning works across different education levels, including
elementary school, middle school, high school, and university/college. However, the evidence for
benefits in adult education is less clear, due to a very small number of published studies. These findings
jointly establish the stability and generalizability of test-enhanced learning to different test formats,
subjects, and populations (Yang et al., 2021).
Does Quizzing Only Enhance “Inert Knowledge”?
Testing (quizzing) has been occasionally met with a criticism that it is a “drill-and-kill” strategy, which
only promotes retention of “inert knowledge” that cannot be applied to solve new problems in new
255
contexts. However, this criticism has been repeatedly contradicted by studies showing that testing can
effectively promote knowledge transfer and application (for a review, see Carpenter, 2012). For
example, Butler (2010) had students study six texts, three of which were repeatedly restudied, and the
other three repeatedly tested (e.g., What is the wing structure of a bat like relative to that of a bird?).
On a final test administered one week later, students were instructed to apply knowledge studied from
the texts to solve new inferential questions in a very different domain (e.g., How would a military
aircraft modeled after a bat wing differ from traditional aircrafts?). The results showed greater accuracy
for new inferential questions for tested passages (68%) than for restudied ones (44%), suggesting that
testing promotes far transfer knowledge application, enabling learners to solve new problems in new
contexts (for related findings, see Karpicke & Blunt, 2011).
In another study conducted by Kromann et al. (2009), 81 medical students studied resuscitation skills in
an in-hospital resuscitation course. In the intervention (testing) condition, students took a 30-minute
quiz following three and a half hours of instruction and training. By contrast, those in the control
condition received four hours of instruction and training without quizzing. Two weeks later, all students
undertook an assessment to measure their resuscitation skills, in which they were provided cardiac
arrest scenarios and required to perform the resuscitation treatment. Assessment performance in the
intervention condition (83%) was much better than in the control condition (73%), implying that testing
can enhance skill learning.
In their meta-analysis, Yang et al. (2021) coded classroom studies into three categories according to
their exam content type: Fact (i.e., memory of specific course contents, such as historical events),
Concept (e.g., concept comprehension and inductive inferences that go beyond direct experience, such
as learning rules of syntax in a foreign language course), and Problem-solving (i.e., applying learned
knowledge or skills to solve problems, such as performing resuscitation treatment in cardiac arrest
scenarios). Through integrating results of previous classroom studies, Yang et al. found that class
quizzing not only promotes learning of facts, but also facilitates concept learning (Jacoby et al., 2010;
Karpicke et al., 2014; Yang & Shanks, 2018) and enhances knowledge application in the service of
problem solving (Kromann et al., 2010; Sennhenn-Kirchner et al., 2018).
In summary, the above empirical and meta-analytic findings jointly contradict the view that testing is a
“drill-and-kill” strategy and only produces mnemonic benefits to inert knowledge. Instead, these
findings imply that testing is also an effective tool to promote knowledge transfer and application.
Does Frequent Quizzing Widen the Achievement Gap among Students?
It is reasonable for teachers to suspect that frequent quizzing may widen the achievement gap (e.g.,
between novices and experts in a subject area, less-skilled and more-skilled readers, students with
overall lower and higher academic performance). For instance, do some students benefit less from class
quizzing or does class quizzing exert detrimental effects on some students’ academic performance
because they typically perform less well on quizzes? Poor relative quiz performance may cause a sense
of failure, leading to poorer academic performance. If this is the case, might frequent quizzing widen the
achievement gap among students? Recent research demonstrates that this concern may be misplaced –
indeed, the opposite seems to be the case. For example, Agarwal et al. (2017) and Yang et al. (2020)
observed that the benefits of testing are larger for individuals with low than with high working memory
capacity. Brewer and Unsworth (2012) found that individuals with lower IQs experience a larger
enhancing effect of testing than those with higher IQs. Although high test anxiety is typically associated
with poor learning achievement, Yang et al. (2020) and Clair et al. (2020) consistently observed little
difference in the magnitude of test-enhanced learning between students with high and low levels of test
anxiety.
256
These findings point to the conclusion that students, in general, will benefit from testing. Hence,
frequent quizzing may narrow, rather than broaden, the achievement gap among students.
Optimizing the Benefits of Quizzing

The above discussion summarizes the broad benefits of classroom quizzing and provides responses to
some of the concerns that teachers might have about frequent quizzing. Below, we offer several
suggestions about when and how to implement quizzes to maximize the magnitude of test-enhanced
learning.
Administer Quizzes After Teaching
In previous studies, quizzes have occasionally been presented before formal teaching, with to-be-taught
content tested prior to teaching. Thus, a student might be quizzed about the causes of an historical
event prior to a lesson on those causes. Under such circumstances, quiz performance is likely to be quite
poor, compared to the more standard procedure in which quizzes are completed at the end of a class
with taught content assessed in these post-class quizzes. Surprisingly, pre-class quizzes have been found
to enhance learning. Although some recent experiments (e.g., Pan & Sana, 2021) have found pre-testing
to be more effective than post-testing, Yang et al.’s (2021) meta-analysis aggregating all available
evidence up to 2019 found that the enhancing effect of post-class quizzes is nearly triple that of pre-
class quizzes. Hence, instructors are advised to administer quizzes after formal lecturing (cf. Carpenter,
et al., this volume).
Provide Corrective Feedback
It has been well established in both laboratory and classroom studies that providing corrective feedback
(i.e., the correct answer to a given test question) is a fortifier of test-enhanced learning since corrective
feedback enables learners to detect and correct wrong answers. Many laboratory and classroom studies
have shown that the memorial benefits of testing are stronger when followed by corrective feedback
(e.g., Bangert-Drowns et al., 1991; Kang et al.,, 2007; Vojdanoska et al., 2010). Two meta-analyses also
reached the same conclusion. Rowland (2014), who synthesized results from 159 laboratory
experiments, found that the magnitude of the testing effect is doubled when feedback is provided, in
comparison with not providing feedback. More recently, Yang et al. (2021), who integrated results from
222 classroom studies, observed that providing corrective feedback following quizzing increased the
magnitude of quiz-induced enhancement by a factor of about 1.5. It is indeed unusual to administer a
quiz without providing corrective feedback in the classroom (Yang et al., 2021).
These consistent findings lead to the strong recommendation that instructors offer corrective feedback
after quizzing.
Administer Repeated Quizzing
Although it has been widely documented that a single test is sufficient to enhance memory compared to
restudying, many laboratory studies have observed that repeated tests (i.e., with studied content tested
repeatedly) produce a larger enhancing effect on knowledge retention and transfer than a single test
(e.g., Butler, 2010; Dunlosky et al., this volume; Roediger & Karpicke, 2006b). The enhancing effect of
repeated tests has been re-confirmed by many classroom studies. Moreover, Yang et al.’s (2021) meta-
analysis coded the number of test repetitions (i.e., how many times the studied information was tested),
and conducted analyses to quantify the relation between the magnitude of test-enhanced learning and
the number of test (quiz) repetitions. The results showed a clear trend that the more occasions on which
class content is quizzed, the more effectively quizzing aids exam performance.
257
These findings endorse a recommendation that teachers should consider incorporating repeated
quizzing on key course concepts into classroom practice. For instance, teachers can test key concepts
repeatedly in quizzes administered immediately after the class, one week later, and one month later. Of
course, teachers should also take the tradeoff between repeated quizzing and didactic teaching into
consideration.
Administer Long-Term Quizzing Interventions
Yang et al.’s (2021) meta-analysis investigated how the effectiveness of test-enhanced learning varies as
a function of quiz treatment duration. Specifically, classroom studies were classified into four sub-
categories according to the duration of their quizzing intervention: Single class (i.e., studies which only
administered a quiz in a single class), < Semester (i.e., studies which implemented the quiz treatment for
less than a semester, but longer than a single class), = Semester (i.e., studies which included quizzes
across a whole semester), and > Semester (i.e., studies which included quizzes across more than a
semester, such as across an academic year). Integrating results across 573 classroom experiments, Yang
and colleagues found that the longer the quizzing intervention, the greater the magnitude of test-
enhanced learning.
Test-enhanced engagement and learning motivation can account for the positive relationship between
the classroom testing effect and treatment duration. For instance, Schrank (2016) found that about 90%
of students in a daily-exam-class attended his Introductory Sociology class, and attendance rates were
maintained steadily across a whole semester. By contrast, attendance rates in a control class, which did
not take a daily quiz, dramatically decreased from about 90% to about 66% across the semester.
Consistently, other studies have observed that frequent tests enhance note-taking (Szpunar et al., 2013),
boost motivation (e.g., attention and study time, see Pastötter et al., 2011; Yang et al., 2017), and
reduce task-unrelated mind wandering (Jing et al., 2016; Szpunar et al., 2013).
To conclude, these findings suggest that instructors consider administering long-term quizzing
interventions, such as regular quizzes across a whole semester or an academic year.
Summary
Hundreds of studies have demonstrated that testing (quizzing) is an effective strategy to boost
classroom learning. However, learners, instructors, and policymakers tend to lack full appreciation of the
virtues of testing, leading to its underemployment in practice (see Rowe & Hattie, this volume, for a
discussion of this issue). The current chapter attempts to bridge this know-do gap and provide
practitioners with scaffolding in implementing quizzing in the classroom. In short, testing consolidates
long-term retention of studied information (i.e., the backward testing effect) and potentiates
subsequent learning of new information (i.e., the forward testing effect). Although teachers may have
concerns about potential drawbacks of quizzing, research findings run counter to many of these
concerns (see Rowe & Hattie, this volume). Instructors are recommended to administer quizzing in the
classroom, and quizzes should be implemented after formal lecturing. Following quizzing, corrective
feedback ought to be provided. Repeated quizzing and long-term quizzing treatment are more beneficial
than single/short-term quizzing treatment.
Author Note
This chapter was supported by the Natural Science Foundation of China (32000742; 32171045), the
Fundamental Research Funds for the Central Universities (2019NTSS28), and the United Kingdom
Economic and Social Research Council (ES/S014616/1).
258
References
Adesope, O. O., Trevisan, D. A., & Sundararajan, N. (2017). Rethinking the use of tests: A meta-analysis
of practice testing. Review of Educational Research, 87(3), 659-701.
Agarwal, P. K., D’Antonio, L., Roediger, H. L., McDermott, K. B., & McDaniel, M. A. (2014). Classroom-
based programs of retrieval practice reduce middle school and high school students’ test
anxiety. Journal of Applied Research in Memory and Cognition, 3(3), 131-139.
Agarwal, P. K., Finley, J. R., Rose, N. S., & Roediger, H. L. (2017). Benefits from retrieval practice are
greater for students with lower working memory capacity. Memory, 25(6), 764-771.
Bangert-Drowns, R. L., Kulik, C.-L. C., Kulik, J. A., & Morgan, M. (1991). The instructional effect of
feedback in test-like events. Review of Educational Research, 61(2), 213-238.
Brewer, G. A., & Unsworth, N. (2012). Individual differences in the effects of retrieval from long-term
memory. Journal of Memory and Language, 66(3), 407-415.
https://doi.org/10.1016/j.jml.2011.12.009
Butler, A. C. (2010). Repeated testing produces superior transfer of learning relative to repeated
studying. Journal of Experimental Psychology: Learning, Memory, and Cognition, 36(5), 1118-
1133. https://doi.org/10.1037/a0019902
Carpenter, S. K. (2012). Testing enhances the transfer of learning. Current Directions in Psychological
Science, 21(5), 279-283. https://doi.org/10.1177%2F0963721412452728
Chan, J. C., Meissner, C. A., & Davis, S. D. (2018). Retrieval potentiates new learning: A theoretical and
meta-analytic review. Psychological Bulletin, 144(11), 1111-1146.
https://doi.org/10.1037/bul0000166
Clair, B. S., Putnam, P., Miller, H. L., Larsen, R., & Jensen, J. L. (2020). Test-enhanced learning and
incentives in Biology education. CBE—Life Sciences Education, 19(3) (ar40), 1-6.
Davis, K. A. (2013). Using low- and no-stakes quizzing for student self-evaluation of readiness for exams.
International Journal of Construction Education and Research, 9(4), 256-271.
Grimstad, K., & Grabe, M. (2004). Are online study questions beneficial? Teaching of Psychology, 31(2),
143-146. https://doi.org/10.1207/s15328023top3102_8
Heitmann, S., Grund, A., Berthold, K., Fries, S., & Roelle, J. (2018). Testing is more desirable when it is
adaptive and still desirable when compared to note-taking. Frontiers in Psychology, 9, 2596.
Jacoby, L. L., Wahlheim, C. N., & Coane, J. H. (2010). Test-enhanced learning of natural concepts: Effects
on recognition memory, classification, and metacognition. Journal of Experimental Psychology:
Learning, Memory, and Cognition, 36(6), 1441-1451. https://doi.org/10.1037/a0020636
259
Jing, H. G., Szpunar, K. K., & Schacter, D. L. (2016). Interpolated testing influences focused attention and
improves integration of information during a video-recorded lecture. Journal of Experimental
Psychology: Applied, 22(3), 305-318. https://doi.org/10.1037/xap0000087
Kang, S. H. K., McDermott, K. B., & Roediger, H. L. (2007). Test format and corrective feedback modify
the effect of testing on long-term retention. European Journal of Cognitive Psychology, 19(4-5),
528-558. https://doi.org/10.1080/09541440601056620
Karpicke, J. D., & Blunt, J. R. (2011). Retrieval practice produces more learning than elaborative studying
with concept mapping. Science, 331(6018), 772-775. https://doi.org/10.1126/science.1199327
Karpicke, J. D., Blunt, J. R., Smith, M. A., & Karpicke, S. S. (2014). Retrieval-based learning: The need for
guided retrieval in elementary school children. Journal of Applied Research in Memory and
Khanna, M. M. (2015). Ungraded pop quizzes: Test-enhanced learning without all the anxiety. Teaching
of Psychology, 42(2), 174-178. https://doi.org/10.1177%2F0098628315573144
Kromann, C. B., Bohnstedt, C., Jensen, M. L., & Ringsted, C. (2010). The testing effect on skills learning
might last 6 months. Advances in Health Sciences Education, 15, 395-401.
Kromann, C. B., Jensen, M. L., & Ringsted, C. (2009). The effect of testing on skills learning. Medical
Education, 43(1), 21-27. https://doi.org/10.1111/j.1365-2923.2008.03245.x
Larsen, D. P., Butler, A. C., & Roediger, H. L. (2013). Comparative effects of test-enhanced learning and
self-explanation on long-term retention. Medical Education, 47(7), 674-682.
https://doi.org/10.1111/medu.12141
Marden, N. Y., Ulman, L. G., Wilson, F. S., & Velan, G. M. (2013). Online feedback assessments in
physiology: Effects on students' learning experiences and outcomes. Advances in Physiology
Education, 37(2), 192-200. https://doi.org/10.1152/advan.00092.2012
McDaniel, M. A., Anderson, J. L., Derbish, M. H., & Morrisette, N. (2007). Testing the testing effect in the
classroom. European Journal of Cognitive Psychology, 19(4-5), 494-513.
strategies. Memory, 24, 257-271. https://doi.org/10.1080/09658211.2014.1001992
Pan, S. C., & Sana, F. (2021). Pretesting versus posttesting: Comparing the pedagogical benefits of
errorful generation and retrieval practice. Journal of Experimental Psychology: Applied, 27(2),
Pastötter, B., & Bäuml, K. H. T. (2014). Retrieval practice enhances new learning: The forward effect of
testing. Frontiers in Psychology, 5, 286. https://doi.org/10.3389/fpsyg.2014.00286
Pastötter, B., Schicker, S., Niedernhuber, J., & Bäuml, K. H. T. (2011). Retrieval during learning facilitates
subsequent memory encoding. Journal of Experimental Psychology: Learning, Memory, and
Cognition, 37(2), 287-297. https://doi.org/10.1037/a0021801
Pyc, M. A., Agarwal, P. K., & Roediger, H. L. (2014). Test-enhanced learning. In In V. A. Benassi, C. E.
Overson, & C. M. Hakala (Eds.), Applying science of learning in education: Infusing psychological
science into the curriculum (pp. 78-90). Society for the Teaching of Psychology.
260
Rawson, K. A., & Dunlosky, J. (2011). Optimizing schedules of retrieval practice for durable and efficient
learning: How much is enough? Journal of Experimental Psychology: General, 140(3), 283-302.
Roediger, H. L., & Butler, A. C. (2011). The critical role of retrieval practice in long-term retention. Trends
in Cognitive Sciences, 15(1), 20-27. https://doi.org/10.1016/j.tics.2010.09.003
Roediger, H. L., & Karpicke, J. D. (2006a). The power of testing memory: Basic research and implications
for educational practice. Perspectives on Psychological Science, 1(2), 181-210.
Roediger, H. L., & Karpicke, J. D. (2006b). Test-enhanced learning: Taking memory tests improves long-
term retention. Psychological Science, 17(3), 249-255. https://doi.org/10.1111/j.1467-
9280.2006.01693.x
Roediger, H. L., Putnam, A. L., & Smith, M. A. (2011). Ten benefits of testing and their applications to
educational practice. Psychology of Learning and Motivation, 55, 1-36.
https://doi.org/10.1016/B978-0-12-387691-1.00001-6
Rowland, C. A. (2014). The effect of testing versus restudy on retention: A meta-analytic review of the
testing effect. Psychological Bulletin, 140(6), 1432-1463. https://doi.org/10.1037/a0037559
Schrank, Z. (2016). An assessment of student perceptions and responses to frequent low-stakes testing
in introductory sociology classes. Teaching Sociology, 44(2), 118-127.
https://doi.org/10.1177%2F0092055X15624745
Sennhenn-Kirchner, S., Goerlich, Y., Kirchner, B., Notbohm, M., Schiekirka, S., Simmenroth, A., &
Raupach, T. (2018). The effect of repeated testing vs repeated practice on skills learning in
undergraduate dental education. European Journal of Dental Education, 22(1), e42-e47.
https://doi.org/10.1111/eje.12254
Sullivan, D. (2017). Mediating test anxiety through the testing effect in asynchronous, objective, online
assessments at the university level. Journal of Education and Training, 4(2), 106-123.
https://doi.org/10.5296/jet.v4i2.10777
Szpunar, K. K., Khan, N. Y., & Schacter, D. L. (2013). Interpolated memory tests reduce mind wandering
and improve learning of online lectures. PNAS, 110(16), 6313-6317.
Tullis, J. G., & Maddox, G. B. (2020). Self-reported use of retrieval practice varies across age and domain.
Metacognition and Learning, 15, 129-154. https://doi.org/10.1007/s11409-020-09223-x
Vojdanoska, M., Cranney, J., & Newell, B. R. (2010). The testing effect: The role of feedback and
collaboration in a tertiary classroom setting. Applied Cognitive Psychology, 24(8), 1183-1195.
learning: A systematic and meta-analytic review. Psychological Bulletin, 147(4), 399–435.
Yang, C., Potts, R., & Shanks, D. R. (2017). The forward testing effect on self-regulated study time
allocation and metamemory monitoring. Journal of Experimental Psychology: Applied, 23(3),
261
Yang, C., & Shanks, D. R. (2018). The forward testing effect: Interim testing enhances inductive learning.
Journal of Experimental Psychology: Learning, Memory, and Cognition, 44(3), 485-492.
https://doi.org/10.1037/xlm0000449
Yang, C., Sun, B., Potts, R., Yu, R., Luo, L., & Shanks, D. R. (2020). Do working memory capacity and test
anxiety modulate the beneficial effects of testing on new learning? Journal of Experimental
Psychology: Applied, 26(4), 724-738. https://doi.org/10.1037/xap0000278
262
In Their Own Words Adapting Peer Instruction for Online Learning
Using a Learning Science Framework
to Apply the Science of Learning in Julie Schell
The University of Texas at Austin
Editors
Lauren L. Kordonowy
Victor A. Benassi
Andrew C. Butler
Division 2, American Psychological Association 2023 Washington University in St. Louis
Overview
Peer Instruction is an empirically and theoretically supported pedagogical method with a large body of
scholarship on implementation that spans three decades. Eric Mazur originally developed Peer
Instruction for use among Harvard physics students in 1991, but the scope and application of Peer
Instruction have expanded dramatically since its inception. Thousands of educators across disciplines
and institutional types, ranging from K-12 language educators to medical school fellowship clinicians
(see Aizer et al., 2020), use the method for its potent effects on learning and creating a positive
experience for students (Schell & Butler, 2018; Tullis & Goldstone, 2020; Vickrey et al., 2015). One
interesting recent trend is the expansion of peer learning and the need for evidence-based pedagogies
to support it in online environments. Although the trend existed before the COVID-19 pandemic, the
collective, worldwide pivot to online instruction has accelerated growth in peer-to-peer collaboration.
The extensive impact of the pandemic on education has demonstrated an unprecedented need for
research-based, adaptable pedagogies that go beyond the face-to-face classroom.
Although Mazur developed Peer Instruction for in-person courses, and the vast majority of educators
have implemented it in face-to-face environments, the pedagogical method is well suited for adaptation
to online courses. The thirty-year history of Peer Instruction includes early adoption and continuing
innovation of online technologies. Mazur initially used scantron technology to facilitate Peer Instruction
at its inception, followed by ClassTalk, an early classroom response system (Dufrense et al., 1996), and
subsequently other forms of technology, including clickers (Bruff, 2009). Indeed, the evolution of Peer
Instruction parallels the evolution of classroom response systems from wired, hand-held devices in the
late 1990s to fully online, web-based apps such as Learning Catalytics, developed by the Mazur Group in
2011. Despite the fact that Peer Instruction has always incorporated technology-enhanced tools and is
highly flexible (see Schell & Butler, 2018), it is also true that nearly all Peer Instruction research,
scholarship, and practice have focused on in-person implementation. Accordingly, we wondered: How
might educators adapt Peer Instruction for online courses while maintaining its effectiveness?
In this chapter, we explore the adaptation of Peer Instruction for three forms of online learning (100%
online, hybrid dual-mode, and asynchronous) with the goal of providing recommendations for educators
who want to use this method in online courses. In doing so, we build on previous work documenting the
benefits of Peer Instruction based on insights from the science of learning (Schell & Butler, 2018). We
begin by defining Peer Instruction and briefly reviewing its effectiveness as a teaching method. Next, we
describe the Community of Inquiry Framework (Garrison et al., 2000) to facilitate understanding of
online instructional design and to guide our subsequent discussion of evidence-based online instruction.
With this framework in place, we also consider the challenges of adaptation of Peer Instruction for the
three different aforementioned online course formats. In our discussion of the challenges and potential
263
solutions for adapting Peer Instruction to these online learning environments, we draw upon theory and
research from the science of learning and motivation as well as previous research on the method itself.
We also offer practical recommendations for educators who want to continue or start using Peer
Instruction in these three online formats or other related modes. The overarching goal of the chapter is
to help educators who are facing the rapidly evolving transition to online instruction by providing them
with actionable advice about the decision-making involved in adapting the established, evidence-based
pedagogical method of Peer Instruction while maintaining its effectiveness.

What is Peer Instruction?
Peer Instruction (Mazur, 1997) is a pedagogical method that involves a purposively designed sequence
of educator posed questions, student responses, and peer-to-peer as well as group discussion. Decades
of research shows that the method is easy to use, promotes an interactive learning environment,
motivates students to engage, and facilitates high-level learning (Crouch, 2007; Mazur, 1997; Muller et
al., 2017; Tullis & Goldstone, 2020; Vicrey at al., 2015). The most common use of Peer Instruction is to
implement it at multiple time points within a lecture, which is possible because one sequence takes
about five to 15 minutes. After educators lecture on a topic for a period of time, they pause to pose a
conceptual question called a ConcepTest that builds upon one or more concepts covered in the lecture.
In essence, students answer the question individually, then discuss their responses with other students,
and finally answer the question again in a second, final round. Table 1 lists the seven steps involved in
the classical implementation of Peer Instruction (Mazur, 1997). We refer to the steps outlined in Table 1
as “classical” Peer Instruction of Mazur’s method because educators frequently modify how they use
Peer Instruction (Dancey et al., 2016; Turpen et al., 2016; Turpen & Finkelstein, 2007).
Table 1
The Classical Peer Instruction Sequence
Sequence Sequence
Sequence Description
Step Categories
Step 1 Content Delivery Educator begins with a mini-lecture on content and then poses a
question on that content to the students.
Step 2 Round 1 Educator gives students time to think about their answer to the
question, individually.
Step 3 Answer Selection Students record individual answers to the question, educator
reviews responses. If the question has right or wrong answers,
educator preferably does not reveal distribution to class during
this round to avoid group bias.
Step 4 Small group Educator directs students to discuss their responses with someone
discussion prompts with a different answer (if possible) and attempt to convince their
peer of their response using evidence from the lecture, reading, or
other learning. Students discuss responses in dyads or triads.
264
Step 5 Round 2 Educator poses the same question again and directs students to
record their answer to the same question, suggesting students can
revise their response or keep it the same.
Step 6 Educator Educator reviews Round 2 responses; If the question has right or
evaluation of wrong answers, educator may now reveal distribution to class.
Round 2 responses
Step 7 Large group Educator facilitates discussion and provides correct answers (if
discussion prompt question has a right answer) and provides explanatory feedback.
Note: Sequence adapted from the Peer Instruction Manual (Mazur, 1997).
In earlier work, we identified six common modifications to the seven steps listed in Table 1 and detailed
the potential consequences of changes to the classical sequence (Schell & Butler, 2018). Even with
modifications, researchers found that Peer Instruction benefits learning in a variety of contexts (e.g.,
Tullis & Goldstone, 2020; Smith et al., 2011). We ultimately concluded that Peer Instruction's high
flexibility across a variety of disciplinary and classroom contexts is one of its greatest assets (Schell &
Butler, 2018). We continue to recommend that educators adapt Peer Instruction for in-person and
online environments as they see fit, but we also encourage thoughtful decision-making regarding the
potential influence that modifications to classical Peer Instruction may have on student learning and
motivation. Such pedagogical reflection can help educators maintain the effectiveness of Peer
Instruction while allowing space for individual tailoring and customization.
Why is Peer Instruction an Effective Pedagogical Method?
In order to make evidence-based decisions about how to modify or adapt Peer Instruction, it is critical to
understand why it is an effective pedagogical method. In this section, we review three primary (and
complementary) explanations for why Peer Instruction benefits student learning.
First and foremost, the effectiveness of Peer Instruction starts with learning objectives. The focus must
be on building the conceptual understanding needed to contextualize the procedural skills and
knowledge that students are often adept at acquiring in many STEM disciplines (e.g., Rittle-Johnson et
al., 2015). Accordingly, the nature and quality of the ConcepTest questions are critical to the efficacy of
achieving content-specific learning objectives through Peer Instruction. Of course, Peer Instruction also
helps with developing domain-general learning objectives, such as critical thinking, metacognitive
monitoring, and collaborative engagement. By featuring learning objectives as a core of the design, Peer
Instruction helps educators deploy student- or learner-centered rather than teacher-centered learning.
Second, Peer Instruction is comprised of a potent pedagogical cocktail of activities that promote
learning. Perhaps the most active ingredient in this cocktail is retrieval practice, the act of retrieving and
working with recently acquired knowledge (for review see Agarwal et al., 2021; Dunlosky et al., 2013;
Rowland, 2014; Yang et al., this volume). Retrieval practice has the direct effect of improving long-term
retention (Roediger & Butler, 2011) and transfer of learning to new contexts (e.g., Butler, 2010; see Pan
& Rickard, 2018). Retrieval practice also produces many indirect benefits, such as incentivizing students
to keep up with material outside of class (Mawhinney et al., 1971), reducing anxiety about assessments
(Agarwal et al., 2014), and providing feedback to both students and educators (Black & Wiliam, 1998;
Hattie & Timperley, 2007). In particular, Peer Instruction features feedback that includes explanation to
promote the development of deeper understanding (Butler et al., 2013). The feedback often comes
from other students, which may increase its effectiveness given the benefits of collaborative learning
(Nokes-Malach et al., 2015). In addition, the Peer Instruction method has other features that contribute
265
to its effectiveness, such as spacing out retrieval practice over time rather than massing it (Cepeda et al.,
2006; Dempster, 1989) and introducing variability during the learning of a particular concept, including
repeated thinking about the same concept but in different ways (Butler et al., 2017; Glass, 2009).
Third, Peer Instruction motivates students to engage by addressing the individual differences that often
influence student success in STEM disciplines (Gonzalez & Kuenzi, 2012). Following Mazur’s (1997) lead,
educators often engage students in meta-level discussions about how Peer Instruction creates an
appropriate challenge during learning and the benefits of grappling individually and in community with
complex concepts. Such discussions about the innate challenges of learning encourage students to
adopt a growth mindset and view good learning as effortful, which often yields greater resilience and
higher achievement (e.g., Blackwell et al., 2007; Yeager & Dweck, 2012). The Peer Instruction manual
(Mazur, 1997) also encourages educators to survey their students in order to better understand student
learning needs, goals, interests, and experiences in the course. More than 20 years since publication of
the manual, Mazur and colleagues continue with these formative assessments to influence the framing
of the pedagogical method as well as the course more generally. When students understand the
purpose and perceive the value of course activities (e.g., self-relevance, interest, importance), it can
have a strong effect on their motivation to engage (Cohen & Sherman, 2014; Harackiewicz & Hulleman,
2010). Peer Instruction also creates a cooperative learning environment rather than a competitive one,
which can lead students to adopt mastery learning goals rather than performance goals (Ames, 1992;
Johnson et al., 1991).
In sum, Peer Instruction is effective because it is built with numerous evidence-based principles from the
science of learning and motivation, including being learner centered, engaging students in retrieval
practice, and making the purpose and value of learning transparent. Although the many different
principles that are incorporated into the method support its flexibility with respect to implementation, it
can be rendered ineffective if modified in the wrong ways. For example, modifications that should be
made with caution include asking questions during Peer Instruction that are not associated with learning
objectives; skipping Round 1, Round 2, or the peer discussion component of the sequence in Table 1; or
neglecting to explain to students the purpose or value of engaging learning through Peer Instruction.
Effective adaptation requires a careful consideration of the various factors that influence learning in any
educational context (see McDaniel & Butler, 2010). Most important are the cognitive processes in which
students engage during Peer Instruction; these processes are determined by the selection of the
learning objectives, which can take many forms and must be determined by a careful analysis of what
students must be able to do in future courses (e.g., Anderson et al., 2001; Koedinger et al., 2012;
Koedinger et al., this volume).
Learning in Online Courses
Although learning in online courses is similar to in-person courses in numerous ways (e.g., student
cognition still works the same regardless of context), it differs substantially in other ways, thus requiring
specialized expertise among those who teach in such formats. By and large, many university educators
generally have little to no training in teaching (Handlesman et al., 2004), and faculty readiness to teach
online is multifaceted and varies in accordance with institutional constraints and supports (Scherer et
al., 2021). Most K-12 educators have extensive training in in-person teaching, but limited training in
online teaching. In addition, there is a wealth of instructional design advice about online teaching in the
scholarship on teaching and learning, but it varies greatly in its quality and applicability. According to
Scherer and colleagues (2021), while online teaching has been part of education for many decades,
“actual implementation and adoption have been persistently inconsistent” (p. 1) resulting in wide
variability in student learning online across disciplines and institutional types. In order to teach online
266
courses effectively, educators need to understand how to design effective instruction when students are
not always face-to-face.
The Community of Inquiry Framework (CoI; Garrison et al., 2000) is one established theoretical
framework to help educators design effective online learning environments. The CoI offers an entry
point for educators to begin thinking about online instructional design, especially those who have been
teaching primarily in-person for most of their careers. Relatedly, Valverde-Berrocoso and colleagues
(2020) found that the CoI framework was the most relevant educational theory about e-learning
scholars used from 2009-2018 in international, high-impact scientific journals. Although critiques of the
framework include limitations such as the extent of empirical testing related to learning outcomes
(Rourke & Kanuka, 2009) and incompleteness (see Castellanos-Reyes, 2020), the Col provides a helpful
supplement for educators to think through and organize intentional online instructional design
decisions, especially if educators use an evidence-based pedagogy, such as Peer Instruction or any other
empirically validated pedagogy, alongside the Coi. In large-scale initiatives to support educators in the
transition to online instruction (see Schell, in press) we have found the Coi to be easy for practiced in-
person—but novice online—educators to understand and adapt proven pedagogical approaches drawn
from their face-to-face teaching.
The core of the Coi framework involves three areas that represent the most "essential elements of a
successful online higher education learning experience" (Castellanos-Reyes, 2020, p. 557). Garrison
(2000, 2017) theorizes that effective online instructional design requires three elements to be active in
the course and in sync: cognitive presence, social presence, and teaching presence. According to
Garrison (2017), the quality of interaction and discourse determines the effectiveness of a student's
online learning experience. Garrison defines a strong online learning community as one that involves "a
process of inquiry associated with reflecting on, questioning, analyzing and testing ideas" (p. 120). In
summary, the Coi framework suggests that extensive and various opportunities for student cognition on
the content, strong leadership from an educator with pedagogical content knowledge and pedagogical
knowledge, and intentional cooperative learning activities make for an effective online course. Each of
these elements also make for effective pedagogy in general, whether or not we are in-person, online, or
hybrid. In an online environment, the community will depend on the successful integration and
synchronicity of these three elements. We define each element below and then describe how they
surface in each Peer Instruction sequence.
Cognitive Presence
Cognitive presence refers to the existence of student thinking and engagement with course content.
Cognitive presence results from student engagement in various learning activities educators design to
promote inquiry, reflection, and problem-solving related to key content. If students do not have
opportunities to enact cognition about course content in an online course, their learning will be
dramatically inhibited and limited. Garrison (2017) lists providing "stimulating" (p. 124) questions, small-
group work, and discussion-based activities as key to facilitating cognitive presence. In addition, he
suggests that educators can spur cognitive presence by modeling disciplinary thinking, providing
feedback, and ensuring there is time for students to reflect on their learning.
Teaching Presence
The second element in the CoI framework is teaching presence, which refers to the educator's role as a
designer, facilitator, and provider of direct instruction within the online learning community. Examples
of teaching presence include the educator's implementation of curriculum and how the educator
organizes the students' learning path in an online course. Educators also exhibit teaching presence
through setup and facilitation of reflection and discourse among students. Finally, an educator exhibits
267
their teaching presence through direct instruction (Garrison, 2017). Teaching presence also involves the
educator's ability to choreograph cognitive and social presence to develop a flourishing online course.
Garrison proposes that a strong teaching presence is especially vital in online environments and
correlates with academic achievement and learning satisfaction.
Social Presence
The third element educators should consider if relying on the CoI for instructional design is social
presence or the ability to recognize oneself, the educator, and one's peers as "real" people. Arbaugh and
colleagues (2008) list measures of social presence as including the ability to form distinct impressions of
peers and educators, comfort with interacting with other course participants, and feeling comfortable
disagreeing with other course participants while still maintaining a sense of trust. A course with no social
presence would dramatically limit the benefits of social learning in online environments.
Types of Online Courses
Courses that are delivered in part or fully online can take many different forms. As we discuss the
adaptation of Peer Instruction in the next section, we focus on three common forms of online courses:
hybrid, synchronous, and asynchronous. The term “hybrid” has been used to describe many different
formats, but here we are using it to denote a course format commonly referred to as “dual mode,”
whereby class sessions occur synchronously with some students attending in-person, while others join
remotely via video conferencing or some other technology. This conceptualization of a hybrid course as
“dual mode” is also sometimes referred to as “blended synchronous”. In synchronous online courses,
the mode of instruction is entirely online, but students regularly meet as an entire class for a
synchronous session via video conferencing or other means. Finally, with asynchronous online classes,
content delivery and learning are self-paced but usually with the requirement to get particular pieces of
coursework done by established deadlines. Although students may meet with the educator individually
or with each other in small groups, they never meet as an entire class for a synchronous session.
Like its benefits in face-to-face classrooms, Peer Instruction also provides the opportunity for high
engagement and active learning in online environments. Educators have tremendous freedom with
planning to use the method in online instruction and can adapt the steps in the sequence based on their
needs or the classroom context. In making modifications, whether in-person or online, the
recommendations that we have made in previous work still apply (Schell & Butler, 2018). Although there
are key differences across hybrid, synchronous, and asynchronous formats, the basic requirements for
Peer Instruction (Mazur, 1997) must be met in order for it to be effective, regardless of modality. Table 2
lists these basic requirements along with the step in which they occur and a description of what the
requirement entails.
Table 2
Requirements for Evidence-Based Implementation of Peer Instruction
Peer Instruction
Requirement Description
Step(s)
Facilitator to Launch Steps 1-2 A facilitator is required to deliver the content and launch
Sequence the first round of questioning.
268
Round 1 Question or Steps 1,2,5,7 Students must have a question or a prompt* to respond to
Prompt during Round 1.
Collection of Round Step 3 Students need a way to record their answers and display
1 Responses those answers for the facilitator to evaluate.
Peer Discussion Step 4 Students need a channel to discuss their responses, either
verbally or through text.
Round 2 Question or Step 5-6 Students must be able to record their answers in Round 2
Prompt and display those answers for the facilitator to evaluate.
Facilitator to Close Step 7 A facilitator is required to provide correct answer or

Sequence explanatory feedback and lead group discussion to close
the Peer Instruction sequence.
Note. *Questions can be open- or closed-ended, have right or wrong answers, or be opinion-based.
In the following sections, we consider the adaptation of Peer Instruction for hybrid and synchronous
online courses together because they are similar and then discuss asynchronous course formats. We use
the CoI Framework and the science of learning to guide our analysis and recommendations.
Peer Instruction Through the Lens of the Community of Inquiry Framework
The entire classical Peer Instruction sequence provides ample opportunities to provoke cognitive,
teaching, and social presence in an online environment in ways that are also directly aligned with
learning science. We briefly outline six of these ways here. First, the mini-lecture portion in Step 1 and
facilitation of discussion in Step 7 allow educators to demonstrate effective ways of thinking and
problem-solving in their disciplines and opportunities to provide feedback on learning; both activities
showcase teaching and cognitive presence. Second, each round of questioning in Peer Instruction
provides students with time for reflection, content-specific inquiry, and, depending on the question,
problem-solving, which facilitates multiple opportunities for cognitive presence. Third, the peer
discussion elements in Step 4 provide an opportunity for small-group, content-specific discourse that
activates both cognitive presence specific to the content and social presence as each peer presents their
thinking to the others. Fourth, throughout the Peer Instruction sequence, educators can spark social
presence by helping students get to know and work together with several different peers rather than
just those peers that students may independently choose to interact with outside of class. Fifth,
students will invariably work with a peer who has a different answer in Peer Instruction; this dissonance
allows students to practice engaging in discourse where they may disagree with their peers or the
educator, driving cognitive presence. Finally, in Peer Instruction, educators engage students in frequent
metacognitive reflection on their learning state during Steps 3, 4, and 7 (Table 1) and through optional
response confidence questions, developing deep cognitive presence through a specific, research-based
instructional approach. By merging CoI and Peer Instruction, educators have a reliable framework for
ensuring an effective teaching and learning environment.
Adapting Peer Instruction to Hybrid and Synchronous Online Courses
In both hybrid and synchronous online courses, it is possible to have synchronous interaction during
class time and thus our recommendations for adaption are similar. One of the best things we can do to
promote students’ learning, whether in-person or online, is to break up or chunk lecture content to
269
allow for active learning. Posing questions that allow students to practice retrieving information
(Roediger & Butler, 2010) or elicit, confront, and resolve misconceptions (McDermott & Redish, 1999;
Miller et al., 2013) about key content can promote active rather than passive engagement.
The demand on an educator's time for pre-planning will be in the effort expended to develop an
effective question. To engage students in active learning writ large, or Peer Instruction more specifically,
educators can use a variety of item types, including multiple-choice questions that have correct answers
or constructed response questions that enable open-ended reflection. Mazur (personal communication,
November 2, 2021) now focuses on developing questions that are difficult for students to Google or
otherwise find online. Given that Peer Instruction provides opportunities for practicing retrieval to
promote learning, we recommend focusing on questions that align with course learning outcomes. The
other consideration for pre-planning is the channel students will use to engage in peer discussion. This
channel could be created through a breakout room, private chat systems, or communication platforms,
such as the chat function in the Learning Management System, Microsoft Teams, Discord, Slack, or
others. The specific communication channel does not matter; educators simply need to designate a
channel for peer discussion to occur.
Educators can also use Peer Instruction "on-the-fly" without pre-planning. For example, a common
lecture-based tool is to check for understanding among students by asking: "Are there any questions?"
or "Can you think of any examples of this concept or topic?" When no one responds, faculty often move
on. An alternative approach would be to run an "on-the-fly" Peer Instruction sequence with a prompt,
such as: “Based on the content we just reviewed, take a moment to think about what you found
interesting, confusing, or difficult. Write down your response." Once students have a chance to
document their answer, an educator needs to provide a mechanism for students to record their
responses and open the channel for students to discuss those responses together.
On-the-fly usage of Peer Instruction may be particularly helpful when educators notice that students are
becoming bored, distracted, or otherwise not engaged. Common cues that attention is waning in an in-
person class include students checking phones, lack of eye contact or glazed eyes, shifting in one's seat,
or actual sleeping. However, it is harder to pick up on these cues in the online environment using
teaching tools such as Zoom, where physicality is 2-D (Peper et al., 2021). Furthermore, for students,
online, synchronous learning can result in "Zoom Fatigue" or the feeling of being drained after endless
Zoom meetings. In addition, paying attention to a system like Zoom requires students to focus on the
screen with a fixed gaze (Fosslein & Duffy, 2020). As a result, student engagement in online lectures may
wane faster than in person.
Using a platform such as Zoom, educators can easily facilitate Peer Instruction following the sequence in
Table 1 (or using a common modification as described in Schell & Butler, 2018). The following protocol
provides a guide for pre-planned deployment of the method in an online course:
Table 3
Steps for Deploying Peer Instruction in Hybrid or Synchronous Courses
Sequence Sequence
Sequence Description
Step Categories
Step 1 Mini-lecture Provide a brief lecture or provide direct instruction through other
means, such as a pre-recorded video.
270
Step 2 Round 1 Display a pre-planned or on-the-fly question using a slide and asking
Question students to think about their response on their own.
Step 3 Collect Round 1 You can use a variety of tools to collect responses. Low-tech
responses approaches include asking students to type their answers in the chat,
displaying printed flashcards with A, B, C, D, and E (Lasry, 2008), or
jotting their responses down on a piece of paper and holding them up
for the camera viewfinder. Higher-tech options include Zoom Polling
(for multiple-choice responses) or classroom response systems, such
as Learning Catalytics or Poll Everywhere for various question types.
Step 4 Peer Discussion The channel is key here. In in-person instruction, the channel is simply
the classroom space or seating arrangement. In online instruction, the
easiest way to structure the peer discussion phase is to place students
in randomly assigned Zoom Breakout Rooms and cue them to discuss
their responses. Place students in triads or dyads. You could also ask
students to self-select into rooms based on predetermined criteria
that you identify. Another way that would require more planning
would be to pre-match student dyads ahead of time and cue them to
discuss their answers using private chat functions, such as Zoom Chat
or those found in Slack, discord, or Microsoft Teams.
Step 5 Round 2 Bring students back into a large group and ask them to re-record their
Question responses using the same method used in Round 1. Remind them they
may change the answer based on their discussion.
Step 6 Review and Educator reviews Round 2 responses; if the question has right or
Display Round 2 wrong answers, educator may reveal distribution to class.
responses
Step 7 Feedback Facilitate discussion, providing correct answers and explanatory

feedback.
To aide in logistics, we suggest that in a hybrid “dual mode” course, students who are online engage in
peer discussion with each other and students who are in-person are paired with each other.
Alternatively, the educator can ask in-person students to bring smart phones, padlets, or laptops to class
and to also join the Zoom session and participate in the Peer Instruction sequence with their peers who
are remote. In this implementation, everyone is participating in the Zoom version of Peer Instruction.
Note that if all students are participating in a single Zoom room, audio interference will provide barriers.
Students should wear headphones or not join via audio and simply use a text-based conversation
channel for the peer discussion element.
Adapting Peer Instruction to Asynchronous Online Courses
In asynchronous online courses, educators face the extra challenge of promoting cognitive, teaching,
and social presence without being live with the students. As a result, educators cannot readily engage
with students on-the-fly, facilitate formative assessment that shapes live teaching, or otherwise interact
synchronously during class time. Designing Peer Instruction sequences for asynchronous learning is
271
possible, and it may dramatically increase availability of the three presences and active student
engagement in course content during asynchronous courses. Using Peer Instruction in asynchronous
online courses does require some modifications to the classical Peer Instruction method in Table 1.
Nevertheless, these modifications can be made in ways that align with the CoI Framework as well as the
other learning science principles that support Peer Instruction as a flexible, evidence-based teaching
method.
There are many ways an educator might design implementation of Peer Instruction in an asynchronous
learning environment. Table 4 lists one set of recommended steps for deploying Peer Instruction in an
asynchronous course in a way that maintains its associated learning benefits. One requirement for our
recommended version of asynchronous online Peer Instruction is that it includes a “live, synchronous”
study group among students that they coordinate and conduct themselves via Zoom or in person at a
time that works for them. Of note, Eric Mazur (see McMurtrie 2021) conducts his most recent iterations
of Peer Instruction in this specific manner and reports great success. Specifically, Mazur organizes and
delivers a Peer Instruction module to occur outside of regularly scheduled class time when he and other
educators, are not present (E. Mazur, personal communication, November 2, 2021). This version of
asynchronous Peer Instruction also involves a significant adaptation to the original “single” ConcepTest
delivery. Instead, we recommend providing students with a set of multiple questions to answer on their
own, before engaging in peer discussion. Most learning management systems have this capability, as do
Learning Catalytics and Kahoot!, both of which are tools that can be used to facilitate Peer Instruction.
Table 4
Steps for Deploying Peer Instruction in an Asynchronous Course
Sequence
Sequence Categories Sequence Description
Step
Step 1 Mini-lecture Facilitator creates a pre-recorded mini-lecture or text-based
set of content.
Step 2 Round 1 Questions Facilitator prepares a set of Peer Instruction questions (i.e., a
quiz) rather than a single question in the Learning
Management System, making sure the lecture and those
questions available at a certain start time that is
communicated to students. Students are instructed to take
the Round 1 quiz on their own.
Step 3 Collect Round 1 Collect students’ individual responses through a Round 1
responses version of the quiz on a Learning Management System.
Step 4 Peer Discussion Students are prompted to engage in Peer Discussion (i.e.,
“identify your response and try to convince your peers of
your answer”) of their answers to the quiz via their own Zoom
meeting or in-person meeting.
Step 5 Round 2 Questions Bring students back into a large group and ask them to re-
record their responses using the same method used in Round
1. Remind them they may change the answers based on their
discussion.
272
Step 6 Review and Display Students respond to a Round 2 version of the same quiz
Round 2 responses. delivered in Round 1 on a Learning Management System.
Step 7 Feedback Educator pre-builds immediate and explanatory feedback into
the quiz on the Learning Management System.
Challenges and Workarounds

Challenge #1: Intelligent Pairing
The most prominent challenge to delivering Peer Instruction on an online platform is the lack of a native,
intelligent pairing system to match students. In 2011, Mazur and his post-doc Brian Lukoff launched
Learning Catalytics, a classroom response system designed to facilitate Peer Instruction that included a
feature that could pair students with peers who have different answers. Intelligent pairing can increase
student engagement due to the disequilibria and desirable difficulty inherent in being matched with
someone with a different answer.
Another benefit to intelligent pairing is that it lowers the activation barrier to discussing an answer, and
by proxy of a student’s learning state (i.e., how much the student knows or does not know), with a peer.
Learning Catalytics launches the discussion between two peers by instructing students with the names
of the other students they should discuss their answer with, thus acting as an “agent” informing
students with whom to talk. The agent serves as an advancement to the original prompt in the classical
implementation: "Turn to your neighbor and discuss your response." It is easier for some students to
begin these conversations when an external agent is responsible for the pairing than when the students
have to overcome the activation barrier of selecting someone to talk to themselves (Schell et al., 2013).
We see intelligent pairing as a positive enhancement to Peer Instruction. Student motivation and
engagement in Peer Instruction may increase due to the desirable difficulty resulting from being
matched with someone with a different answer (Schell et al., 2013). Educators can promote intelligent
pairing in in-person classrooms by instructing students to find someone with a different answer, but
intelligent pairing is much more difficult in online classrooms without the aid of third-party software.
Intelligent Pairing Workarounds
Intelligent pairing is only needed when the educator uses multiple-choice questions or constructed
response questions with correct answers. In online, synchronous classrooms, it may be possible to
achieve intelligent pairing using tools such as Learning Catalytics and private chat systems; however, we
believe the benefits of intelligent pairing do not outweigh the associated logistical difficulties and time
constraints in the three modes of online courses discussed in this paper. Instead, we offer two options
for educators: The first is to use open-ended, constructed response questions or questions that do not
have correct answers. Educators can design questions that engage students in higher-order cognition,
such as analysis, estimation, prediction, or creation, which may be particularly beneficial to learning
(Schell & Butler 2018).
The second option is to adopt Mazur's (1997) method of asking for confidence ratings as part of the
script in Step 3 to add a layer of dissonance. Educators can use the 3-point Likert scale Mazur (1997)
outlined (1. Not quite sure, 2. Just Guessing, or 3. Pretty Sure). Alternatively, Tullis and Goldstone (2020)
ask students to rate their confidence using a 10-point Likert scale from "not at all confident" to "highly
confident" (see Tullis & Goldstone, 2020). Before sending students into peer discussion activities,
prompt them as follows:
273
Discuss your confidence in your answer with your peer/s.
If you have different answers, try to resolve those differences using evidence from our class
readings or discussion.
If you have the same answers, use evidence from our class readings or discussion to develop a
rationale for why that answer is correct, but the other answer choices are not.
The confidence ratings have the added benefit of adding an explicit metacognitive layer into the Peer
Instruction sequence. In a Peer Instruction sequence without confidence questions, students may or
may not reflect overtly on their knowledge state and adding confidence questions forces them to
engage in the metacognitive monitoring that we believe is ultimately beneficial to learning and may
promote even better peer-to-peer discussions. However, because the original question combined with
the script-based instructions above includes a lot of information, we recommend putting this script on
the same slide as the questions and giving students plenty of time to read the question on their own
before moving on. We also encourage instructing students to take a picture or screenshot of the
question and instructions, so they can keep them as they move throughout the sequence.
Moreover, for 100% online synchronous instruction, using the Zoom Breakout Room feature, educators
can maintain the positive benefit of an “agent” creating the pairs. Zoom will place students in a random
(or manually assigned breakout room) rather than requiring students to self-organize into a dyad or
triad. In this way, the activation barrier for discussing one's answer with peers is lowered in the same
ways that it is through the intelligent pairing system and the "agent" telling them with whom to talk.
With these workarounds, we believe the loss of intelligent pairing will be mitigated.
Challenge #2: Scoring
Another challenge of running Peer Instruction in 100% online environments in the manner we describe
above is the difficulty of keeping a record of and scoring student responses. Depending on the course
Learning Management System, there may be an integration with a classroom response system, such as
Poll Anywhere or Learning Catalytics. In addition, Zoom Polling records student responses to polls in a
report attached to the meeting. Some of the other response collection methods we suggest above, such
as chat or displaying responses using written notes or flashcards in the camera view, could be
documented through the course and chat recordings. However, Peer Instruction researchers have not
established enough evidence on the benefits of grading student responses to warrant the time required
for such schemes.
Scoring Workarounds
In in-person classrooms, educators often customize their own approach to scoring Peer Instruction
activities. For example, some educators attach points to correct answers, some grade on participation
(response or no response), and some do not grade Peer Instruction responses at all (Vickrey et al.,
2015). Vickery and colleagues (2015) recommend that if students are graded, "low-stakes" grading on
participation only rather than "high-stakes" scoring on correctness should be considered. That said, the
scholarship on Peer Instruction does not provide enough empirical data for us to recommend grading
students at all. We believe that motivating students by posing desirably difficult questions with the
support of peer discussion can promote mastery versus performance orientation among students, which
will benefit their learning.
Challenge #3: Student Resistance
One concern we anticipate educators may have about implementing Peer Instruction online is student
resistance to the provided protocol or instructions. Some may wonder, what if students do not follow
274
instructions to discuss the content when they are in Zoom breakout rooms? Or, what if students use
their mobile devices or computers to search for the answers during the Peer Instruction sequence? Or
what if during the asynchronous version, students work together on the first round of questions instead
of independently?
Student Resistance Workarounds
One thing educators should keep in mind is that these resistant behaviors are equally possible in in-
person, face-to-face classrooms. Students can also elect to undermine the benefits of Peer Instruction
by refusing to engage as instructed whether they are online, hybrid, asynchronous, or in-person. For
Mazur (personal communication, November 2, 2021), creating a classroom culture that promotes
mastery orientation and self-directed learning is key no matter the mode of instruction. Mazur
approaches his current implementation of Peer Instruction to promote mastery behaviors. He increases
the challenge of Peer Instruction by developing questions that students cannot “look up” and that
require independent thought to answer. At the same time, he increases the support available for
students, including additional office hours, mastery-oriented incentives, and analytics-based feedback
and scoring that is based on effort rather than correctness. Educators can mimic this feedback cycle by
evaluating Peer Instruction on effort and by using local Learning Management Systems analytics to
review indicators of low effort such as: spending shortened periods of time in the individual session or
producing high percentages of incorrect answers in Round 1. Educators can provide effort-based
feedback and grading that motivates students to adjust their behaviors to be more in line with the Peer
Instruction guidelines. Ultimately, classroom culture is developed through teaching presence, and
educators should draw on their repertoire of experience to explain the benefits of learning through Peer
Instruction, create intellectual challenge and opportunities for mastery, and support students to realize
the benefits of self-regulated learning.
Conclusion
Peer Instruction offers an evidence-based, easy-to-implement method that can: promote an interactive
learning environment; motivate students to engage with course material, with fellow students, and with
the educator; facilitate high-level learning in online courses. By using established instructional design
frameworks for delivering effective online instruction, such as the Community of Inquiry Framework,
along with theory and findings from the science of learning, educators can adapt this potent pedagogical
method while maintaining its effectiveness online or in-person. Although different formats of online
instruction pose unique requirements and approaches, it is possible to keep the essential components of
Peer Instruction intact. The key to effective implementation of Peer Instruction, online or not, involves
educators understanding how the pedagogical method works from a learning science perspective and
how to deliver Peer Instruction in a classroom culture that supports self-regulated learning behaviors.
Using Peer Instruction based on insights from the science of learning will help educators make informed
decisions about how to adapt Peer Instruction for their specific educational context and mode of
teaching.
Author Note
While co-writing this chapter, the second author (Butler) was supported by the JAMES S. MCDONNELL
FOUNDATION 21st Century Science Initiative in Understanding Human Cognition – Collaborative Grant
No. 220020483. Direct correspondence to Dr. Julie Schell: [email protected] The authors
would like to acknowledge Nikki M. Rodelas for assistance with citations.
275
References
Agarwal, P. K., D'Antonio, L., Roediger, H. L., McDermott, K. B., & McDaniel, M. A. (2014). Classroom-
based programs of retrieval practice reduce middle school and high school students' test
anxiety. Journal of Applied Research in Memory and Cognition, 3(3), 131–139.
Agarwal, P. K., Nunes, L. D., & Blunt, J. R. (2021). Retrieval practice consistently benefits student
learning: A systematic review of applied research in schools and classrooms. Educational
Psychology Review, 33, 1409–1453. https://doi.org/10.1007/s10648-021-09595-9
Aizer, J., Schell, J. A., Frey, M. B., Tiongson, M. D., & Mandl, L. A. (2020). Learning to critically appraise
Rheumatic Disease literature: Educational opportunities during training and into
practice. Rheumatic Diseases Clinics of North America, 46(1), 85–102.
https://doi.org/10.1016/j.rdc.2019.09.006
Ames, C. (1992). Classrooms: goals, structures, and student motivation. Journal of Educational
Psychology, 84(3), 261–271. https://doi.org/10.1037/0022-0663.84.3.261
Anderson, L. W., Krathwohl, D. R., Airasian, P. W., Cruikshank, K. A., Mayer, R. E., Pintrich, P. R., et al.
(2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom's taxonomy of
educational objectives. Addison Wesley Longman, Inc.
Arbaugh, J. B. (2008). Does the Community of Inquiry Framework predict outcomes in online MBA
courses? The International Review of Research in Open and Distributed Learning, 9(2).
https://doi.org/10.19173/irrodl.v9i2.490
Black, P., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education: Principles,
Policy, and Practice, 5(1), 7–74. https://doi.org/10.1080/0969595980050102
Blackwell, L. S., Trzesniewski, K. H., & Dweck, C. S. (2007). Implicit theories of intelligence predict
achievement across an adolescent transition: A longitudinal study and an intervention. Child
Development, 78(1), 246–263. https://doi.org/10.1111/j.1467-8624.2007.00995.x
Bruff, D. (2009). Teaching with classroom response systems: Creating active learning
environments. Jossey-Bass.
Bunce, D. M., Flens, E. A., & Neiles, K. Y. (2010). How long can students pay attention in class? A study of
student attention decline using clickers. Journal of Chemical Education, 87(12), 1438- 1443.
https://doi.org/10.1021/ed100409p
studying. Journal of Experimental Psychology Learning Memory and Cognition, 36(5), 1118–
Butler, A. C., Black-Maier, A. C., Raley, N.D., & Marsh, E. J. (2017). Retrieving and applying knowledge to
different examples promotes transfer of learning. Journal of Experimental Psychology: Applied,
23(4), 433–446. https://doi.org/10.1037/xap0000142
Butler, A. C., Godbole, N., & Marsh, E. J. (2013). Explanation feedback is better than correct answer
feedback for promoting transfer of learning. Journal of Educational Psychology, 105(2), 290–298.
Castellanos-Reyes, D. (2020). 20 years of the Community of Inquiry Framework. TechTrends 64(4), 557–
276
Cepeda, N. J., Pashler, H., Vul, E., Wixted, J. T., & Rohrer, D. (2006). Distributed practice in verbal recall
tasks: a review and quantitative synthesis. Psychological Bulletin, 132(3), 354–380.
Cohen, G. L., & Sherman, D. K. (2014). The psychology of change: self-affirmation and social
psychological intervention. Annual Review of Psychology, 65, 333–371.
https://doi.org/10.1146/annurev-psych-010213-115137
Crouch, C., Watkins, J., Fagen, A., & Mazur, E. (2007). Peer Instruction: Engaging Students One-on-One,
All at Once. In Redish, E., & Cooney, P. (Eds.), Research-Based Reform of University
Physics (Volume 1). American Association of Physics Teachers. http://www.per-
central.org/document/ServeFile.cfm?ID=4990
Dancy, M., Henderson, C., & Turpen, C. (2016). How faculty learn about and implement research-based
instructional strategies: The case of Peer Instruction. Physical Review Physics Education
Research, 12(1), https://doi.org/10.1103/PhysRevPhysEducRes.12.010110
Dempster, F. N. (1989). Spacing effects and their implications for theory and practice. Educational
Psychology Review, 1(4), 309–330. https://doi.org/10.1007/BF01320097
Dufresne, R.J, Gerace, W. J., Leonard, W. J., Mestre, J. P., & Wenk, L. (1996). Classtalk: A classroom
communication system for active learning. Journal of Computing in Higher Education, 7(2), 3-47.
https://www.learntechlib.org/p/80137/
Fosslien, L. & Duffy, M.W. (2020, April 29). How to combat Zoom fatigue. Harvard Business Review.
https://hbr.org/2020/04/how-to-combat-zoom-fatigue
Garrison, D.R. (2017). E-learning in the 21st Century: A community of Inquiry Framework. Routledge.
Garrison, D. R., Anderson, T., & Archer, W. (2000). Critical inquiry in a text-based environment:
Computer conferencing in higher education. The Internet and Higher Education, 2(2-3), 87-105.
Glass, A. L. (2009). The effect of distributed questioning with varied examples on exam performance on
inference questions. Educational, 29(7), 831–848. https://doi.org/10.1080/01443410903310674
Gonzalez, H. B., & Kuenzi, J. J. (2012, August). Science, Technology, Engineering, and Mathematics
(STEM) Education: A Primer. Congressional Research Service.
https://sgp.fas.org/crs/misc/R42642.pdf
Handlesman, J., Ebert-May, D., Beichner, R., Bruns, P., Chang, A., DeHaan, R., Gentile, J., Lauffer, S.,
Stewart, J., Tilghman, S. M., Wood, W. B. (2004). Scientific Teaching. Science, 304(5670), 521-
522. https://doi.org/10.1126/science.1096022 Digital Object Identifier System
Harackiewicz, J. M., & Hulleman, C. S. (2010). The importance of interest: The role of achievement goals
and task values in promoting the development of interest. Social and Personality Psychology
Compass, 4(1), 42–52. https://doi.org/10.1111/j.1751-9004.2009.00207.x
Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–
277
Johnson, D. W., Maruyama, G., Johnson, R., Nelson, D., & Skon, L. (1981). Effects of cooperative,
competitive, and individualistic goal structures on achievement: A meta-analysis. Psychological
Bulletin, 89(1), 47–62. https://doi.org/10.1037/0033-2909.89.1.47
Johnstone, A.H. & Percival, F. (1976). Attention Breaks in Lectures. Education in Chemistry, 13(2), 49-50.
Koedinger, K. R., Corbett, A.T., Perfetti, C. (2012). The Knowledge-Learning-Instruction Framework:
Bridging the Science-Practice Chasm to enhance robust student learning. Cognitive Science,
36(5), 757–798. https://doi.org/10.1111/j.1551-6709.2012.01245.x
Mawhinney, V. T., Bostow, D. E., Laws, D. R., Blumenfeld, G. J., & Hopkins, B. L. (1971). A comparison of
students studying-behavior produced by daily, weekly, and three-week testing
schedules. Journal of Applied Behavior Analysis, 4(4), 257–264.
https://doi.org/10.1901/jaba.1971.4-257
Mazur, E. (1997). Peer instruction: A user's manual. Prentice Hall.
McMurtrie, B. (2021, May 27). Teaching: Why an Active-Learning Evangelist Is Sold on Online Teaching.
The Chronicle of Higher Education. https://www.chronicle.com/newsletter/teaching/2021-05-
27
McDaniel, M. A., & Butler, A. C. (2010). A contextual framework for understanding when difficulties are
desirable. In A. S. Benjamin, Successful remembering and successful forgetting: Essays in honor
of Robert A. Bjork (pp. 175-199). Psychology Press.
Müller, M. G., Araujo, I. S., Veit, E. A., & Schell, J. (2017). Uma revisão da literatura acerca da
implementação da metodologia interativa de ensino Peer Instruction (1991 a 2015). Revista
Brasileira de Ensino de Física, 39(3), 1-20. https://doi.org/10.1590/1806-9126-rbef-2017-0012
Nokes-Malach, T. J., Richey, J. E., & Gadgil, S. (2015). When is it better to learn together? Insights from
research on collaborative learning. Educational Psychology Review, 27(4), 645–656.
Pan, S. C., & Rickard, T. C. (2018). Transfer of test-enhanced learning: Meta-analytic review and
synthesis. Psychological Bulletin, 144(7), 710-756. https://doi.org/10.1037/bul0000151
Peper, E., Wilson, V., Martin, M., Rosegard, E., & Harve, R. (2021). Avoid Zoom fatigue, be present and
learn. NeuroRegulation, 8(1),47-56. https://doi.org/10.15540/nr.8.1.47
Rittle-Johnson, B., Schneider, M., & Star, J. R. (2015). Not a one-way street: Bidirectional relations
between procedural and conceptual knowledge of mathematics. Educational Psychology
Review, 27, 587–597. https://doi.org/10.1007/s10648-015-9302-x
Roediger, H. L. III., & Butler, A. C. (2010). The critical role of retrieval practice in long-term
retention. Trends in Cognitive Sciences, 15(1), 20–27. https://doi.org/10.1016/j.tics.2010.09.003
Rourke, L. & Kanuka, H. (2009). Learning in Communities of Inquiry: A review of the literature. Journal of
Distance Education / Revue de l'ducation Distance, 23(1), 19-48.
https://www.learntechlib.org/p/105542/.
Schell, J. (in press). Designing for academic resilience in hands-on courses in times of crisis: Two models
for supporting hands-on online learning drawn from the COVID-19 pandemic. American
Behavioral Scientist. https://doi.org/10.1177/00027642221118
278
Schell, J. A., & Butler, A. C. (2018). Insights from the science of learning can inform evidence-based
implementation of Peer Instruction. Frontiers in Education, 3, 33.
https://doi.org/10.3389/feduc.2018.00033
Scherer, R., Howard, S., Siddiq, F. (2020). Profiling teachers’ readiness for online teaching and learning in
higher education: Who’s ready? Computers in Human Behavior, 118(106675), 1-16.
https://doi.org/10.1016/j.chb.2020.106675
Smith, M. K., Wood, W. B., Krauter, K., & Knight, J. K. (2017). Combining peer discussion with educator
explanation increases student learning from in-class concept questions. CBE- Life Sciences
Education, 10(1), 55–63. https://doi.org/10.1187/cbe.10-08-0101
Tullis, J. G., & Goldstone, R. L. (2020). Why does peer instruction benefit student learning? Cognitive
Research: Principles and Implications, 5(1), 15. https://doi.org/10.1186/s41235-020-00218-5
Turpen, C., Dancy, M., & Henderson, C. (2016). Perceived affordances and constraints regarding
educators' use of Peer Instruction: Implications for promoting instructional change. Physical
Review Physics Education Research, 12(1), 010116.
https://doi.org/10.1103/PhysRevPhysEducRes.12.010116
Turpen, C., & Finkelstein, N. D. (2007). Understanding how physics faculty use Peer Instruction. In AIP
conference proceedings (pp. 204–207). AIP. https://doi.org/10.1063/1.2820934
Valverde-Berrocoso, J., Garrido-Arroyo, M. del C., Burgos-Videla, C., & Morales-Cevallos, M. B. (2020).
Trends in educational research about e-Learning: A systematic literature review (2009–2018).
Sustainability, 12(12). MDPI AG. https://doi.org/10.3390/su12125153
Vickrey, T., Rosploch, K., Rahmanian, R., Pilarz, M., & Stains, M. (2015). Research based implementation
of peer instruction: A literature review. CBE Life Sciences Education, 14(1), 1-11.
Yeager, D. S., & Dweck, C. S. (2012). Mindsets that promote resilience: When students believe that
personal characteristics can be developed. Educational Psychologist, 47(4), 302–314.
279
In Their Own Words
Part 3
Preparing Faculty, Educational
Developers, Student Success
Professionals, and others to Apply
the Science of Learning
Editors
Lauren L. Kordonowy
Victor A. Benassi

280
In Their Own Words Cultivating and Sustaining a Faculty Culture of
Data-Informed Teaching: How Centers for
Teaching and Learning Can Help
Marsha Lovett, Chad Hershock, and Judy Brooks
Editors
Carnegie Mellon University
Lauren L. Kordonowy
Victor A. Benassi
Faculty in higher education are innovators – not only in their scholarly work, but also in their teaching.
Many faculty try something new in their teaching over time, and others engage in substantial course
transformations. Often, these changes are not informed by peer-reviewed research but rather are
driven by intuition. They tend to be evaluated via instructor or student perceptions, rather than direct
measurements of learning outcomes. This is not surprising, given faculty have limited time and
resources to systematically study and improve their teaching based on data. And yet, there is such
potential to enhance student outcomes if only universities could make data-informed approaches to
education standard practice.
Data-informed approaches to education take two forms, both of which are consistent with the learning
science approaches highlighted in this book. The first approach leverages extant data and results from
the published literature. An instructor following this approach might read the overwhelming evidence
that students learn better with active learning compared to lectures (Freeman et al., 2014) and then
decide to apply active learning strategies in their lecture-based course. This approach is often called
evidence-based teaching practice. The second approach involves collecting new data in one’s own
course. We call this approach teaching as research (TAR).
These two approaches to data-informed education are both applications of learning science: the first
approach applies learning science results, and the second applies learning science methods. Either
approach may be applied to enhance the variety of changes faculty make in their courses (e.g., adopting
or innovating new teaching practices, adjusting elements of course design, or incorporating technology-
enhanced learning (TEL) tools). Additionally, the two approaches can be applied individually or, even
more powerfully, in tandem as Figure 1 illustrates.
Figure 1
Two Approaches for Applying Learning Science to Teaching
281
To illustrate the two approaches working together in the context of enhancing a TEL-based innovation,
imagine an instructor developing an interactive, online learning module that includes animations
illustrating key concepts and active learning exercises providing practice and feedback on these
concepts. She might wonder how best to locate the animations and the exercises within the module to
maximize student learning. Learning science research on multimedia principles (Mayer, 2009; Mayer,
this volume) and active learning in instructional videos (Szpunar et al., 2013) suggest that co-locating
animations and exercises within the module, rather than at the end, would be most efficacious. To
replicate these findings and test specific variations within her teaching context, she could collect data
from her students using different versions of the module. This is exactly what Brasier and colleagues
(2019) did. They leveraged the course management system so different students completed one of four
possible versions of the module, i.e., animations co-located with instruction or not, and active-learning
exercises located within or at the end of the module. They then compared students’ performance on
pre/post tests and subsequent exam questions. The results not only contributed further evidence to the
literature but also guided their subsequent online module design. This example illustrates how the two
approaches – using learning science to leverage past data and collect new data – complement each
other.
However, it is far from easy for faculty to enact either approach – let alone both – in the context of their
busy lives, especially when the time investment seems high (e.g., they may lack expertise or experience
in learning science and these data-informed approaches), and when the return on investment seems low
(e.g., their institution may not reward or recognize the value of such work). How can institutions of
higher education create an environment in which faculty regularly use data (from the literature and/or
their own classroom) to guide their work? First, the institutional culture must value collecting and using
data to improve learning outcomes. Second, the institutional infrastructure must enable faculty – as a
feasible part of their regular practice – to collect and use learning data (quantitative and/or qualitative).
In this chapter, we describe a framework for promoting change in both the culture and practice of
teaching, and we illustrate this framework in action, using our institution as a case study. We posit that
Centers for Teaching and Learning (CTLs) are a natural hub for this work, and we highlight how CTLs can
connect with university administration and other units on campus to strengthen this effort. We conclude
by sharing a set of potentially transferable lessons learned and future directions.
Framework
We developed a framework to shift educational culture and practice toward data-informed teaching
(Figure 2, hereafter MEFD): (1) Motivate faculty to collect data as they teach; (2) Educate faculty so they
can confidently collect actionable data in their own courses; (3) Facilitate this work by providing human
and technical support to reduce the workload involved; and (4) Promote venues to disseminate the
results. Steps 1 and 4 focus on cultural aspects by promoting the value of this work, and steps 2 and 3
focus on practical aspects by lowering barriers.
An advantage of this framework is that, although its components can work independently, when the
steps are implemented together, they interact in positive ways for greatest impact (see Lovett &
Hershock, 2020). Moreover, the MEFD cycle is flexible and transferable in that each step specifies a goal
but does not prescribe a particular strategy for achieving that goal. Each step may be implemented in
different ways, across institutions, or even within an institution across time. Consequently, we believe
the MEFD framework allows for modular and evolving strategies to meet the changing needs and
resource constraints at hand.
282
Figure 2
A Framework for Cultivating and Sustaining Data-Informed Teaching
Note: From “Cultivating and sustaining a faculty culture of data-driven teaching and learning: A systems
approach,” by M. Lovett and C. Hershock, 2020, To Improve the Academy, 39, p. 67
(http://dx.doi.org/10.3998/tia.17063888.0039.104). CC BY-NC-ND 4.0.
CTLs are a natural hub to support and promote a campus-wide shift towards data-informed teaching.
Cook and Kaplan (2011) illustrate the many ways that CTLs can position themselves to provide effective
professional development on teaching for both current and future faculty (e.g., through consultations
and workshops that disseminate evidence-based teaching approaches). In addition, CTLs often provide
technology infrastructure and support for faculty to incorporate TEL tools in their teaching. As part of
this support, CTLs can help faculty collect data on tools’ use and impact, often by facilitating access to
the digital data automatically logged by TEL tools.
Many CTLs already engage in this work, reaching substantial numbers of instructors and future faculty.
We argue that CTLs can also readily apply the MEFD framework, adjusting their programs and services to
actively support and/or lower barriers to data-informed teaching.
Our Case Study

Carnegie Mellon University (CMU) serves as our case study for the MEFD framework. CMU is a research-
intensive, private institution with approximately 14,000 graduate and undergraduate students and 1400
faculty. CMU has a long-standing CTL – the Eberly Center for Teaching Excellence and Educational
Innovation – that provides seminars, workshops, and one-on-one teaching consultations on evidence-
based teaching strategies, TEL, and formative assessments of student outcomes. Annually, half of our
work supports faculty and staff in their instructional responsibilities. The other half supports graduate
students. During the 2020-21 academic year, our CTL served more than 600 faculty and 600 graduate
students. These services included consulting with 362 faculty and staff educators (see Figure 3).
283
Figure 3
Number of Educators Served by Eberly Consultations Across Time
The Eberly Center employs 23 staff members specializing in evidence-based pedagogy, TEL, and
formative assessments of student outcomes. As little as eight years ago, we had eight employees, and
we piloted many approaches described in this chapter with far fewer staff. In part, those successful pilots
helped make the case for our CTL’s growth.
A prominent goal of our CTL over the past several years has been supporting faculty to regularly reflect
and act on learning data collected in their own courses. In the next four sections, we describe how we
have pursued this goal by implementing each step of the MEFD framework.
Motivate
Given Competing Demands, What Incentives Might Persuade Faculty to Engage in Data-
Informed Teaching and Course Design?
The first step of the MEFD Framework involves motivating faculty to collect data as they teach. There are
multiple ways to approach this through financial and other direct incentives that implicitly communicate
the value an institution ascribes to collecting and using data in one’s teaching. A common financial
incentive involves creating a faculty grants program (e.g., Cook et al., 2011) or providing special stipends
and/or teaching relief for projects that involve collecting and using data. Another approach involves
changing the reward structure, such as criteria for promotion and tenure and/or education-related
awards. Although effective, these strategies can be costly to implement effectively. We have explored
modest- and moderate-cost versions of these strategies along with appealing to faculty members’
intrinsic motivations.
In 2014, CMU launched a Faculty Seed Grants Program to encourage educational innovation. Awardees
receive both a cash award of $15K and $15K of in-kind personnel support from the Eberly Center (e.g.,
online/multimedia design, software development, technological and pedagogical consultation, and/or
study design and assessment). The application process involves not only describing the innovation, but
also (1) identifying how technology would be used, (2) providing a rationale for why the innovation
should produce enhanced student outcomes by citing research, and (3) articulating how new data would
284
be collected and used to test and guide refinement of the innovation. In other words, faculty had to
explain how their project leveraged learning science results and methods (TAR). Moreover, to select
awardees, CMU used criteria that emphasized the strength of evidence supporting the proposed
innovation (“Does the research suggest it will work?”) and the quality of the data/assessment plan (“Are
direct measures of student learning and performance included?”). One successful project proposed
developing a set of online modules on writing composition. The modules would provide students
instruction, practice, and feedback on basic composition skills. The proposal referenced ample research
on the benefits of practice and feedback, and explained how pre/post-test data, along with practice data
from the modules, would be used to identify targeted improvements.
In addition to this university-administered grant program, our CTL administers several faculty fellow
programs (with funding from various sources). These programs are selective and application-based. They
offer modest stipends ($3-$5K/fellow). Over time, we have increased the degree to which data collection
was, at first, encouraged, then expected, and now required as part of these programs. Once selected,
each faculty fellow is paired with a CTL team, including teaching, assessment, and technology
consultants. This team facilitates the design, implementation, and evaluation of their proposed project.
Teaching consultants provide a Small Group Instructional Diagnosis (see Finelli et al., 2008; Finelli et al.,
2011) in each fellow’s course to gather students’ feedback on how the course is/is not supporting their
learning. Additionally, assessment consultants help fellows collect, analyze, and interpret at least one
direct measure of student learning. Consequently, all fellows use multiple sources of data to refine their
projects, and some have engaged in full-on educational research (Christian et al., 2019; Barrett et al.,
2021; Eberly Center, 2021).
When money is not available, award programs can leverage prestige and/or special treatment to
motivate faculty (with help) to collect data on student outcomes. For example, the CMU Teaching
Innovation Award (TIA) recognizes faculty members’ educational innovations, based on both their
transferability to other teaching contexts and effectiveness for improved student outcomes. While many
teaching awards focus on lifetime achievement, the TIA celebrates small changes, such as a single
assignment, TEL activity, or teaching technique, resulting in measurable differences for student learning
or engagement. Consequently, the award promotes a data-valuing culture of teaching among faculty
who may be unable to execute large-scale course transformations. The aforementioned faculty seed
grants program and our CTL’s faculty fellowship programs provide special treatment in the form of in-
kind staff support to motivate and sustain faculty engagement, by providing critical just-in-time
assistance to faculty who already have a full plate.
The Framework in Action (Motivation)
Faculty fellowship motivates adoption of a learning science approach. Sarah Christian is an
Assistant Teaching Professor in Civil and Environmental Engineering who was selected to participate in
one of our faculty fellows programs. Her project revised cookbook-style labs in her Material Properties
course. Exam data showed that students were not transferring concepts from labs to exams. Christian
wanted help to implement a more effective teaching strategy and to assess how it was working. Her
collaboration with our CTL started as a teaching consultation focused on Inquiry-Based Learning (IBL) as
the evidence-based pedagogy of choice (Pedaste et al., 2015). The consultation revealed that her labs
actually included a light-weight component of IBL (i.e., students made a prediction and tested it by
following a provided procedure; referred to as “structured IBL” by Tafoya et al., 1980). Although she was
originally unaware of the research on IBL, Christian worked with a teaching consultant to learn more
about this approach and to build an open-ended form of IBL into one of her labs (“guided IBL” by Tafoya
et al., 1980). Christian also worked with an assessment consultant to revise exam questions to best
measure the learning outcomes associated with each of her labs and set up a study design in which the
285
impact of structured versus guided IBL could be measured. By enhancing assessments already embedded
in the course, she could directly and rigorously measure learning without creating extra work for her or
her students. Students performed 10% higher on the exam questions related to the guided IBL lab
compared to the structured IBL labs (Christian et al., 2019). Christian is now adding guided IBL to more
labs and continuing to collect data for ongoing refinement. She repeatedly highlights that the faculty
fellowship motivated her to pursue a project that she otherwise would not have done, primarily by
providing a support structure, recognition for her teaching efforts via multiple campus events and
websites, and mentorship for engaging in teaching as research.
Pandemic + CTL Program = Motivation to Use Data. Out of the necessary move to remote
teaching during COVID-19, and with the goal of showcasing deliberately designed remote learning
experiences, CMU created the Signature Course Initiative in Summer 2020. Faculty nominated for this
initiative worked with Eberly Center colleagues to make enhancements to their course for fall 2020,
incorporate a TEL component that would be used in fall and beyond, and iteratively improve the course
based on data. As with our CTL’s other faculty fellow programs, we assembled an Eberly support team
(Teaching Consultant, Learning Engineer, and Assessment Consultant) for each signature course.
Throughout Summer 2020, these teams (faculty and Eberly) met weekly to identify and design
“signature” aspects of each course (e.g., more personalized experiences for students; building
community). Building Virtual Worlds was one such course, designed and taught by Dave Culyba,
Associate Teaching Professor in the Entertainment Technology Center. Building Virtual Worlds is a team-
based project course, so the re-design focused on helping students learn to work as effective remote
teams: Eberly colleagues drew on evidence-based strategies for designing teamwork assignments; online
modules were identified and incorporated to provide students practice on collaboration and conflict-
management skills; and Culyba developed a set of worksheets/activities for students to reflect and
evaluate themselves and their teams. Then in Fall 2020, when the course was launched, our teaching
consultant collected Small Group Instructional Diagnosis data to give Culyba a read on how students
were experiencing this new approach. In addition, new activities and tools (online modules, pre- and
post-tests, personal and team reflections, and peer evaluations) have provided rich sources of qualitative
data to inform iterative improvements. Although comprehensive data analysis is still underway, Culyba
has already adjusted his teaching based on preliminary results and is eager to continue collaborating
with the CTL, especially for support in analyzing and interpreting this dataset.
Educate
How Can Faculty Acquire (or Strengthen) Skills Required to Design, Conduct, and Interpret
Classroom Research?
A common goal of CTLs is to provide professional development for faculty in their roles as educators. The
Educate step in our framework aims to design professional development that meets faculty where they
are, in terms of interest and readiness to engage in data-informed teaching.
Several years ago, we noticed a trend that numerous faculty consultations involved issues around data for
testing the impact of teaching innovations. In Fall 2014, we hosted a Faculty Special Interest Group (SIG)
on Collecting Data to Inform Teaching that brought together faculty interested in (or already doing)
classroom-based research. Compared to developing a new workshop series around TAR, this discussion-
based SIG required fewer resources. It also gave us a chance to learn about faculty participants’ goals and
struggles, while offering support, guidance, and a venue for peer discussion, learning, and feedback. Our
SIGs usually run for a single semester, with 12-15 faculty meeting 3-4 times. This SIG was immensely
successful in that it drew over 25 participants, and the participants requested that we continue—for four
more semesters!
286
The SIG gave us a sense of the key issues from the faculty perspective and an early read on how this topic
could potentially draw a crowd. As with most CTL programs, we wanted to move beyond the early
adopters to faculty who could become the next wave. Therefore, we launched the Teaching as Research
(TAR) Institute. Our goal for this program was to provide more structured educational support for
colleagues who were new to collecting data in their courses, while leveraging resources previously
developed for the SIG.
Now an annual event, our TAR Institute engages approximately 20 faculty per year in a four-day program
focused on study design, data sources, and ethical research practices. However, the Institute is pitched as
an opportunity for faculty to do both new pedagogical development (e.g., incorporate active learning
into your course) and to collect data for improving student learning. Participants leave this program ready
to implement a new, evidence-based teaching strategy and collaborate on classroom-based research with
CTL support. Across offerings of the TAR Institute, we have often focused on a theme (e.g., active
learning, inclusive teaching) and then wove into the program primary source literature on that theme. We
have found this to be a natural way to introduce evidence-based practices and give faculty a window into
how researchers study the impact of those practices in their classroom contexts.
Faculty participating in the TAR Institute need not come with a research question in mind. As we engage
with the pedagogical theme and the body of associated research, we also share and discuss a menu of
questions that have not yet been investigated or fully answered. (Within active learning, Freeman et. al.
[2014] calls these “second-generation questions.”) Faculty then work one-on-one with Eberly consultants
to select and refine a question that is aligned with their interests and teaching context.
To date, over 90 faculty and staff have attended a TAR Institute, jump-starting over 50 data-informed
teaching projects. Some clients wished to publish from the outset. Others engaged in this work to better
inform their future teaching. Given that our goal is to promote data-driven teaching, publication rate is
not necessarily a valid measure of impacts. Nevertheless, ten TAR projects (and counting) have produced
peer-reviewed publications and/or conference presentations.
The Framework in Action (Educate)
CTL Institute prepares faculty for second-generation research. Copious research suggests that
infusing active learning into traditional lectures leads to greater learning outcomes and reduces course
failure rates (Freeman et al., 2014). However, questions remain about how best to implement active
learning to maximize student outcomes. For example: Does the way one debriefs an activity impact
learning gains? Three Information Systems faculty, Marty Barrett, Michael McCarthy, and Joe Mertz, who
attended our TAR Institute, decided to explore this question in their Distributed Systems course, which
enrolls over 100 students. Their students discuss concept-based multiple-choice questions in small groups
and then vote on the correct answers. The faculty opted on a classroom research design to test whether
debriefing all answer choices added value beyond just explaining the correct answer (a less time-
consuming approach). They did not enter the Institute with this question, but the Institute helped them
formulate it as part of refining their active learning intervention. Results replicated the benefit of active
learning over traditional lecture by showing a 13% increase in overall exam performance (Barrett et al.,
2021). Furthermore, while the debrief type did not impact exam performance on recall questions, it did
show an effect on application questions: when instructors debriefed all answer choices, the average exam
performance on application questions increased more than 5% (as one might predict from literature on
“explanation feedback”; Butler et al., 2013). Consequently, following application-focused classroom
activities, the instructors now debrief all answer choices rather than only the correct answer.
CTL Institute empowers faculty as learning science leaders. In 2017, Sara Moussawi, Assistant
Teaching Professor, Information Systems, was curious about how research in teaching might work, but
287
she had never launched a TAR project. She decided to attend our TAR Institute in 2018 with the goal of
developing a study for her and her co-instructor’s Information Systems Capstone course. They wanted to
explore how students could give and receive higher quality peer feedback on team presentations. They
compared two different presentation approaches: the traditional approach (teams presenting to the
whole class in sequence) and a speed dating approach, where teams paired up to reciprocally present
their projects and provide feedback via a survey after each presentation. Students reported giving and
receiving more helpful feedback and being more significantly engaged during the speed dating sessions as
compared to the traditional class-wide presentations. She later presented these findings at CMU’s
Teaching & Learning Summit (a half-day campus-wide conference).
But this was just the beginning. Moussawi next wanted to repeat the TAR process for other courses. She
met regularly with CTL assessment consultants to develop questions to explore, identify data to collect,
and discuss study designs and data analysis. Over time, she has become more independent, mainly
reaching out to validate her approach and analysis of the data. This past year, she recruited 12
departmental colleagues and ran a training session to onboard them to TAR goals and procedures. Over
time, they created a comprehensive set of questions to study, coded student assignments accordingly,
and are now performing data analyses. CTL assessment consultants were invited to the training session
and observed that Moussawi handled the training like an expert. Now she is a leader and mentor for
other faculty wanting to do educational research. Moussawi remains connected with CTL assessment
colleagues, as a sounding board and for feedback on data analysis. As one of our CTL colleagues said, “If
a faculty member comes to us asking, ‘Where can I end up if I engage in TAR?’, we can say, ‘Look at
Sara.’” We are delighted to have been a part of Moussawi’s development as a teacher-researcher.
Facilitate
Are the Logistics of Conducting a Study a Deal-Breaker for Faculty to Adopt Data-
informed Teaching?
In our experience, they most certainly are. For example, a university’s Institutional Review Board (IRB)
must review and approve the research plan before an instructor can conduct a research study in their
course. To the uninitiated, the IRB process can be opaque, convoluted, and time-consuming, despite the
best efforts of well-meaning IRB offices. Typically, although many faculty wish to use similar types of
interventions and data sources to test their classroom research questions, they must individually write
their research goals, study protocol, and consent form from scratch at the same time as they are learning
to navigate the IRB system. To lower this barrier, our CTL collaborated with colleagues in the IRB Office to
establish a broad protocol for classroom research, focused on improving the design and delivery of
specific courses. The broad protocol allows instructors-of-record to collaborate closely with CTL
consultants to conduct classroom research, implement particular study designs, and collect certain types
of data to conduct research within an approved protocol. As long as particular eligibility requirements are
met, this piece of infrastructure eliminates the need for each faculty member interested in educational
research to navigate the IRB system on her own. Our approach is similar to that pioneered at other CTLs
(e.g., Wright, 2008; Wright et al., 2011).
Augmenting the IRB broad protocol, our CTL’s TAR Consulting Service also provides tangible support for
several other steps of the research process. This includes help with study design, background research,
assessment design, and data analysis and interpretation. Consultations focus on how best to implement
an intervention, including comparison groups and accounting for potential confounding factors.
Background research may entail finding relevant literature to inform the study design as well as
identifying and vetting appropriate assessment instruments. Assessment design may involve
collaboratively workshopping an instructor’s assessments to align with learning objectives and
288
embedding valid, reliable, direct measures of learning outcomes into coursework. While these planning
and design steps of educational research are quite familiar to our CTL consultants and faculty from some
areas within the social sciences, they are naturally outside the disciplinary expertise of most faculty. By
offering this consultation service, we support faculty to engage in TAR without them needing to become
educational research experts themselves.
Because our CTL also provides support for TEL tools and infrastructure, we often support faculty in
implementing their innovations in TEL tools. This has several advantages. For example, a TEL-based
intervention can enhance implementation fidelity and validity by creating a consistent set of instructional
interactions that are designed to meet the study design’s specifications. In addition, using a technology
tool can make it easier to administer multiple (randomized) experimental conditions. In one case, our
learning engineers implemented multiple versions of an online module by carefully varying specific
features to create the different conditions of a study. And in other cases, our assessment consultants
have, on behalf of the teacher-researcher, set up different versions of an instructional activity within the
LMS so they were randomly assigned to different subsets of students. Another important advantage of
technology-based innovations is that it is often natural to leverage the tools’ log files (e.g., LMS data,
online homework system data) rather than having to manually enter and/or code student responses. For
example, faculty using students’ LMS interactions or assignments as data sources in their research can
save a lot of time thanks to our CTL’s ability to programmatically export LMS data on faculty members’
behalf. And finally, once results are interpreted to inform improvements, the TEL intervention can often
be easily modified accordingly. The fact that our CTL supports the university’s educational technology
ecosystem enables us to draw on a consistent set of tools to support both TEL and TAR, facilitating more
standardized use of common tools and approaches than would otherwise be likely.
After data collection, we often function as a data broker by analyzing data for faculty who lack the
necessary quantitative or qualitative skills or time to do this step. We also coordinate matching course-
level data with other student data from the Registrar. In most cases, instructors do not have permission
to see Registrar data outside the scope of their course, but such data can be helpful because
demographic and/or academic variables (e.g., gender, incoming GPA) can be used to establish that an
intervention equitably benefits different student groups or to control for particular sources of variation
between students or cohorts. Thus, we developed an automated, honest-broker process by which the
faculty member and Registrar can send de-identified data to us in such a way that we are not able to
reconstruct the original data sets with student identifiers but can match entries corresponding to the
same student.
The Framework in Action (Facilitate)
CTL is a lever for data-informed "flipped classrooms." Hakan Erdogmus, Teaching Professor in
Electrical and Computer Engineering, was already devoting the majority of class time to active learning in
order to engage students. He was committed to further innovating his approach by “flipping the
classroom” but did not know how best to optimize the implementation. After attending the TAR Institute,
he wanted to leverage his new skills to investigate how best to support students’ pre-class learning via
instructional videos. Consultations helped him identify evidence-based practices from multimedia
learning research – e.g., interpolating assessment questions within the videos to focus student attention
(Szpunar et al., 2013). Consultations also helped him identify data sources and an appropriate study
design to test the effectiveness of inserting these guiding questions. We also helped him analyze and
interpret the data (brokered from his course and the Registrar) to inform future course design choices
and to prepare a paper for an international engineering conference (Erdogmus et al., 2019). Because
students performed significantly better on exam questions related to videos containing guiding
questions, Erdogmus now requires students to submit answers to these questions for all videos.
289
CTL lowers barriers to classroom research. Paulo Carvalho, Kody Manke, and Ken Koedinger are
three CMU faculty with significant expertise in social science research. In the context of teaching
students social psychological theories, they wanted to compare the effectiveness of two different
approaches to active learning, each supported by past research. The first approach involved providing
students practice and feedback on the social psychology theories being taught, and the second approach
involved having students make a prediction before reading and explaining the theories’ observed results.
While these colleagues were very experienced in learning science research methods, they still benefited
from the Eberly Center’s broad IRB protocol and our team’s expertise in implementing active-learning
instructional materials online. We partnered with them on this project to facilitate several steps. First,
we onboarded them to the requirements of our IRB protocol, eliminating significant effort on their part.
Second, we implemented the two conditions of their study as content-matched modules in CMU’s Open
Learning Initiative (OLI) platform and set up the modules for random assignment of students. Third, we
assisted in exporting data from the online modules so it could be included in the analyses. These
colleagues published the results in a learning science journal (Carvalho et al., 2018) and returned the
following year to partner on a follow-up study. This case illustrates how our support model is flexible in
meeting faculty members’ needs – whether they are new to educational research or highly experienced.
Disseminate
How Can Teaching and Classroom Research Create a Community of Practice, Rather Than
Remain Solitary Endeavors?
CTLs can effectively position themselves as a clearinghouse of resources on learning and teaching as well
as a nexus for fostering dialogue, community-building, and a culture of teaching on campus (Cook &
Kaplan, 2011). To this end, our CTL provides a number of pathways to “close the loop” on TAR projects,
disseminate findings, and create community. For instance, annually, we host a university-wide teaching
and learning conference. This CMU Teaching & Learning Summit provides a venue for teacher-
researchers to share their work with each other and the broader university community. Through posters,
roundtable discussions, demonstrations, and interactive presentations, instructors share both their
teaching innovations and the data they have collected. Each year, roughly 200 faculty and graduate
students attend, with over 40 instructors presenting their work. Many presentations include quantitative
or qualitative data on student outcomes, which is encouraged as part of the event’s request for proposal
process.
Seeing the work of colleagues at this event has motivated other instructors to join this community of
practice and take advantage of the TAR programs and services described above. For example, one
instructor commented, “I had no idea my colleague was doing that and that there was support available.
If she can do it, maybe I can, too.” This outcome is one we hope for – namely, for the Summit to be a
driver of interest and a mechanism for connecting instructors with the multiple entry points into the
MEFD Framework.
Our Teaching & Learning Summit is not the only pathway for disseminating TAR findings. We highlight
faculty members’ evidence-based teaching and TAR across our CTL events. We also feature TAR projects
prominently on our CTL website (Eberly Center, 2021). Some instructors wish to disseminate beyond our
university by publishing in relevant journals focused on Discipline Based Education Research (DBER) and
SoTL and/or presenting at disciplinary conferences. Our TAR Service encourages and supports these
goals. Often, faculty have no idea where to disseminate their work, so we help them find appropriate
venues. Additionally, our consultants guide faculty in writing, provide templates and examples to follow,
and offer feedback on drafts.
In addition to these avenues for scholarly dissemination, our CTL is poised to support dissemination of
290
the actual teaching innovations that our faculty colleagues generate. For example, if a faculty member
creates a new instructional or assessment resource that is transferable to other courses, we are glad to
(with their permission) suggest that resource, when appropriate, to other CMU faculty working with us.
By supporting the dissemination of our colleagues’ educational innovations into broader practice at
CMU, we are helping the original faculty member (who is glad for their work to be shared), the
‘receiving’ faculty member (who is glad to incorporate a demonstrably effective resource), and the
students of both faculty (who are learning more thanks to the data-informed innovation).
The Framework in Action (Disseminate)
CTL showcases culturally responsive teaching. Alexa Woloshyn, Assistant Professor of
Musicology, was interested in diversifying her course content beyond the traditional canon. She applied
to one of our faculty fellow programs for support in redesigning her course to incorporate evidence-
based Culturally Responsive Teaching (CRT, Ginsberg & Wlodkowski, 2009). Specifically, her course goals
were expanded to include students interrogating the existing musical canon and exploring
underrepresented composers, especially women and BIPOC (Black, Indigenous, and People of Color). In
addition to supporting Woloshyn in applying various evidence-based CRT practices, we identified her
project as a great example to highlight at the CMU Teaching & Learning Summit. In this way, we were
able to highlight Woloshyn’s work and share it with a broad audience (around 200 other CMU educators)
in a way that elevated her voice in discussing the project, its goals and methods, and her experience
working with our CTL.
CTL helps faculty apply effective innovation across campus. Many faculty who assign team
projects nevertheless avoid teaching teamwork because it is outside their expertise. Martha Harty,
Distinguished Service Professor of Philosophy, was interested in developing online modules that other
faculty could assign, enhancing their students’ teamwork skills without requiring the faculty to become
experts themselves. Given the Eberly Center’s TEL support, we were a natural partner in this work. Eberly
learning engineers worked with Harty to develop several online modules in the OLI platform, infusing
multiple evidence-based practices into the design (e.g., active learning, multimedia design principles).
We also helped Harty find a well-matched course context for testing the modules’ efficacy, given our
support for Mellon College of Science colleagues in launching their new Core Curriculum, which included
a first-year course with a major teamwork assignment. We brought all parties together to discuss a
possible TAR collaboration and quickly achieved consensus on conducting a quasi-experimental study to
test Harty’s modules (Gadgil et al., 2018). Given that study’s positive results, we were all interested in
making the modules more broadly available across CMU. Harty identified additional courses across
campus that could profitably use the modules. And when those faculty decided to adopt the modules,
CTL colleagues were able to walk them through incorporating the modules (technologically and
pedagogically). Additionally, the OLI modules have become a “go to” strategy for CTL consultants to
recommend to clients who are incorporating teamwork in their course designs.
Conclusions and Lessons Learned

Over several years, we have refined our strategies for implementing the MEFD framework and seen
increased uptake of our TAR services (see Lovett & Hershock, 2020). Two conspicuous challenges remain.
First, TAR projects most commonly stall at the data analysis phase. This is especially likely when an
instructor wishes to analyze the data themselves (rather than seeking CTL support). We have found it
challenging to navigate this– balancing the instructor’s autonomy over their research and our desire to
provide support. Second, we have not incorporated qualitative data in this work nearly as much as we
would like, either in leveraging qualitative research from the literature or supporting TAR projects. We
recognize the value of qualitative and mixed methods, but few faculty have adopted them in their TAR
291
projects. This may be a function of the disciplinary biases of our faculty clients (who may be more
comfortable with quantitative data). Given the Educate step in MEFD and our professional development
mission, we are actively working to address this opportunity. One strategy involves cultivating inclusive
teaching projects and measuring student experience outcomes, to which qualitative methods naturally
align.
In addition to iterating CTL programs and services to mitigate these challenges, we have learned several
lessons that we believe are transferrable:
● For institutions with a funding program for educational innovations, consider adding selection
criteria that address plans for data collection and data-informed improvement; then, provide
support for faculty to execute those plans.
● To promote data-informed improvement, explore incentives and rewards beyond financial, such
as:
○ Personnel support to help faculty collect and analyze data,
○ Awards to recognize data-informed teaching and learning, and/or
○ Venues (e.g., programs, websites) to showcase data-informed projects.
● Discover what worked for early adopters and where they struggled. Then, build that into CTL
programs and services.
● Find university partners who can help lower barriers to data collection and use (e.g., CTLs, IRB
office, Registrar’s office). In addition, if pedagogy, technology, and assessment are not all
represented in your CTL, collaborate with other units or faculty who contribute needed skillsets.
● Cultivate peer networks of faculty who can talk about their use of data and help make the
case for its value, inspiring colleagues to try something similar.
● Provide multiple venues to help faculty disseminate their innovations and data-informed
practices– across campus and beyond.
We hope leaders of CTLs and institutions of higher education will find this framework helpful for strategic
planning in two ways. First, it can help CTLs intentionally plan programs and services towards a shared
goal of data-informed teaching. Second, this framework can provide a useful lens for identifying and
prioritizing new opportunities to promote data-informed teaching.
Author Note
Portions of the work discussed in this chapter have been reported in Lovett & Hershock, 2020. We
appreciated the opportunity to expand on that discussion and incorporate more recent activities,
outcomes, and lessons.
References
Barrett, M., Hershock, C., McCarthy, M., Melville, M., & Mertz, J. (2021). What type of debrief is best for
learning during think-pair-shares? Teaching and Learning Inquiry, 9(1), 45-60.
Brasier, D.J., Melville, M.C., Hershock, C., & Rule, G. (2019). Pairing practice and feedback with
animations optimizes student learning in an online module. Journal of Computer Assisted
Learning, 35(6), 782-793. https://doi.org/10.1111/jcal.12388
292
Butler, A. C., Godbole, N., & Marsh, E.J. (2013). Explanation feedback is better than correct answer
feedback for promoting transfer of learning. Journal of Educational Psychology, 105(2), 290–298.
Carvalho, P.F., Manke, K.J, & Koedinger, K.R. (2018). Not all active learning is equal: Predicting and
explaining improves transfer relative to answering practice questions. In T.T. Rogers, M. Rau, X.
Zhu, & C. W. Kalish (Eds.), Proceedings of the 40th Annual Conference of the Cognitive Science
Society (pp. 1458-1463). Austin, TX: Cognitive Science Society.
Christian, S. J., Hershock, C., & Melville, M. C. (2019, June), Guided Inquiry-based Lab Activities Improve
Students’ Recall and Application of Material Properties Compared with Structured Inquiry. Paper
presented at 2019 ASEE Annual Conference & Exposition, Tampa, Florida.
https://peer.asee.org/32881
Cook, C. E., & Kaplan, M. (2011). Advancing the culture of teaching on campus: How a teaching center
can make a difference. Stylus Press.
Cook, C. E., Meizlish, D., & Wright, M. C. (2011). The role of a teaching center in curricular reform and
assessment. In C. E., Cook & M. Kaplan (Eds.) Advancing the culture of teaching on campus: How
a teaching center can make a difference (pp. 121-136). Stylus Press.
Eberly Center (2021). Teaching as research: Discovering what works for student learning.
https://www.cmu.edu/teaching/teaching-as-research/.
Erdogmus, H., Gadgil, S., & Peraire, C. (2019) Introducing low-stakes just-in-time assessment to a flipped
software engineering course. Paper presented at the 2019 Hawaii International Conference on
System Sciences (HICSS), Honolulu, Hawaii.
https://scholarspace.manoa.hawaii.edu/handle/10125/60201
Finelli, C. J., Ott, M., Gottfried, A. C., Hershock, C., O’Neal, C., & Kaplan, M. (2008). Utilizing instructional
consultations to enhance the teaching performance of engineering faculty. Journal of
Engineering Education, 97(4), 397-411. https://doi.org/10.1002/j.2168-9830.2008.tb00989.x
Finelli, C. J., Pinder-Grover, T., & Wright, M. C. (2011). Consultations on teaching: Using student
feedback for instructional improvement. In C. E., Cook & M. Kaplan (Eds.) Advancing the culture
of teaching on campus: How a teaching center can make a difference (pp. 65-79). Stylus Press.
mathematics. PNAS, 111(23), 8410-8415. https://doi.org/10.1073/pnas.1319030111
Gadgil, S., Braun, M., Harty, M., Hovis, K., & Lovett, M. (2018). Investigating the impact of an online
collaboration course on students’ attitudes and learning. In Kay, J. and Luckin, R. (Eds.).
Rethinking learning in the digital age: Making the learning sciences count, 13th International
Conference of the Learning Sciences (ICLS) Volume 1 (pp. 536-543). International Society of the
Learning Sciences. https://doi.dx.org/10.22318/cscl2018.536
Ginsberg, M. B. and R. J. Wlodkowski. (2009). Diversity & motivation: culturally responsive teaching in
college. John Wiley & Sons. https://doi.org/10.1111/j.1467-9647.2011.00724.x
Lovett, M., & Hershock, C. (2020). Cultivating and sustaining a faculty culture of data-driven teaching
and learning: A systems approach. To Improve the Academy, 39(1), 63-93.
http://dx.doi.org/10.3998/tia.17063888.0039.104
Mayer, R. E. (2009). Multimedia learning (2nd ed.). Cambridge.
293
Pedaste, M., Mäeotsa, M., Siimana, L. A., de Jong, T., van Riesen, S. A. N., Kamp, E. T., Manolic, C. C.,
Zachariac, Z. C., & Tsourlidakid, E. (2015). Phases of inquiry-based learning: definitions and the
inquiry cycle. Educational Research Review, 14, 47-61.
Szpunar, K. K., Khan, N. Y., & Schacter, D. L. (2013). Interpolated memory tests reduce mind wandering
and improve learning of online lectures. PNAS, 110(16), 6313-6317.
Tafoya, E., Sunal, D. W., & P. Knecht, P. (1980). Assessing inquiry potential: A tool for curriculum decision
makers, School Science and Math, 80(1), 43–48. https://doi.org/10.1111/j.1949-
8594.1980.tb09559.x
Wright, M. (2008). Encouraging the Scholarship of Teaching and Learning: Helping instructors navigate
IRB and FERPA. Presentation at Organization of Professional Organizers and Developers, Reno,
NV.
Wright, M. C., Finelli, C. J., & Meizlish, D. (2011). Facilitating the scholarship of teaching and learning at a
research university. Change: The Magazine of Higher Learning, 43(2): 50-56.
294
In Their Own Words The Learning Scientists: Promoting
Communication About the Science of Learning
to Apply the Science of Learning in Megan A. Sumeracki
Rhode Island College
Editors
Lauren L. Kordonowy
Althea N. Kaminske
Victor A. Benassi
St. Bonaventure University
Carolina E. Kuepper-Tetzel
University of Glasgow
Cynthia L. Nebel
Vanderbilt University
The Learning Scientists is an international group of cognitive psychologists dedicated to making the
science behind effective learning more accessible to a general audience. The mission of the Learning
Scientists is to communicate the science behind learning in a way that is easily accessible to many and to
provide a platform for bi-directional communication about putting that science into practice.
Research from cognitive psychology can offer invaluable insights into how learning works for anyone
interested in learning: teachers, parents, students, and other educators. However, cognitive researchers
cannot simply conduct research, publish the findings in journals, and hope for the best. Although
publishing in scientific peer-reviewed journals is essential to move our science forward, these scientific
journals contain several barriers to access. First, many scientific journals charge a fee—paid by either an
individual or a library subscribing to the journal. Researchers are typically affiliated with university
libraries that cover those subscription costs, but people outside higher education are unlikely to have
access to those resources. Second, even if one does acquire an exciting new research paper on learning
(they found a free pdf or the journal was open access), most of these articles are written for researchers
and contain jargon and conventions with which non-academics are unlikely to be familiar. Third,
applying research findings directly to teaching or learning is nuanced. It is an iterative process, requiring
a high level of understanding of both a large body of research in cognitive and educational psychology
and the educational domain to which it is being applied. This takes a great deal of time, of which most
educators have very little. Our experience is in accord with other cognitive scholars (e.g. Mayer, 2012;
Roediger & Pyc, 2012): the application of evidence-based practice to education works best when
practitioners and researchers collaborate, engaging in bi-directional communication. Unfortunately,
Roediger and Pyc (2012) note that after over a century of trying, this bi-directional communication is
infrequent. We created The Learning Scientists to try to make this type of communication the norm.
We are a collaborative team of four female cognitive scientists, passionate about making the science of
learning freely accessible to anyone and everyone for whom it can benefit. We collaborate on the
Learning Scientists project as a part of our professional service and as volunteers, focusing primarily on
providing free resources available to everyone who can access the Internet. Each of us is a full-time
faculty member at an institution of higher education. We each work within programs that are different
from one another, giving each of us a unique perspective (public medium-sized college with a focus on
teaching and research, large public research university, small private liberal arts university, and a
doctoral program at a large university). As faculty in higher education, we are actively engaged in
295
research and scholarship, teaching, and service to our institutions and fields. This allows us to not only
stay current on important learning research, but also to be practitioners as we implement best practices
in our own classrooms.
From the Lab to the Classroom

Cognitive psychologists study mental processes: perception, attention, learning, and memory. Our field
can provide practical applications to education by applying our understanding of how we learn.
However, before applying our work to education, it is important to make sure that the strategies that
work in a cognitive psychology laboratory still apply in authentic classroom settings. Therefore, we—
cognitive psychologists applying research to education—use the lab-to-classroom model (Weinstein &
Sumeracki, 2019). In this model, researchers first begin with basic laboratory research. At this level, we
use very simple materials, such as nonsense syllables, lists of words, or pairs of words. This basic
research allows us to maintain tight control over many factors within the experiment to determine what
processes are actually causing learning to increase (or decrease). Once we establish cause-and-effect
relationships, we then move to the applied laboratory level. At this level, we still conduct research in the
laboratory to maintain some experimental control, but we have participants engage with and learn
realistic educational materials, such as passages from a textbook or a video lecture. Finally, we bring the
research into authentic educational settings at the applied classroom level. At this level, we alter
instruction in a real classroom setting. We do our best to maintain some experimental control, but
maintaining the same tight control that we have in laboratory settings is unlikely. Research spanning
these three levels allows cognitive psychologists in our field to determine what factors affect learning in
actual educational settings. At first, research on a particular learning strategy may progress through the
three levels in order, but as it progresses, research continues at all levels of the model. Through this lab-
to-classroom research process, cognitive psychologists have identified several highly effective learning
strategies that we are confident recommending to students and educators (see Pashler et al., 2007).
Although we have identified highly effective strategies that can be implemented in the classroom and
during independent study, it is less clear how to best instruct students and educators to utilize these
effective learning strategies. Despite many decades of research demonstrating robust and flexible
learning strategies, a report published in 2016 by Pomerance and colleagues indicated that few teacher-
training textbooks cover the use of effective learning strategies from cognitive psychology (cf. Rowe &
Hattie, this volume). Further, survey data suggest that many students in higher education are unaware
of these effective practices, frequently choosing to reread their course materials instead of engaging in
effective strategies (Bjork et al., 2013; Hartwig & Dunlosky, 2012). Thus, research from cognitive
psychology needs to be more widely accessible in order to have a positive impact on education. We, the
four authors of this chapter and The Learning Scientists team, believe it is vital to identify the best ways
to make research on the science of learning more accessible to students and educators and to teach
students to engage in effective learning strategies on their own (cf. chapters in Part 4 of this volume).
To truly have an impact in education, researchers need to investigate the best ways to teach students
and educators about learning strategies, such that they put their knowledge about effective strategies
into practice in many different learning situations. This is a complicated issue. Cognitive psychology
research shows us how we ought to teach and promote learning; teaching about learning strategies
should be no different. For example, we know that retrieval practice and spacing of study are largely
beneficial at producing durable long-term learning. Therefore, to teach about retrieval practice and
spacing, we should implement retrieval practice and spacing in our courses as we teach about effective
strategies. However, a big challenge is that learning tends to be very context specific (Detterman, 1993).
For example, a high school student may learn to use retrieval practice in their first-year history class, but
fail to recognize that this strategy should also be used in first-year literature classes, or even second year
296
history with a different teacher. Our goal is to encourage teachers and students to use effective
strategies in the classroom and during independent study across a variety of learning situations and to
continue to use them spontaneously (i.e., without prompting) (Manalo, et al., this volume).
Spontaneous use of effective study strategies by students is essentially an issue of transfer. Transfer
refers to the generalization of prior knowledge across different contexts (Barnett & Ceci, 2002). Transfer
can be relatively near (e.g., generalizations to situations that are highly similar) or far (generalizations to
situations that are not similar, such as very different times, settings, knowledge domains, etc.). Transfer
of prior knowledge, especially far transfer, can be remarkably challenging for students (Detterman,
1993). In general, people struggle to spontaneously transfer concepts from one context to another. A
recent survey of study habits among school-age children found that transfer of learning strategies is no
exception. Tullis and Maddox (2020) found that students’ use of retrieval practice varied by domain, for
example, students were more likely to use retrieval practice to study for their math class than for their
social studies class. Thus, students may learn how to use an effective learning strategy in one subject
area or with one teacher, but they may fail to generalize that strategy to other contexts. Cognitive
psychology research gives us hints about how we might support transfer (e.g., Butler, 2010; Kaminske et
al., 2020); however, far and spontaneous transfer is still difficult to achieve (e.g., Manalo, et al., this
volume; Wooldridge et al., 2014;).
There has been some limited research investigating ways of teaching students about effective learning
strategies that induce transfer of the learning strategy to future study opportunities. In a classroom
study, Einstein and colleagues (2012) found that participating in a retrieval practice experiment as part
of a class project led students to report greater use of retrieval practice later in the course. Presumably,
experiencing how retrieval practice improved their own learning in the experiment led them to use the
strategy on their own. However, this study did not include a control group in which students did not
participate in the retrieval practice experiment, and the results relied on accurate self-report of strategy
use across an entire semester.
Ariel and Karpicke (2018) provided a set of laboratory experiments that address these problems. In their
experiments, they implemented a simple intervention intended to increase students’ use of repeated
retrieval practice during learning. College students were randomly assigned to the retrieval practice
instructions group or the control group. The retrieval practice instructions group was briefly taught
about the benefits of retrieval practice and instructed on how to use retrieval practice to learn foreign
language translations. The control group did not receive retrieval practice instructions, and instead were
told to learn the translations so that they could remember as many as possible later. Students then
learned Lithuanian-English translations, and they were given a choice about whether to continue by
either reading the translations, practice retrieval by typing in the English translation, or dropping the
translation from further study. Relative to the control group, the retrieval practice instructions led to
greater use of retrieval practice during learning, and subsequently, students showed greater
performance on an assessment test. Notably, in a second experiment, the students then returned to the
laboratory to learn Swahili-English translations, and those who were in the retrieval-practice instructions
condition were much more likely to engage in retrieval practice during learning than those in the control
group. While this study is encouraging, this is an example of relatively near transfer. More research is
needed to determine whether a brief intervention, like the one in Ariel and Karpicke’s (2018)
experiments, would lead students to continue to use learning strategies taught during an intervention
across different learning situations.
Research on the best ways to teach students and educators about learning strategies that promote
transfer is an evolving area. The goal of The Learning Scientists project is to make the science of learning
more easily accessible to teachers and other educators, students, and parents, in the hopes that it will
297
be used more in education, and as more research is conducted in this area, we will continue to improve
our methods of providing information about the science of learning.
Communicating the Science of Learning

Our goal is to communicate the science behind learning in a way that is easily accessible to those who
may benefit from this knowledge—teachers, students, parents, or even researchers—and to provide a
platform for bi-directional communication between researchers and educators about putting that
science into practice. With that in mind, we have created a variety of resources and ways of interacting
with our audience. When it comes to science communication, it is important to allow your target
audience to interact with the content in different ways and through different channels (Eveleth, 2016).
This contributes to a more enriching experience for them – and to be completely honest – it also makes
our work much more exciting. To that end, we have built a portfolio of resources and activities to
communicate with a diverse audience about the science of learning.
In this section, we will highlight the core components of our current Learning Scientists’ portfolio. Most
of our resources and services are free to use and can be accessed via our website. Since our 2016
launch, the yearly visits on our website have continuously increased from initially 207,586 visits in 2016
to 887,127 visits in 2020. To date, we have had 3.4M visits to our website. We currently have 7K+
subscribers to our weekly emails announcing new blogs and podcast episodes. With many requests for
more individualized consultation or workshops, we sometimes offer Continuing Professional
Development (CPD) events, talks, or consulting work, which we tailor specifically to schools or
organizations.
Blog Posts
Among the most popular content of the Learning Scientists are the blog posts, with some posts
attracting a total of 300K+ page views. A new post is published every week on Thursdays. We feature
three types of blog posts: Learning Scientists posts, Digest posts, and Guest posts. Our posts are usually
targeted towards our main audience (i.e., teachers, students, parents, and/or researchers), but we also
have posts that are for a general audience.
The Learning Scientists posts are written by one of the core members and can take different forms, such
as posts on new research findings (e.g., “Increasing Academic Performance Through Mark
Withholding”), on the application of cognitive science in the classroom (e.g., “Behaviorism in the
Classroom”), or on introductions to research concepts in cognitive psychology (e.g., “Different Research
Methods: Strengths and Weaknesses”). We have also ventured into publishing video blog posts too,
which have been welcomed by our audience (e.g., “How To Take Notes In Class – A Video for Students”).
Our Digest posts are collections of five resources per post revolving around a specific topic. These posts
are a way for us to provide our readers with a wider perspective on a topic beyond what they can find
on our website. Digests offer a quick way to learn about a concept from different sources, but we also
use them to highlight different perspectives of an issue. The five resources are carefully curated to make
sure they fit our mission. We have digests focusing on practical topics (e.g., “How To Read An Academic
Paper”), introducing specific tools (e.g., “There’s an App for That”), and discussing general issues (e.g.,
“Students Under Stress”). Most of the digests are written by one of the Learning Scientists, but we also
have guest digest posts.
For the Guest posts, we encourage anyone interested in the science of learning to contribute posts to
our website. We have Guest posts from teachers, parents, students, and other researchers. All Guest
posts are reviewed by a member of the Learning Scientists to ensure that they align in regard to scope,
format, and quality (see our Guest post guidelines). Similar to the Learning Scientists post, the Guest
298
posts can take different forms: We have interview-style posts (e.g., “An Interview with an Elementary
School Teacher and Doctoral Student”), digests (e.g., “Making Your Material Digitally Accessible”), or
traditional posts (e.g., “Using Evidence Based Strategies to Improve the Distance Learning Experience”).
Downloadable Materials
Our downloadable materials are intended to give a quick overview of six evidence-based learning
strategies in the form of posters, presentation slides, bookmarks, and stickers. All materials feature
illustrations by Oliver Caviglioli and were created in collaboration with him. The materials are free to be
used for educational purposes under a creative commons license (CC BY-NC-ND 4.0). To date, we have
had 630K page views of our downloadable materials, and we have seen our materials being used in
classrooms around the globe. Because many teachers print the posters for entire classes, schools, and
even send them home to parents, we know the page views vastly underestimates their reach. The
popularity of these materials, and our aim to cater to an international audience, led us to work with
translators to translate the posters into 11 other languages—i.e., Czech, Dutch, Finish, French, German,
Korean, Polish, Portuguese, Spanish, Swedish, and Turkish, and we translated the presentation slides
into 3 other languages—i.e., Dutch, German, and Ukrainian. We receive translation requests to other
languages on a regular basis. Although we want to offer our materials in more languages, we lack the
time and financial resources to work on further translations. However, this is certainly something on our
list for the future.
Podcast
We started our Learning Scientists podcast in 2017, and we have recorded 57 episodes to date. Podcasts
offer an exciting medium to communicate the science of learning and allow our audience yet another
way to interact with our work. Our podcast episodes vary in format to keep it interesting. We have
episodes that feature interviews (e.g., “An Interview with the Co-Founders of Podsie”, “An Interview
with Memory Expert Boris Konrad”), that discuss a recent research paper in depth (e.g., “Metacognitive
Monitoring of Adolescents and Young Adults”), or that give a research overview into a topic (e.g., “Bite-
Size Research on Seductive Details”).
Other Outreach Activities
Besides the previously described resources, we also have established a presence on social media. Our
Twitter account, @AceThatTest, has currently 27.6K followers, and we are mentioned in many tweets
each day. Twitter has allowed us to increase our reach and to directly interact with our audience. In fact,
Twitter has been a catalyst for our activities in general because of the large education-focused Twitter
community. We host a monthly Twitter chat (#LrnSciChat), where we discuss topics in more detail and
sometimes invite guest hosts too (e.g., ‘#LrnSciChat on 27 May 2020 | Applying Cognitive Science
Principles During Remote Learning’, ‘#LrnSciChat on 23 Nov 2020 | Teaching Students How To Study’).
Our Learning Scientists Facebook page has 7.5K followers, and we mainly use it to share new content
from our website. People can leave comments or questions which we address directly. We also have a
Learning Scientists Q&A Group on Facebook, where people post questions, and the community provides
answers or shares resources.
As mentioned previously, most of our activities and resources are free to access and use. We do,
however, have operation costs (e.g., website hosting costs and newsletter services). To cover these
expenses, we have created a Patreon Learning Scientists page where people can support our work. We
currently are supported by 70 patrons and offer five tiers that come with different benefits. Our most
popular tier gives patrons access to a monthly office hour video that features one of the Learning
Scientists explaining a concept or describing a research finding and its application.
299
How We Encourage Educators to Use the Science of Learning
In addition to the many free resources available on our website, we are also frequently asked to engage
in talks and workshops for a variety of audiences. Because this type of work involves a great deal of
additional time and sometimes travel resources, we do charge for talks and workshops. One of the more
common questions that we receive during these workshops involves how to apply these principles in
very specific circumstances. Educators, understandably, are asking not just what the strategies are, but
how exactly they should be implemented in their unique classrooms. In general, we do not provide
consults at that level. We can offer ideas, suggestions, and concrete examples of the strategies in use,
but we know from the research that these strategies do not always work the same way with all learning
materials (e.g., Wooldridge et al., 2014) or students (e.g., Karpicke et al., 2014). In addition, our
understanding of learning science is constantly evolving. As new research is conducted on the boundary
conditions surrounding these strategies, so too does our advice change about how and when they
should be used.
Instead of treating these strategies as “silver bullets” or absolutes, we encourage educators to view
them as flexible guiding principles for how to achieve greater learning in the classroom. This can be
achieved in one of two ways. Educators can use effective learning strategies to update their own
curriculum and classes in order to achieve better learning in the classroom, or they can instruct students
to utilize these strategies in their own study. For example, an educator could utilize spaced or
distributed review in their course by mapping out when various topics will be reviewed throughout the
academic year. They could, instead (or in addition), instruct students in how to use their planners to
develop a spaced review schedule for their own review at home.
While we cannot necessarily provide exact advice for any given learning environment, we do often
provide concrete examples that apply to varied audiences and disciplines. We have written reviews that
discuss how to apply strategies for effective learning in the food sciences (Sumeracki et al., 2019),
medical education (Nebel & Sumeracki, in press), biology (Kaminske et al., 2020), and continuing health
professions education (Van Hoof, Madan, & Sumeracki, 2021; Van Hoof, Sumeracki, & Madan, 2021). In
these reviews, we work with co-authors or editors from the target discipline to ensure that the concrete
examples provided are accurate and showcase an effective use of the learning strategy. We have given
talks and workshops for a variety of audiences, including museum docents, federal employees working
in the State Department, and medical educators and residents, in addition to our primary audience of K-
12 educators and those in higher education. Our audiences also vary by age and role; we have spoken to
elementary students, parents, and administrators. For varied situations such as these, we are able to
provide concrete examples of how the strategies could be applied, but the decision of how to apply
them in individual classrooms and learning situations must remain up to the instructors.
Fundamentally, there is an art and science to teaching and instruction. As relative experts in the science
of learning, we can provide information about basic principles of memory and cognition and about
effective learning strategies that build on those principles. We can draw from literature filled with well-
controlled experiments to teach about the processes that cause learning and various ways those
processes can be infused into various learning activities. Educators, however, know their classrooms and
their content best, and while they should use caution in determining when and how to apply those
principles (Nebel, 2020), they have a better understanding of their unique learning situations. Just as,
using medicine as an analogy, ibuprofen can reduce pain, but the actual type of pain, reason for the
pain, type of patient, dosage, etc. determine when ibuprofen may be appropriate; it is up to the
individual practitioners to make those decisions when treating their patients.
300
Conclusion
The Learning Scientists project has grown and evolved by responding to the requests of the individuals
we aim to support. Our mission to provide accessible resources, bi-directional communication, and to
spread the science of learning has remained, even as our portfolio of resources has grown and expanded
over time. Perhaps most important to us as researchers and science communicators, is how much we
have learned about the field of education in general, as well as the myriad situations where the science
of learning can be applied. We believe, more strongly than ever, that it takes a village. We need diverse
experiences and expertise to come together to push education forward. We encourage researchers and
educators alike to push for more bi-directional communication so that we can move forward together in
improving educational outcomes.
Author Note
Megan A. Sumeracki, Psychology Department, Rhode Island College; Althea N. Kaminske, Psychology
Department, St. Bonaventure University; Carolina E. Kuepper-Tetzel, School of Psychology and
Neuroscience, University of Glasgow; Cynthia L. Nebel, Department of Leadership, Policy, and
Organizations, Vanderbilt University Peabody College.
Correspondence concerning this article should be addressed to Megan Sumeracki, Psychology
Department, Rhode Island College, Providence, RI 02908. Contact: [email protected].
References
Ariel, R., & Karpicke, J. D. (2018). Improving self-regulated learning with a retrieval practice intervention.
Barnett, S. M., & Ceci, S. J. (2002). When and where do we apply what we learn? A taxonomy for far
transfer. Psychological Bulletin, 128(4), 612–637. https://doi.org/10.1037//0033-2909.128.4.612
Bjork, R. A., Dunlosky, J., & Kornell, N. (2013). Self-regulated learning: Beliefs, Techniques, and Illusions.
Annual Review of Psychology, 64(1), 417-444. https://doi.org/10.1146/annurev-psych-113011-
143823
studying. Journal of Experimental Psychology: Learning, Memory, & Cognition, 36(5), 1118-1133.
Detterman, D. K. (1993). The case for prosecution: Transfer as an epiphenomenon. In Detterman, D. K.,
& Sternberg, R. J. (Eds.), Transfer on trial: Intelligence, cognition, and instruction (pp. 1–24).
Ablex.
Einstein, G. O., Mullet, H. G., & Harrison, T. L. (2012). The testing effect: Illustrating a fundamental
concept and changing study strategies. Teaching of Psychology, 39(3), 190-193.
Eveleth, R. (2016). Getting interactive. In C. Wilcox, B. Brookshire, & J. G. Goldman (Eds.), Science
blogging: The essential guide (pp. 79-86). Yale University Press.
Hartwig, M. K., & Dunlosky, J. (2012). Study strategies of college students: Are self-testing and
scheduling related to achievement? Psychonomic Bulletin & Review, 19(126), 126-134.
301
Kaminske, A. N., Kuepper-Tetzel, C. E., Nebel, C. L., Sumeracki, M. A., & Ryan, S. P. (2020). Transfer: A
review for biology and the life sciences. CBE--Life Sciences Education, 19(3), 1-11.
Karpicke, J. D., Blunt, J. R., Smith, M. A., & Karpicke, S. S. (2014). Retrieval-based learning: The need for
guided retrieval in elementary school children. Journal of Applied Research in Memory and
Mayer, R. E. (2012). Advances in applying the science of learning to education: An historical perspective.
Journal of Applied Research in Memory and Cognition, 1(4), 249-250.
Nebel, C. (2020). Considerations for applying six strategies for effective learning to instruction. Medical
Science Educator, 30, 9-10. https://doi.org/10.1007/s40670-020-01088-8
Nebel, C. & Sumeracki, M. (in press). The science of learning. In K. Huggett & W.B. Jeffries, An
introduction to medical teaching. Springer. https://doi.org/10.1007/978-94-017-9066-6
Pashler, H., Bain, P. M., Bottage, B. A., Graesser, A., Koedinger, K., McDaniel, M., & Metcalfe, J. (2007).
Organizing instruction and study to improve student learning (NCER 2007-2004). Washington,
DC: National Center for Education Research, Institute of Education Sciences, U.S. Department of
Education. https://files.eric.ed.gov/fulltext/ED498555.pdf
Pomerance, L., Greenberg, J., & Walsh, K. (2016, January). Learning about learning: What every teacher
needs to know. National Council of Teacher Quality.
http://www.nctq.org/dmsView/Learning_About_Learning_Report
Roediger, H. L., & Pyc, M. A. (2012). Applying cognitive psychology to education: complexities and
prospects. Journal of Applied Research in Memory and Cognition, 1(4), 263-265.
Sumeracki, M. A., Weinstein-Jones, Y., Nebel, C. L., & Schmidt, S. J. (2019). Encouraging knowledge
transfer in food science and nutrition education: Suggestions from cognitive research. Journal of
Food Science Education, 18(3), 59-66. https://doi.org/10.1111/1541-4329.12158
Tullis, J. G., & Maddox, G. B. (2020). Self-reported use of retrieval practice varies across age and domain.
Metacognition and Learning, 15, 129-154. https://doi.org/10.1007/s11409-020-09223-x
Van Hoof, T. J., Madan, C. R., & Sumeracki, M. A. (2021). Science of learning strategy series: Article 2,
retrieval practice. Journal of Continuing Education in the Health Professions, 41(2), 119-123.
https://doi.org/10.1097/CEH.0000000000000335
Van Hoof, T. J., Sumeracki, M. A., & Madan, C. R. (2021). Science of learning strategy series: Article 1,
distributed practice. Journal of Continuing Education in the Health Professions, 41(1), 59-62.
https://doi.org/10.1097/CEH.0000000000000315
Weinstein, Y., & Sumeracki, M. A. (2019). Understanding how we learn: A visual guide. David Fulton,
Routledge.
Wooldridge, C., Bugg, J., McDaniel, M., & Liu, Y. (2014). The testing effect with authentic educational
materials: A cautionary note. Journal of Applied Research in Memory and Cognition, 3(3), 214-
302
In Their Own Words Different Goals Imply Different Methods: A
Guide to Adapting Instructional Methods to
Your Context
Kenneth R. Koedinger
Editors
Lauren L. Kordonowy
Victor A. Benassi

Martina A. Rau
University of Wisconsin, Madison
Elizabeth A. McLaughlin
Overview
It would be great if effective methods for teaching worked in all contexts, irrespective of the nature of
the content and student prior knowledge. It would be terrible if every context required a different,
unpredictable, method for ideal learning. Learning research has shown us that the likely truth is
somewhere between. Although the best choices for teaching methods do not work in all contexts, there
are predictable features of contexts, and especially of learning goals, that can guide effective choices.
This chapter provides such guidance so you can effectively select and adapt appropriate teaching
methods for the particular learning goals you have in your instructional context.
A key message here is that debates about the best teaching methods can often be resolved by
recognizing that different learning goals imply different kinds of knowledge, skills, and dispositions,
which, in turn, suggest different ideal learning methods (cf., Metcalfe, this volume). Sometimes
memorizing arbitrary facts is important, like the Earth is the third planet from the sun. Sometimes deep
reflective understanding is important, like the reasoning behind scientific principles. These different
goals, memorizing facts versus understanding general principles, imply different ideal teaching methods.
There are other distinctions between learning goals, besides fact memorization and principle
understanding, that are relevant to selecting effective methods. This chapter will present those
distinctions and, for each kind of learning goal, suggest evidence-based teaching methods that have
been demonstrated to be effective in rigorous experimental research on learning.
Our recommendations draw upon the Knowledge-Learning-Instruction Framework (Koedinger et al.,
2012), which is called KLI (pronounced “klee”) for short. The KLI framework is a consequence of a 10+
year effort involving a large group of researchers and educators to apply evidence-based teaching
methods to new subject matter and to perform randomized controlled experiments within courses in
math, science, and second language at the middle school, high school, and college levels. Sometimes
these experiments produced the result predicted by prior studies, but not always. We were often
surprised by cases where we did not get the predicted result. For example, instructors (ourselves
included) tend to expect that presenting students with multiple visualizations of an idea (e.g., circle and
bar representations of fractions) enhances their learning. However, research shows that this is not
necessarily the case, as detailed below.
The development of KLI was not only driven by these experiments, but also by theories of learning both
from cognitive science and from the Artificial Intelligence area of machine learning. Just as we saw a
303
great variety of learning goals and teaching methods, the literature also reveals a great variety of
learning theories, including theories of memory, of pattern learning, of skill acquisition, of information
comprehension, and of sense making. In short, human learning is highly varied and richly complex.
KLI Helps Categorize Learning Goals and Direct Selection of

Appropriate Instructional Methods
A key observation of KLI is that different learning processes are needed for different learning goals. For
some learning goals, verbatim (or literal) recall is needed, for example, to memorize foreign language
vocabulary, that “k” makes the ka sound, the value of constants in science or math, and the date of an
important event. For some learning goals, learning situational patterns and appropriate responses to
those patterns are needed, for example, to acquire skills in reading or writing (e.g., knowing when to
add “ies” to pluralize a noun) and in math or science (e.g., recognizing which two formulas to apply to
find the area of an irregular shape made up of two regular shapes). For some learning goals, reflecting
and making sense of a situation is critical. Making sense is important in evaluating the merits of an
argument, in understanding how molecular structure constrains a chemical reaction, and in analyzing
the motivations of Shakespeare’s Hamlet.
Different learning processes are needed for memory of a verbatim fact, for generalizing or inducing a
pattern or skill, and for making sense of a general principle. In each case, different instructional methods
are ideal to achieve that goal. As we discuss dependencies between learning goals and different
methods of teaching, we will provide examples of results from experimental research studies showing
both when a particular method works well and when it does not (i.e., its opposite method may work
better).
What Are Important Distinctions Between Goals?
You can distinguish learning goals based on the nature and complexity of the knowledge to be learned.
The KLI Components of knowledge involve a retrieval condition, which can be either a single item
(“constant”) or a general pattern (“variable”) and a response, which can also be either a single element
or a general pattern. Knowledge can be in a verbal or a non-verbal form (e.g., something one can do that
they cannot directly express). Some knowledge is arbitrary, a matter of design or convention, and some
knowledge has a rationale.
Facts to Remember
The KLI framework uses the term “fact” to refer to relatively simple types of knowledge that express an
arbitrary or conventional association between two particular items, like a word and its definition, a
historical event and the date it occurred, or a foreign language word and its English translation. Learning
facts, in the KLI sense, involves only verbatim memory—there is no need for nuanced generalization and
no opportunity for understanding because the association is a matter of convention (e.g., the French
picked “amour” whereas the English picked “love” to express the same idea).
In KLI, facts are types of knowledge where the retrieval condition is one item (e.g., English for amour)
and the response is one item (e.g., love). Note that this technical use of “fact” differs somewhat from
everyday use. People sometimes use the term “fact” to describe a piece of information that is
uncontestable. However, although some uncontestable pieces of information are indeed facts (e.g., π is
the symbol chosen to represent the ratio of a circle’s circumference to its diameter), other “facts” are
actually principles (e.g., Newton’s first law) in the KLI sense. Thus, in KLI, a fact is more narrowly
defined—it is indeed something that is true, but merely by convention (e.g., Саша in Russian is
304
pronounced "Sasha" in English). In contrast, a "principle" in the KLI sense is not arbitrary; it can be
rederived or rediscovered.
Students learn facts through memory processes. Memory-based learning is improved through
repetition, spaced exposure or practice, and active retrieval practice (Anderson & Lebiere, 1998; Logan,
1988). Facts cannot be induced based on generalizations: many examples of German words for different
berries do not produce a pattern that yields an unknown translation. Additionally, KLI facts cannot be
understood based on principled reasoning: there is no derivation for why π is the symbol for the ratio of
a circle’s circumference to its diameter. As we move on to more complex forms of knowledge, it is worth
noting that all forms of acquired knowledge are subject to forgetting and will fade without use. Thus,
memory-enhancing instructional methods, like repetition, spacing, and practice, remain relevant even if
other methods are more distinctly effective.
Skills to Generalize
Skills are more complex than facts because they are adaptable. We define skills as types of knowledge
where a variety of retrieval conditions can be mapped to a variety of responses (cf., Gentner et al.,
2009), while this mapping is nonverbal in nature (Alibali & Koedinger, 1999; Dienes & Perner, 1999). For
example, the skill of finding the area of triangle can be applied in a variety of conditions (i.e., the triangle
can have a different sized base or height), and the response varies accordingly (i.e., the computed area
depends on the particular base and height). Competence in this skill does not guarantee that it can be
fully and accurately explained in words, especially nuances of condition. Try expressing precisely how to
identify the base and height of a triangle or all the ways you choose to use the article “the” instead of
“a”. Certainly, many motor skills are not easily expressed in words and, in fact, experts who attempt to
express them are sometimes wrong (e.g., some expert tennis players indicate flipping their wrist
imposes top spin when it is actually the upward motion of the racket). Skills, however, are much more
ubiquitous—everything you learn to do is based on skill acquisition: reading, writing, problem solving,
composing, analyzing, designing, inventing, and interacting. Many skills are learned outside of conscious
awareness, such as skills of social interaction (e.g., nods, gestures, and facial expressions) that can
indicate attentiveness, confusion, a desire to move on or to say something, etc. Skill acquisition requires
accurate generalization from one situation to another (Anderson & Lebiere, 1998). For example, an
expert reader has acquired skills to comprehend text with words used in combinations they have never
seen before.
Students learn skills by observing and practicing them. For example, young children get an immense
amount of practice in forming plural nouns by listening to and producing them in speech. Tennis players
practice hitting balls in a variety of situations when they play. Practice situations generally offer
feedback; for example, turn-taking in social interactions offers feedback by either being successful or
resulting in a social error (e.g., unintentionally interrupting another person).
Principles to Understand
Principles are yet more complex than skills (Koedinger et al., 2012). KLI principles are defined as types of
knowledge where, like skills but unlike facts, a variety of retrieval conditions can be mapped to a variety
of responses (Gentner et al., 2009), where, unlike facts, the mapping has rationale and, unlike skills, this
mapping is verbal in nature (Alibali & Koedinger, 1999; Dienes & Perner, 1999). That is, principles are
similar to skills in the sense that they are adaptable. For example, there are many ways to explain how
global warming results in rising sea levels. However, there is a rationale that links the retrieval condition
and the response that can be explained. For example, the rationale linking global warming and rising sea
levels is that land-based polar ice caps melt, adding more water in the oceans.
305
Like skills, well-acquired principles generalize across situations (Anderson et al., 1995; Li et al., 2015). For
example, a student who has understood the relation between melting ice caps and rising sea levels
might wonder whether the level of liquid water in a glass with ice will go up when the ice melts.
One way students learn principles is through explanations (Asterhan & Schwarz, 2009; Chi et al., 1989;
Graesser et al., 2005). Students may construct explanations themselves; for example, they may read a
sentence stating that global warming causes rising sea levels and may wonder why, which may cause
them to come up with an explanation by drawing on what they already know about the issue. Students
can learn from “self-explaining” even without saying it out loud (see Chi & Boucher, this volume).
Students can get further benefits from being asked to state or write an explanation, whereby a teacher
can provide feedback. It is often important to give students feedback on their explanations so they are
more accountable to engaging in sense-making and can refine the scope and accuracy of their
understanding (cf., Aleven et al., 2001).
Which Teaching Methods Are Best for Each Kind of Learning Goal?
There are many different ways to teach and, more generally, to support student learning. Teaching is
obviously not just about providing clear instructions or explanations. Teaching is also about structuring
activities, including selecting content for those activities, prompting students to perform actions (e.g.,
write, solve, etc.), and providing feedback to students on those actions. Many questions arise: When
should students read versus practice? How much reading and how much practice? Should content and
practice be repeated, and if so, how often and when? Should content and practice on a single topic be
provided all at once or intermixed with content and practice on other topics? Should you have students
read rules or study examples? Should you provide the explanations for the steps in examples or ask
students to generate those explanations? This book is about answers to these questions. This chapter is
particularly about when the answer to these questions changes depending on the context, including
content you are teaching and the prior knowledge of students (cf. Metcalfe, this volume).
Methods for Supporting Long-term Retention of Facts
School is much more than learning and retaining arbitrary facts or associations. At the same time, often
learning facts is important. Fast and easy retrieval of core facts is often crucial for future learning. Fluent
learning of the fact (as per KLI) that the letter “s” makes the sound “es”, is critical to future learning
from reading. Fluent learning of 3+4=7 and other math facts is critical to future learning of multi-digit
arithmetic procedures, like how to add 53+84. Beyond importance of learning some facts, probing what
teaching methods best support fluent fact learning also provides us with a good contrast for exploring
how different methods best support more complex skill and principle learning, as we discuss in the next
sections.
One of the most powerful teaching methods for learning facts is retrieval practice, sometimes called
“testing” (e.g., Roediger & Karpicke, 2006a&b; Yang et al., this volume). The core idea is that repeated
practice retrieving (e.g., recalling the sum of 3+4, of 5+3, etc.) yields better learning outcomes than
repeated study (e.g., reading through 3+4=7, 5+3=8, etc.).
There are other instructional methods that are particularly effective for supporting student learning of
facts, particularly students’ long-term retention of and fluency with them. One simple method for
supporting long-term retention of facts is for students to get repeated practice. Most facts are not
learned with just a few practice opportunities, but require many opportunities, like 10 or more, and
even more to get from accurate to fast and fluent performance (cf., Dunlosky et al., this volume).
Another method important for learning facts is spaced practice (e.g., Pavlik & Anderson, 2005; Rohrer &
Hartwig, this volume). Repetition and spacing are also worthwhile for learning skills and principles—
306
skills and principles need to be remembered and applied. However, as we will see later, more practice
instead of studying examples, while best for learning facts, is not always best for learning more general
problem-solving skills.
To better understand what teaching methods work best for facts, it is helpful to contrast methods that
do not aid learning of facts. Consider the multimedia principle (e.g., Clark & Mayer, 2016; Mayer, this
volume), which states that having students read and study text that includes relevant images or
diagrams often produces better learning than having students read the text alone. Below we discuss
effective uses of the multimedia principle for learning principles, but this method does not improve
memory of arbitrary facts. For example, in one study (Mayer, 1989), students given text with
illustrations (e.g., about hydraulic drum brakes) were no better or somewhat worse at memorizing
arbitrary facts (e.g., “drum brakes consist of a cast-iron drum”) than students just given the text (e.g.,
41% vs. 43% correct in recognizing verbatim text). As we discuss later, students given illustrations were
better at answering transfer questions (e.g., “Why do brakes get hot?”). But when it comes to fact
memory, as the Mayer (1989) study indicates, adding illustrations “should not help in verbatim retention
of factual material” (p. 243).
Methods for Supporting General Skill Learning and Transfer
Skill learning is ubiquitous in academic learning, including decoding skills in reading, math skills for
problem solving, scientific skills for reasoning from data, interpretation skills for extracting meaning
from a story or historical event, writing skills for expressing ideas in compelling and clear terms, social
communication skills for empathetic listening, and contributing new ideas that build on others, etc.
Much of this skill learning happens implicitly through practice; arguably, some of the most powerful
methods for supporting skill learning are summarized in the notion of deliberate practice (Ericsson et al.,
1993). Key features of deliberate practice include: 1. learners are given well-designed practice tasks at
the edge of their competence, 2. they get timely feedback on their performance on these tasks, 3. they
get multiple repeated opportunities to move toward more accurate, desired performance, and 4. tasks
are varied strategically to foster accurate generalizations that facilitate transfer of learning of these skills
into new contexts.
Each of these four factors has been further elaborated. For example, research on feedback (2 above)
suggests that high quality timely feedback a) highlights errors and avenues for improvement, b)
illustrates, when needed, correct or desired performance, and c) provides explanations for why such
performance is correct or desirable.
Together, these deliberate practice recommendations have much in common with the
recommendations for retrieval practice, which are powerful for enhancing memory of verbatim facts.
However, there are both additional and contradictory recommendations that emerge when the learning
goal is general skill learning rather than when it is verbatim fact memory. These differing
recommendations emerge from the nature of skills needed to apply generally but accurately (i.e., not
over-generally) across a variety of different conditions. Thus, varied task presentation (4 above) and
explanatory feedback (c above) are additional recommendations that aid generalized skill learning but
may slow down verbatim fact learning. For skill learning, one not only needs verbatim retrieval practice
but also generalization practice in varied contexts.
Perhaps more important, there are situations in skill learning in which an exclusive focus on retrieval
practice can be counterproductive. Evidence on learning algebra equation solving (Kalyuga et al., 2003),
for example, indicates that novice learners learn better when about half of the retrieval and
generalization practice problems are replaced with worked example solutions (already solved
problems). Multiple studies on this worked-example effect (see Recommendation 2 in Pashler et al.,
307
2007; see also Renkl, this volume) demonstrate that passive study of worked examples prepares
students for generalization practice, facilitating more accurate initiation for student induction of skills
and yielding less overwhelming cognitive load than when novice students engage too soon in practice
problems. Benefits of worked examples for skill learning can be further enhanced by prompting students
to “self-explain”, that is, to provide their own explanations for the decisions/steps illustrated in worked
examples (see Recommendation 7 in Pasher et al., 2007). This self-explanation prompting method is
particularly powerful for procedural skills that convert general principles (covered next) into action.
Methods for Supporting Learning and Understanding of Principles
Some students explain principles spontaneously. However, research shows that many students do not
spontaneously engage in explanation processes (Ainsworth et al., 2002). Hence, instructional
interventions that support learning of principles often seek to engage students actively in explanations
(Chi, 2009). For example, before reading a textbook passage, a teacher may tell students to underline
causal claims and explain these to themselves. An online learning platform may show a message that
asks students to think about examples from their own experiences while watching a video. A workbook
may offer fill-in-the-blank sentences for students to complete to construct an explanation. Such self-
explanation prompts have proven effective in multiple instructional contexts (Aleven & Koedinger, 2002;
Chi et al., 1994; Renkl et al., 1998).
Generally, the goal is to engage students actively in the explanation process (Chi, 2009). A multitude of
methods can achieve this goal; for example, collaboration (Hausmann et al., 2004; Teasley, 1995),
teacher-led classroom discussions, peer tutoring, or computer-based tutoring. When students
collaborate, they may disagree on how to explain a principle, which may prompt them to discuss their
view and agree on a jointly constructed explanation (Schwartz, 1995). Teacher-led classroom discussions
can function the same way. For example, a teacher prompts students to construct explanations
themselves and then invites or provides challenges to their initial ideas (Cobb, 1995). One-on-one
tutoring may involve prompts for explanations (Grasser et al., 2001). Computer-based tutors can prompt
students to input explanations via text or menu-based entry followed by feedback that drives further
thinking (Aleven & Koedinger, 2002; Johnson & Mayer, 2010; van der Meij & de Jong, 2011). Finally, the
methods mentioned above for supporting learning facts or skills can be helpful for learning principles as
well (Koedinger et al., 2012).
Practice-Based Advice in Applying KLI

This section provides practice-based advice on using the KLI framework in making decisions about
teaching or designing instruction. It does so in the form of scenarios where you are asked to think
through what you would do and why, and to compare your predictions and explanations with actual
research results and theory. Consistent with KLI, we especially highlight how the same instructional
method may work well for some learning goals but not for others.
Science Text Recall Scenario
Goals and Assessment. In this scenario, the learning goal for undergraduate students is to enhance
memory of written passages on scientific topics like “The Sun” and “Sea Otters”. These passages
involved about 300 words. Students’ memory of the passage was tested by prompting them with the
name of the passage and counting how many of the 30 key ideas in the passage they were able to
generate in free recall (allowing for some variation in expression of a key idea).
Instructional Method and Your Prediction. To determine which instructional method better helps
students’ textual memory, two methods were investigated. The first method uses retrieval practice
308
where students study (S) the passage once, then are tested (T) three times by asking them to recall the
text (STTT). The second method involves repeated study of the prose (SSSS). In both cases, no feedback
was provided.
What instructional method (practice testing or repeated study) do you predict will better help text
memory, in the short term (5 minutes later) and longer term (1 week later)? Try to answer yourself
before reading on!
Observe Results and Explain. Roediger and Karpicke (2006a) conducted such an experiment and found
that students who practiced testing (STTT) did better on a 1-week delayed post-test than the students
given the method emphasizing study (SSSS), 61% vs. 40% on average, respectively. Interestingly, the
study group (SSSS) did better on the immediate post-test (5 minutes later), recalling 83% of the ideas on
average, than the testing group (STTT), who recalled 71% of the ideas.
What is the theory behind why these results were observed? How can you explain these results? Try to
answer yourself before reading on!
Theoretical Explanation. Applying the KLI framework, we first note that the learning goal involves
memory of facts, that is, memory for the ideas expressed in the text. While verbatim recall of these
ideas was not required, students were asked to list only the ideas present in the text. They were not
asked questions requiring any generalization of the ideas, like making inferences about sea otter
behavior in a context that was not described in the text. The learning goal was to learn specific facts, not
general skills or principles. Memory is the key learning process for facts, neither induction nor
understanding are required. Memory is enhanced by repetition, spacing, feedback, and testing (i.e.,
retrieval practice). The better long-term results for the testing group are a consequence of retrieval
practice, neither feedback nor spacing were provided. Memory is enhanced by testing for various
reasons. The instructional activity (recall) matches the test activity (recall)—one learns best what one
does. Retrieval practice may also provide learners with opportunities to create multiple paths for
retrieval.
What about the immediate test results? Because there was no feedback on the test practice, the test
group had only one exposure to the passage (the first and only S) whereas the study group had the
initial exposure plus three repetitions (the four S’s). With this extra repetition, the study group initially
learned more, reaching 83% rather than 71% correctness on the immediate test. Interestingly, students
thought they learned more from the study group, as evidenced by higher predictions than the test group
that they would remember 1 week later. This result is consistent with the KLI notion that we cannot
directly observe our own learning processes.
What if feedback had been provided? Then the test group would have had equivalent opportunities for
exposure to the content and they would also have tended to perform better than the study group, even
on the immediate test.
Boundary Conditions. If you want students to feel or perform better right after instruction on facts, then
repeated study is better than practice without feedback. However, in the longer-term, practice,
especially with feedback, yields better retention outcomes. When the learning goal is skills, especially
skills involved in multi-step problem solving, then retrieval practice after one opportunity to study text
and an example is not ideal. Instead, novice students learn complex skills better (e.g., Sweller & Cooper,
1985) when they do just as much example study as they test themselves in problem solving practice
(STST). More practice is needed to achieve expertise in skills, and the benefit of worked examples
reverses as students get closer to expertise (e.g., Kalyuga et al., 2003).
309
Geometry Problem Solving, Explanation, and Transfer Scenarios
Geometry Scenario 1
Goals and Assessment. In a geometry class, the learning goals were for students to apply and
understand properties of angles, such as in the context of parallel lines, perpendicular lines, and/or
triangles. Three types of learning were assessed. To assess student learning of applying properties in
problem solving, students were given problems with geometric figures labeled with some properties
and/or angle values. For example, one problem asks: given triangle ABC with two equal sides (AB = BC),
and a value for one base angle, <A = 30, what are the values of the other angles? (Answer: <B=30,
<C=120.) To assess understanding, students were asked to explain their steps. For example, why is
<B=30? (Answer: Because the base angles of an isosceles triangle are equal.) In a second assessment of
understanding, students were given novel “not-enough-information” problems where they were
instructed to decide whether or not they could solve a problem. For example, given triangle ABC, and a
value for one base angle, <A = 30, do you have enough information to find the values of the other
angles? (Answer: No, there is not enough information.)
Instructional Method and Your Prediction. Toward achieving these learning goals, students received
practice with an intelligent tutor system. They were randomly assigned to one of two versions of the
system that used different instructional methods. In version A, students were tutored with as-needed
feedback, hints, and adaptive problem selection on problems like the application assessment question
above. In version B, students were tutored on the same kinds of problems, but they were also asked to
explain each of their steps by typing in or selecting the relevant geometry principle. They received as-
needed feedback and hints on their explanations as well as their problem-solving steps. Students using
version B spent more time on average than students using version A (7.3 hours vs. 6.1 hours). Based on
the results of the three kinds of assessments above, which instructional method, A or B, leads to better
student learning? Consider whether results may be different for the problem-solving application test
versus one of the tests of understanding.
Observe Results and Explain. Researchers (study 1 in Aleven & Koedinger, 2002) found that students
who practiced giving explanations (version B), learned more as demonstrated by better performance on
all three tests (statistically reliable in all cases) compared to students in version A who did not provide
explanations. Student average post-test scores in version A versus version B were 49% vs. 68% for the
problem-solving test, 30% vs. 57% for the explanation test, and 24% vs. 55% for the not-enough-
information transfer test. Why did students in version B learn more?
Theoretical Explanation. Were you tempted to consider reasons why having students provide
explanations would enhance their learning? Providing explanations may have enhanced learning, but
another possibility is that students learned more from version B because they had more opportunities to
practice. The next scenario resolves this uncertainty.
Geometry Scenario 2
Prediction. Now, imagine we give students the same amount of time in the two groups, but otherwise
keep the instruction and assessments the same. This means that students with version B, doing
explanation as well as problem solving, will likely not be able to practice solving as many problems as
students in version A, who are only problem solving. Based on the results of the three kinds of
assessments, which instructional method, problem solving only (A) or problem solving and self-
explanation (B), leads to better student learning?
Observe Results and Explain. This experiment was study 2 in Aleven & Koedinger (2002). With
instruction time the same for the two versions, indeed, the students providing explanations in version B
310
practiced half as many problems (83) as students using version A (166 problems). Not surprisingly,
version B students learned how to provide better explanations than those in version A (52% vs. 42%). It
may be more of a surprise that, despite getting half as much practice, version B students performed just
as well as version A students on the problem-solving post-test (60% vs. 59%). Most important, the self-
explainers (version B) performed significantly better than the pure problem solvers (A) on the not-
enough-information transfer test (59% vs. 41%). Why did students in method B learn to better explain
and transfer their knowledge while learning problem-solving just as well (but not better)?
Theoretical Explanation. Key to explaining these results is the KLI distinction between skills, sometimes
called procedural knowledge (Anderson et al., 1995), and principles, sometimes called declarative
knowledge (Aleven & Koedinger, 2002) or conceptual knowledge (Rittle-Johnson et al., 2001). Self-
explanation supports conceptual learning of the geometry principles and, quite sensibly, practice
explaining produces better outcomes on the explanation assessment. Because principle knowledge is
verbal, it is accessible for reflection—thus, the self-explainers are better able to resist temptation in the
not-enough-information transfer assessment. They can access the verbal explanation (e.g., “base angles
of an isosceles triangle are equal”) and reflect (e.g., “I don’t have enough information because I do not
know if the triangle has two equal sides”). Because skill knowledge is acquired through doing practice
tasks with feedback, the problem-solvers (version A) develop better skill knowledge. Yet, they do no
better on the problem-solving post-test. Why is that? The principle knowledge can also be used, though
more slowly and deliberately, in problem solving. So, the ground self-explainers lost through less
repeated practice toward learning skills, they gained through more opportunities to learn principles
through interactive explanation.
Boundary Conditions. It is important to note that self-explanation is not always an effective instructional
method. Verbatim memory of a face (a kind of fact in KLI) is actually harmed by instructions to verbalize
(or explain) what you saw (Schooler et al., 1997). Second-language learning of accurate selection of the
proper English article to precede a noun phrase (e.g., in this sentence, “a” before “noun phrase”, “the”
before “proper English article”, and nothing before “accurate selection”) is an interesting case. At least
for first language learners, this is non-verbal skill knowledge in KLI, and most of the rules are arbitrary
(e.g., “the Pacific Ocean” but not “the Lake Michigan”). This knowledge is in the skill category, not in the
principle category, in KLI. Thus, KLI predicts self-explanation will not help. Indeed, for this learning goal
(picking the right article), multiple learning experiments (Wylie, 2011) demonstrated no enhanced
learning by inserting self-explanation activities in place of practice.
Fraction Representations and Explanation Scenarios
Fractions Scenario 1
Goals and Assessment. Imagine you are an elementary school math teacher teaching students about
fractions. Your textbook uses number lines, pie charts, and sets to illustrate various fraction scenarios.
You want students to understand that equivalent fractions have the same magnitude.
Instructional Method and Your Prediction. You create tasks with various representations where
students have to draw a representation of an equivalent fraction and decide whether the magnitude has
changed. You only have one class period available for this exercise, so you wonder whether it is
necessary to prompt students to self-explain why the magnitude remains the same (version A), as doing
so will take time. Alternatively, you consider skipping the self-explanation prompts to provide additional
repeated practice opportunities (version B). Which instructional method, A or B, will lead to better
learning (as assessed by a delayed post-test given six days after instruction)?
311
Observe Results and Explain. Rau and colleagues (2015) compared these two conditions in an
experiment on fractions learning and found that self-explanation prompts led to higher learning gains.
Students with version A scored 2.4 out of 3 whereas those with version B scored 1.42 out of 3. How
might you explain this result?
Researcher Explanation. Rau and colleagues (2015) reasoned that, without self-explanation problems,
students were unable to integrate the different representations into one picture and mental model
about fractions, leading to confusion. With self-explanation, students may have been able to understand
the complementary viewpoints of fractions depicted by the different visual representations.
Fractions Scenario 2
Instructional Method and Your Prediction. Regarding the same fractions lesson, you might now wonder:
would it not be better to use only one representation (e.g., only pie charts), so that students are not
confused about how fractions can be represented in so many different ways? Will practicing and
explaining with one representation (version C) yield better learning than practicing and explaining with
multiple representations (version A)?
Observe Results and Explain. Rau and colleagues’ (2015) found that multiple representations with self-
explanation prompts (version A) were more effective than a single representation with self-explanation
prompts (version C): students with version C scored 1.97 out of 3 compared to 2.4 out of 3 for version A.
How might you explain this result?
Researcher Explanation. Rau and colleagues (2015) reasoned that it is good for students to be exposed
to multiple representations that highlight complementary fraction subcontracts (e.g., fractions as parts
of a whole or fractions as parts of a length), but only if students are prompted to self-explain how the
different representations depict information about fractions. Without prompts to self-explain, students
are unlikely to spontaneously reflect on the different fraction interpretations. Therefore, the potential
benefit of multiple representations is only realized when combined with self-explanation prompts.
Altogether, these findings illustrate the importance of including supports that engage students actively
in explanation processes when the goal is to understand complex principles, such as how various
representations depict fractions.
Conclusion
We hope this chapter supports you in making good selective decisions from the very rich and growing
literature on how people learn and how to enhance student learning. It is important to consider the
learning goal you want your students to achieve: Does it involve facts that need to be memorized, skills
that need to be generalized, or principles that need to be understood? Given your goal, select
associated methods like retrieval practice and spacing to support fact memory, worked examples and
varied repeated practice with feedback to support skill generalization, self-explanation, and classroom
dialogue to support principle understanding. Refer to Koedinger et al. (2013) for 30 methods specifically
associated with memory, generalization, or understanding support. Finally, we hope that by prompting
you to explain why and when different methods work and do not work, you are now better able to
select and adapt methods based on your and your students’ needs.
Author Note
This work was supported in part by National Science Foundation grant #BCS-1824257.
312
References
Ainsworth, S., Bibby, P., & Wood, D. (2002). Examining the effects of different multiple representational
systems in learning primary mathematics. Journal of the Learning Sciences, 11(1), 25-61.
https://doi.org/10.1207/S15327809JLS1101_2
Aleven, V., & Koedinger, K. R. (2002). An effective metacognitive strategy: Learning by doing and
explaining with a computer-based Cognitive Tutor. Cognitive Science, 26(2), 147-179.
Aleven, V., Popescu, O., & Koedinger, K. R. (2001). Towards tutorial dialog to support self-explanation:
Adding natural language understanding to a cognitive tutor. In Proceedings of Artificial
Intelligence in Education (pp. 246-255).
Alibali, M. W. & Koedinger, K. R. (1999). The developmental progression from implicit to explicit
knowledge: A computational approach. Behavioral and Brain Sciences, 22(5), 755-756.
https://doi.org/10.1017/S0140525X99222182
Anderson, J. R., & Lebiere, C. (1998). The atomic components of thought. Lawrence Erlbaum Associates
Publishers.
Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995). Cognitive tutors: Lessons learned.
The Journal of the Learning Sciences, 4(2), 167-207.
Asterhan, C. S. C. & Schwarz, B. B. (2009). Argumentation and explanation in conceptual change:
Indications from protocol analyses of peer-to-peer dialogue. Cognitive Science, 33(4), 374-400.
10.1111/j.1551-6709.2009.01017.x
Chi, M. T. (2009). Active-constructive-interactive: A conceptual framework for differentiating learning
activities. Topics in Cognitive Science, 1(1), 73-105. https://doi.org/10.1111/j.1756-
8765.2008.01005.x
Chi, M.T.H., Bassok, M., Lewis, M., Reimann, P., & Glaser, R. (1989). Self-explanations: How students
study and use examples in learning to solve problems. Cognitive Science, 13(2), 145-182.
Chi, M.T.H., de Leeuw, N., Chiu, M.H., & LaVancher, C. (1994). Eliciting self-explanations improves
understanding. Cognitive Science, 18(3), 439-477.
Clark, R. C., & Mayer, R. E. (2016). e‐Learning and the science of instruction: Proven guidelines for
consumers and designers of multimedia learning, 4th edition. John Wiley & Sons, Inc.
Cobb, P. (1995). Cultural tools and mathematical learning: A case study. Journal for Research in
Mathematics Education, 26(4), 362-385. https://doi.org/10.2307/749480
Dienes, Z. & Perner, J. (1999). A theory of implicit and explicit knowledge. Behavioral and Brain Sciences,
22(5), 735-808. https://doi.org/10.1017/S0140525X99002186
Ericsson, K. A., Krampe, R. T., & Tesch-Römer, C. (1993). The role of deliberate practice in the acquisition
of expert performance. Psychological Review, 100(3), 363–406. https://doi.org/10.1037/0033-
295X.100.3.363
313
Gentner, D., Loewenstein, J., Thompson, L., & Forbus, K. D. (2009). Reviving inert knowledge: Analogical
abstraction supports relational retrieval of past events. Cognitive Science, 33(8), 1343-1382.
Graesser, A.C., McNamara, D.S., & VanLehn, K. (2005). Scaffolding deep comprehension strategies
through Point & Query, AutoTutor, and iSTART. Educational Psychologist, 40(4), 225-234.
Graesser, A. C., VanLehn, K., Rose, C. P., Jordan, P., & Harter, D. (2001). Intelligent tutoring systems with
conversational dialogue. AI Magazine, 22(4), 39–41.
http://dx.doi.org/10.1609/aimag.v22i4.1591
Hausmann, R. G, Chi, M. T, & Roy, M. (2004). Learning from collaborative problem solving: An analysis of
three hypothesized Mechanisms. Proceedings of the Annual Meeting of the Cognitive Science
Society, 26, 547-552. https://escholarship.org/uc/item/2ts2g2j6
Johnson, C. I., & Mayer, R. E. (2010). Applying the self-explanation principle to multimedia learning in a
computer-based game-like environment. Computers in Human Behavior, 26(6), 1246-1252.
https://doi.org/10.1016/j.chb.2010.03.025
Koedinger, K.R., Corbett, A.C., & Perfetti, C. (2012). The Knowledge-Learning-Instruction (KLI)
framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive
Science, 36(5), 757-798. https://doi.org/10.1111/j.1551-6709.2012.01245.x
Li, N., Matsuda, N., Cohen, W. W., & Koedinger, K. R. (2015). Integrating representation learning and skill
learning in a human-like intelligent agent. Artificial Intelligence, 219, 67-91.
https://doi.org/10.1016/j.artint.2014.11.002
Logan, G. D. (1988) Toward an instance theory of automatization. Psychological Review, 95(4), 492-527.
https://doi.org/10.1037/0033-295X.95.4.492
Mayer, R. E. (1989). Systematic thinking fostered by illustrations in scientific text. Journal of Educational
Pashler, H., Bain, P., Bottge, B., Graesser, A., Koedinger, K., McDaniel, M., & Metcalfe, J. (2007).
Organizing Instruction and Study to Improve Student Learning (NCER 2007-2004). Washington,
Education
Pavlik Jr, P. I., & Anderson, J. R. (2005). Practice and forgetting effects on vocabulary memory: An
activation‐based model of the spacing effect. Cognitive Science, 29(4), 559-586.
Rau, M.A., Aleven, V., & Rummel, N. (2015). Successful learning with multiple graphical representations
and self-explanation prompts. Journal of Educational Psychology, 107(1), 30–46.
Renkl, A., Stark, R., Gruber, H., & Mandl, H. (1998). Learning from worked-out examples: The effects of
example variability and elicited self-explanations. Contemporary Educational Psychology, 23, 90-
108. https://doi.org/10.1006/ceps.1997.0959
314
Roediger, H.L. and Karpicke, J.D. (2006a). Test-enhanced learning: Taking memory tests improves long-
9280.2006.01693.x
Roediger, H. L. & Karpicke, J. D. (2006b). The power of testing memory: Basic research and implications
for educational practice. Perspectives on Psychological Science, 1(3), 181-210.
Schooler, J. W, Fiore, S., & Brandimonte, M. A. (1997). At a loss from words: Verbal overshadowing of
perceptual memories. Psychology of Learning and Motivation: Advances in Research and Theory,
37, 291–340. https://doi.org/10.1016/S0079-7421(08)60505-8
Schwartz, D. L. (1995). The emergence of abstract representations in dyad problem solving. Journal of
the Learning Sciences, 4(3), 321-354. https://doi.org/10.1207/s15327809jls0403_3
Sweller, J., & Cooper, G. A. (1985). The use of worked examples as a substitute for problem solving in
learning algebra. Cognition and Instruction, 2(1), 59–89.
Teasley, S. D. (1995). The role of talk in children's peer collaborations. Developmental Psychology, 31(2),
207– 1040. https://doi.org/10.1037/0012-1649.31.2.207
van der Meij, J., & de Jong, T. (2011). The effects of directive self-explanation prompts to support active
processing of multiple representations in a simulation-based learning environment. Journal of
Computer Assisted Learning, 27(5), 411-423. https://doi.org/10.1111/j.1365-
2729.2011.00411.x
Wylie, R. (2011). Examining the generality of self-explanation [Unpublished doctoral dissertation].
Carnegie Mellon University.
315
In Their Own Words Are Study Strategies Universal? A Call for More
Research With Diverse Non-WEIRD Students
to Apply the Science of Learning in Roberta Ekuni
Universidade Estadual do Norte do Paraná
Editors
Lauren L. Kordonowy
Sabine Pompeia
Victor A. Benassi
Universidade Federal de São Paulo
Pooja K. Agarwal
Berklee College of Music
Findings from a growing body of research on students’ preferred study strategies are clear: students
overwhelmingly report rereading textbooks, notes, and course materials as their primary approach to
studying (Dunlosky et al., 2013; Ekuni et al., 2020; Miyatsu et al., 2018). What remains unclear, however,
is whether these study strategy preferences are universal. Research on students’ study strategy
preferences is largely limited to WEIRD student populations (Western, Educated, Industrialized, Rich,
and Democratic; e.g., Agarwal et al., 2014, Hartwig & Dunlosky, 2012; Karpicke et al., 2009; Kornell &
Bjork, 2007), which represent only a small portion of humanity (Henrich et al., 2010). If we are to inform
students and teachers about effective strategies to improve learning, it is critical to take diversity into
account and determine whether students’ selection and implementation of study strategies is culturally,
economically, and/or demographically dependent.
First, we present an overview of what is known about students’ preferred study strategies, based on
research conducted with WEIRD student populations. Second, we discuss students’ perceptions about
the effectiveness of their preferred strategies compared to demonstrated benefits of less popular
strategies. Third, we review findings from the limited research currently available with non-WEIRD
student populations. Finally, we offer future directions and recommendations for conducting inclusive
research with diverse populations to better understand students’ study strategy preferences and
implementation.
Study Strategy Preferences in WEIRD Student Samples

In order to achieve good grades, students must make several decisions about which study strategies to
use (e.g., rereading textbook chapters or their own notes, highlighting parts of the material that they
think are important, testing themselves, rewatching video lectures, etc.), when they will implement
them (e.g., a little every day, the night before a test, etc.), and for how long (e.g., 30 minutes per study
session; Gettinger & Seibert, 2002). How and when students study have large effects on what
information they retain (Anthenien et al., 2018; Geller et al., 2017; Gurung, 2005).
In addition, how students study encompasses many factors, such as self-regulation of learning,
motivation to learn, and attitudes towards learning, all of which are associated with academic success
(Credé & Kuncel, 2008). Therefore, we will focus here on learning strategies discussed in the seminal
study by Dunlosky et al. (2013), which describes ten strategies (henceforth referred to as study
strategies) because they are clearly defined and their individual effectiveness has been established.
Additionally, these strategies can be easily taught and have minimal costs, so they can be implemented
widely to improve academic outcomes.
316
Study strategy preferences have been mostly investigated among students from the United States of
America (USA) using a variety of techniques: rank ordering a list of study strategies (Karpicke et al.,
2009), selecting strategies from a list that they use regularly (Hartwig & Dunlosky, 2012), and reporting
all study strategies that they use regularly without providing a list to choose from (Morehead et al.,
2015). As an added complication, “study strategies” are often ill-defined in survey research, which could
involve a wide variety of conditions, such as when, where, and with whom studying occurs, as well as
how long students study (Dunlosky et al., 2013). Although researchers have used a wide variety of
techniques to inquire about study strategy use among USA populations, rereading and highlighting tend
to be the most popular study strategies (Karpicke et al., 2009).
In one survey of WEIRD undergraduates from the USA (Karpicke et al., 2009), students ranked 11 study
strategies in order of preference, resulting in the following order: 1. rereading notes or textbook; 2.
doing practice problems; 3. using flashcards; 4. rewriting notes; 5. studying in groups; 6. memorizing; 7.
using mnemonics; 8. making outlines; 9. self-testing; 10. highlighting; and 11. thinking of real-life
examples. Using the same list of study strategies and a few others, Persky & Hudson (2016) similarly
found that students preferred studying by rereading, rewatching lecture videos, and completing practice
problems. Additional studies carried out in the USA show that students report using self-testing
strategies, followed by rereading, flashcard use, making outlines, and highlighting (Hartwig & Dunlosky,
2012; Morehead et al., 2015; Miyatsu et al., 2018). Recopying notes, asking questions or verbally
participating during class, and making diagrams, charts or pictures were reported as being used least
often (Morehead et al., 2015). Strategies of repeated exposure to content, such as rereading, were also
found to be the primary way of studying among Canadian high school and University students (Wood et
al., 1998).
An overview about study strategies conducted by Blasiman et al. (2017) included 13 studies (and one
meta-analysis), all of which involved students from the USA, with the exception of one study from
Canada. They found that most studies assessed, cross-sectionally, which strategies are used (±50% of
study strategies; discussed above), when (±30%; most students study just before exams), or for how
long (±40%; most students study less than they believe is necessary). These authors then conducted a
longitudinal analysis on USA undergraduates and found that students who were finishing a course
studied less often than expected with 10 techniques (mostly the same ones cited above) than they had
initially reported they intended to use. Students also studied for less time than intended, except for just
before exams (Blasiman et al., 2017).
The only study to assess ways of studying in a WEIRD population, but outside the USA and Canada, was
conducted in Italy (Poscia et al., 2015) and inquired about different aspects of ways of studying. The
researchers found, for instance, that students reported higher rates of studying in groups compared to
studying alone, and they preferred using books/papers to study compared to the internet (Poscia et al.,
2015).
Reasons for Use of Effective and Ineffective Study Strategies in WEIRD

Student Samples
Although rereading notes or texts is mostly the first or second most preferred study strategy among the
majority of students drawn from WEIRD USA samples, Dunlosky et al. (2013) rated it as a low utility
strategy. In other words, rereading was found to be less effective than other strategies that promote
long-term learning, such as retrieval practice (Dunlosky et al., 2013). Retrieval practice is one of the
most effective strategies to increase students’ long-term memory of academic material (Agarwal & Bain,
2019; Agarwal, et al., 2013, Yang et al., this volume). Students can use retrieval practice by studying with
flashcards, answering quizzes, testing themselves, explaining what they learned to other parties, or in
317
other similar ways (Agarwal et al., 2021). There is a consistent body of evidence that retrieval practice
promotes: 1) transfer (Pan & Rickard, 2018), that is, the capacity to apply knowledge in contexts that
differ from the one in which information was learned, and 2) long-term retention (for a recent review,
see Agarwal et al., 2021; McDermott, 2021), through various processes such as strengthening of
semantic memory representations and inhibiting irrelevant associations (van den Broek et al., 2016).
Furthermore, academic advantages are observed when retrieval practice is distributed over time, a
strategy known as spaced retrieval practice (Kang et al., 2014).
Yet, the majority of students in the USA claim to learn better when they use ineffective strategies such
as rereading (Kornell & Son, 2009; Son, 2004). For example, students predict that they would perform
better on an exam after rereading content than after being tested on the content, that is, after engaging
in retrieval practice activities (Agarwal et al., 2008; Blasiman et al., 2017; Karpicke & Roediger, 2008).
This finding may occur because the more students reread the material, the more fluent and familiar it
seems to them, which elicits a feeling that the content will be easy to retrieve at will. This is known as
the illusion of competence (Koriat & Bjork, 2005) because familiarity does not facilitate recall of
information. Consequently, students’ test performance turns out to be lower than they expect.
Another phenomenon that leads students to use rereading or similar low utility study strategies is the
misinterpreted-effort hypothesis (Kirk-Johnson et al., 2019): the false perception that the higher the
effort engaged in studying, such as is required for actively trying to recall information (retrieval
practice), the lower its effectiveness for learning (see also Blasiman et al., 2017; Palmer et al., 2019).
This perception leads students to prefer strategies that involve less effort, such as passively rereading
texts or notes or rewatching lectures, even though they generally result in lower levels of long-term
retention (Kirk-Johnson et al., 2019). Additionally, students often believe that their peers use ineffective
study strategies more often than they actually do, so some researchers suggest that this may influence
them to use counterproductive ways of studying (e.g., Anthenien et al., 2018). It seems that, overall,
WEIRD students prioritize studying with the least effort to maximize their short-term goals, which
include performing well on exams (see Kornell & Bjork, 2007).
Apart from the high effectiveness of retrieval practice, Dunlosky et al. (2013) identified distributing
study (spaced learning, rather than cramming; Cepeda et al., 2006; Smolen et al., 2016) as a high utility
and effective study strategy for a wide range of student ages. Controlled studies have shown that
content learned by cramming or massed studying is forgotten sooner compared to when students space
out their study over the course of days or longer even when the total study time is equivalent in both
conditions (Hartwig & Dunlosky, 2012; Kornell & Bjork, 2007; Morehead et al., 2015).
Despite such findings, data from USA undergraduates show that about 50% or more of students
reported that they prefer cramming instead of spacing their study sessions over time (Kornell et al.,
2010; Susser & McCabe, 2013; Persky & Hudson, 2016; Morehead et al., 2015). The same pattern was
found in a more diverse sample in the USA (non-students recruited by M-Turk; Yan et al., 2014).
However, cramming can lead to good grades when exams take place soon after studying (Cepeda et al.,
2006), which might encourage students to persist in this habit, even though people often forget the
information over the long term.
When students report using retrieval practice, they do so to check how well they know or have learned
pieces of information, not because this strategy promotes long-term learning (Morehead et al., 2015;
Kornell & Son, 2009). Thus, most people are not aware of the benefits of spacing study sessions and of
using retrieval as a learning strategy, but rather use it as a formative assessment strategy to check for
understanding. Many teachers suggest students use retrieval practice to assess their learning
(Morehead et al., 2015).
318
How do students form study strategy preferences in the first place? Most students report that their
choice of how to study is based on their own experience (Karpicke et al., 2009; Kornell & Bjork, 2007).
Studies in the USA find that only a small set of students report being explicitly taught about study
strategies (Hartwig & Dunlosky, 2012; Morehead et al., 2015), in contrast to the majority of teachers
who profess they provide study tips to their students (Morehead et al., 2015).
Study strategy preferences may also change according to students’ academic experience. A longitudinal
study across four years with USA undergraduate students (Persky & Hudson, 2016) found that as they
advanced in their courses, they tended to use less effective study strategies than first-year students.
Although this seems to indicate that they progressively choose the less effective ways of studying, it
cannot be excluded that they may have learned to change the way they use these strategies, doing so in
a more efficient manner (see Dunlosky et al., 2013; Miyatsu et al., 2018). For example, underlining is
regarded as a relatively ineffective technique, but more experienced students may learn how to select
passages in texts that are more important and underline them, which might be helpful in directing their
attention to these passages when they study for exams.
Apart from this, data on WEIRD educators suggest they are not prone to teaching the best study
strategies. According to Morehead et al. (2015), when teachers do suggest high utility ways of studying
that involve retrieval practice, most do so not because they think students will learn better, but because
they think that this will allow students to find gaps in their knowledge that require further attention,
mirroring the reason students use this type of strategy, as mentioned above. Instead, the focus of
education seems to be on providing academic content and not on preparing students to study in ways
that will lead them to retain information for longer periods (Dunlosky, 2013).
Based on research conducted primarily in the USA using a WEIRD sample, students often study by
rereading, and many do not use high-utility strategies such as distributed/spaced study and retrieval
practice. In addition, students’ beliefs about how learning works (e.g., less effort equals more learning)
differ from well-established findings that challenging strategies improve learning (e.g., more effort
equals more learning; Agarwal & Bain, 2019). However, it is unclear whether students’ preferences
regarding study strategies, their use of ineffective strategies, and their beliefs about effective strategies
are specific to WEIRD samples or whether they are universal phenomena. Next, we review findings from
the limited research currently available on study strategy preferences with non-WEIRD student samples.
Study Strategy Preferences in Non-WEIRD Student Samples

Our exploration of the research literature on study strategies use identified only a small number of
relevant studies conducted outside the USA, specifically Italy (Poscia et al., 2015), Brazil (Ekuni et al.,
2020), India (Chamundeswari et al., 2014; Chand, 2013), and Nigeria (Ebele & Olofu, 2017; Fakeye &
Amao, 2013). The single study conducted in Brazil was the only one which investigated the same study
strategies inquired about in WEIRD samples reviewed above. The other studies did not investigate study
strategies per se, having instead described aspects of “study habits” more generally (e.g., study
environment, level of concentration).
In research carried out in Brazil, a non-WEIRD developing nation, Ekuni et al. (2020) inquired about the
frequency of use among pre-college students of the same study strategies listed by Karpicke et al.
(2009). They found that the pattern of the most to least frequently used strategies was similar to that
found in the studies conducted in the USA: rereading content came first, closely followed by doing
practice exercises, highlighting texts, and summarizing. Next, came thinking about real life examples,
self-testing/practicing recall, rewriting content, memorizing, and using mnemonics. Studying in groups
came last (Ekuni et al., 2020), which was the only strategy preference that contrasted with results from
most USA-based studies (Karpicke et al., 2009; Morehead et al., 2015) and from Italy (Poscia et al.,
319
2015), which reported higher rates of studying in groups. Brazil ranks very low on collaborative problem
solving in the Program for International Student Assessment (PISA), a factor that varies significantly
among nations (OECD, 2019), pointing to a possible cultural reason for differences in the use of this
particular strategy, which must therefore be explored in contexts other than WEIRD populations.
There are some additional examples of studies in non-WEIRD samples that focused on study habits more
generally. Unfortunately, these studies conducted with non-WEIRD student samples did not report
results regarding students’ preferences for specific study strategies. In two studies from Nigeria (Ebele &
Olofu, 2017; Fakeye & Amao, 2013) and two studies from India (Chamundeswari et al., 2014; Chand,
2013), researchers found that some study habits (e.g., higher levels of self-reported engagement or
concentration) were positively associated with academic achievement.
Based on this very limited research with student populations outside the USA, it is not possible to draw
conclusions about whether study strategy preferences are universal. Studies are needed which include
standardized ways of inquiring about study strategies use. Surveys regarding students’ and teachers’
beliefs about study strategies were also lacking in non-WEIRD samples. Likewise, information about
factors that might affect preference and use of study strategies (e.g., socioeconomic status, ethnicity,
sex, gender, age, grade) were rare or non-existent in both WEIRD and non-WEIRD samples.
Potential Moderator Variables in Study Strategy Preferences

Students’ study strategy preferences may differ between WEIRD and non-WEIRD countries due to
several factors, such as being from a minority and minoritized group, and/or being from different
socioeconomic backgrounds. Ethnicity, culture, and socioeconomic status vary widely around the world,
not only between developed and developing nations, but also within nations (for social inequality
information in the USA, see Saez & Zucman, 2016; Gibson-Davis & Hill, 2021). Until additional research is
conducted in non-WEIRD countries and results from both WEIRD and non-WEIRD student samples are
stratified by demographic and cultural factors, we cannot state with certainty that study strategy
preferences are universal.
For example, the effects of being from underrepresented or minoritized groups are typically not
addressed in studies with WEIRD samples (see Buchanan et al., 2021), even though there are studies
that show that some minority students from the USA can have lower academic success compared to
their White and Asian peers (Rodriguez et al., 2018). The effects of socioeconomic status are also largely
unexplored in WEIRD samples, possibly due to the misconception that high income disparities are
present only in developing countries, even though the unequal distribution of wealth in the USA today is
extremely high, especially in households with children, which has wide implications, including academic
achievement (Gibson-Davis & Hill, 2021).
Critically, minority status and lower socioeconomic status have been linked with lower academic
outcomes (Farah, 2017; OECD, 2019; Thompson, 2018; Sirin, 2005), lower executive functioning, and
lower working memory capacity (Farah, 2017; Lawson, Hook, & Farah, 2018; Leonard, Mackey, Finn, &
Gabrieli, 2015), which may affect–or be affected by–students’ study strategy preferences and use. For
example, researchers have found that the lower the socioeconomic status, the lower the students’ self-
perception of how well they did academically (Poscia et al., 2015). At the same time, Brazilian students’
study strategy preferences did not differ based on socioeconomic background (Ekuni et al., 2020),
whereas Indian students with higher socioeconomic status demonstrated more frequent use of effective
study habits (Chamundeswari et al., 2014).
Until additional research on students’ study strategy preferences is conducted in non-WEIRD countries
and also cross-culturally, this “chicken or the egg” situation will remain problematic. To speculate,
320
minority status and lower socioeconomic status may result in lack of access to information about
effective study strategies, lack of motivation while studying, and differential benefits from study
strategies. Alternatively, study strategy preferences of students from marginalized backgrounds may, in
turn, lead to lower academic achievement.
Furthermore, it is premature to conclude that study strategy preferences are universal until sex and
gender are taken into account (for the difference between sex and gender, see Johnson et al., 2007).
Consider that women are known to consistently outperform men academically (O’Dea et al., 2018;
OECD, 2019; Voyer & Voyer, 2014), and research suggests that younger girls and boys may study
differently (Agarwal et al., 2014), which might be partly explained by the use of different strategies per
sex/gender. However, in some studies, the number of men and women in samples is not even quantified
(e.g., Karpicke et al., 2009), nor data on other inclusive or diversity-related information (see Buchanan et
al., 2021). Other researchers provide data on the number men and women, but do not report results
pertaining to each gender/sex (e.g., Anthenien et al., 2018).
Data from one USA study suggest that both middle and high-school girls and boys may indeed study
differently. When asked what strategies they use to study outside of class, the most frequent option was
reviewing materials (girls - 50%, boys - 39%), followed by repeating facts (girls - 47%, boys - 37%), and
being tested by someone else (girls - 39%, boys - 31%) (Agarwal, et al., 2014). However, there was no
statistical comparison between answers by gender in this study, nor an analysis determining whether
different ways of studying influenced students’ grades depending on their gender.
In terms of sex/gender differences in non-WEIRD populations, the study with the Brazilian sample
described above found that young women reported higher frequency of use of ineffective strategies
(highlighting, summarizing class material, and use of mnemonics) than their male peers, but effect sizes
were small (Ekuni et al., 2020). Chamundeswari et al. (2014) found that girls from India reported better
study habits than boys. However, no details of the study habits scale were made available for readers. In
addition, compared to their male peers, female students from Italy preferred studying alone compared
to studying in groups (Poscia et al., 2015), so these results are not very informative with respect to sex
differences regarding study strategies.
Future Directions and Recommendations for Inclusive Research

Although some researchers recognize the importance of taking diversity into account, not only among
samples but also among researchers, in terms of minority status, socioeconomic status, sex/gender, etc.,
this is usually considered only within WEIRD countries (Buchanan et al., 2021). Diversity, in terms of
representation of different populations and research carried out in non-WEIRD nations, must also be
addressed in the literature on student study strategy preferences. Most research on study strategies has
been restricted to WEIRD student populations, particularly from the USA. The literature is in need of
exploring cross-cultural similarities and differences, replicating studies in different samples (Klein et al.,
2018), and determining the factors that may influence study strategy use, such as sex/gender,
socioeconomic status, minority and minoritized status, and age.
To this end, some issues must be considered in future research. There are many differences between
studies that investigate study strategies, even in WEIRD samples. For instance, researchers have asked
students to rank order a list of study strategies (Karpicke et al., 2009), select strategies from a list that
they use regularly (Hartwig & Dunlosky, 2012), report all study strategies that they use regularly without
providing a list from which to choose (Morehead et al., 2015), and indicate the frequency of study
strategies they use from a list (Ekuni et al., 2020). Furthermore, study strategies (individually and as a
whole) are often ill-defined in the existing studies. They involve techniques, habits, when, where, and
with whom to study, study session length, people’s attitudes, motivations, and many other factors. For
321
this field to advance, it is paramount that a framework be proposed as to how these factors are
associated with each other and with academic outcomes, with accompanying standardized and
validated scales that can be adapted for use in different populations.
Another issue that needs addressing is that the literature has practically ignored how students use
different types of study strategies. For example, Dunlosky et al. (2013) posited that, although
highlighting is not an effective study strategy, it can be highly effective if students are trained to identify
the most important aspects of a text (see also Miyatsu et al., 2018). Indeed, probing for more detail
about how students study can alter results (Wood et al., 1998), suggesting that framing questions in
more specific ways can be advantageous in terms of obtaining more accurate information on how and
why people study the way they do. Thus, researchers should investigate not only preferred study
strategies, but also inquire in detail about how they are used, for how long, if in a spaced manner, etc. It
will be important to relate differences in responses to such measures with student academic
performance.
In light of the findings discussed here, it is still not possible to determine if study strategies preferences
are universal because this has practically only been studied in USA-based samples. Yet, the high
prevalence of ineffective strategies (e.g., rereading) in USA samples was also reported in Brazil (Ekuni et
al., 2020), so such findings may well be generalizable to other populations. Additional factors that could
alter preferences for study strategies, such as minority status, socioeconomic status, and sex/gender,
must also be further investigated.
Maximizing the use of evidence-based effective learning strategies, such as retrieval practice and
distributed practice (i.e., spacing learning sessions), requires debunking myths and beliefs about
learning. Cognitive scientists must investigate this issue in a more detailed and inclusive manner cross-
culturally and better communicate their findings, especially to educators (Vaughn & Kornell, 2019; Rowe
& Hattie, this volume) regarding how best to study to promote lasting learning. Initiatives like the
Cognition Toolbox can be helpful in this respect; they not only promote a dialogue between researchers
and educators, but they also allow them to work together to develop research in classrooms and to test
and analyze their results based on evidence-based interventions to promote lasting learning (Benassi et
al., this volume; Benassi et al., 2014). Approaches must also be developed to increase students’
willingness to engage in the strategies that required higher effort to study more effectively (see the
chapters in Part 4 of this volume).
This is no easy endeavor. Teachers and students may persist in recommending and using relatively
ineffective strategies, because it is hard to ascertain that some ways of studying do not work well in
each persons’ particular experience (Fiorella, 2020). This type of evidence can only be obtained in
controlled experiments, including participants who are randomly assigned to different study strategy
conditions using the same measure or assessment of learning. Diversifying the characteristics of the
students (different cultures, backgrounds, etc.) is also essential to further our understanding about how
factors that relate to academic success might also relate to students´ study strategy preferences and
implementation.
Notwithstanding the difficulties in reaching these goals, there are some positive examples of
interventions in the USA that promote effective study strategies. For instance, students who attend
activities (e.g., workshops, meetings with faculty) on how best to study subsequently report less
frequent use of rereading compared with students who do not take part in these activities (Persky &
Hudson, 2016). In another study, students were provided with a reading assignment about learning
strategies (distributed practice, rereading, retrieval practice, or thinking about mental images) to make
them aware of their effects, which decreased the use of low-utility strategies compared to peers who
did not complete the assignment (Brown-Kramer, 2021; Brown-Kramer, this volume). Therefore, simple,
322
inexpensive interventions can be developed to help students avoid using low-utility study strategies, and
this can and should be implemented in non-WEIRD samples as well to determine if they improve
learning. Initiatives, like providing tips for optimizing the use of study strategies, are also important,
such as those proposed by Miyatsu et al. (2018). Additionally, these strategies can be easily taught, so
they can be implemented widely by a diversity of students to improve academic outcomes.
Take-Home Message
It is unclear if students from WEIRD and non-WEIRD countries choose similar or different study
strategies. Several USA-based studies demonstrate that students tend to use ineffective study
strategies, and only one similar investigation was conducted in the non-WEIRD country of Brazil (Ekuni
et al., 2020). At present, there is no clear definition in the literature on what constitutes study
strategies. Also, there is little evidence regarding the role of factors that differ among WEIRD and non-
WEIRD students—such as ethnicity, culture, and socioeconomic status—that could influence the way
students study and academic consequences. Researchers should standardize the way they inquire about
study strategies using a consistent theoretical framework and scales or instruments that are adaptable
to many cultures, taking diversity into account. Equipped with more study strategy research that
includes diverse non-WEIRD students, researchers can more appropriately develop effective studying
interventions and improve academic outcomes worldwide.
Author Note
Acknowledgments: Conselho Nacional de Desenvolvimento Científico e Tecnológico – CNPq (fellowship
awarded to SP), Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - CAPES (finance code
001 awarded to SP), and Associação Fundo de Incentivo à Pesquisa (AFIP; awarded to SP).
Email of corresponding author: [email protected]
References
Agarwal, P. K., & Bain, P. M. (2019). Powerful teaching: Unleash the science of learning. Jossey-Bass.
Agarwal, P. K., D’Antonio, L., Roediger, H. L., McDermott, K. B., & McDaniel, M. A. (2014). Classroom-
Agarwal, P.K., Nunes, L.D. & Blunt, J.R. (2021). Retrieval practice consistently benefits student learning:
A systematic review of applied research in schools and classrooms. Educational Psychology
Review, 33, 1409–1453. https://doi.org/10.1007/s10648-021-09595-9
Agarwal, P. K., Roediger, H. L., McDaniel, M. A., & McDermott, K. B. (2013). How to use retrieval practice
to improve learning. Retrieval Practice.
http://pdf.retrievalpractice.org/RetrievalPracticeGuide.pdf
Anthenien, A. M., DeLozier, S. J., Neighbors, C., & Rhodes, M. G. (2018). College student normative
misperceptions of peer study habit use. Social Psychology of Education, 21(2), 303–322.
Benassi, V. A., Tappin, E. M., Overson, C. E., Lee, M. J., O´Brien, E. J., & White, B. P. (2014). Applying the
science of learning: the cognition toolbox. In V. A. Benassi, C. E. Overson, & C. M. Hakala (Eds.),
323
Applying science of learning in education: Infusing psychological science into the curriculum.
Society for the Teaching of Psychology (pp. 194 205).
http://teachpsych.org/ebooks/asle2014/index.phpf
comparing intended versus actual study behavior. Memory, 25(6), 784–792.
how to learn” assignment. Teaching of Psychology, 48(1), 48–54.
Buchanan, N. T., Perez, M., Prinstein, M. J., & Thurston, I. B. (2021). Upending racism in psychological
science: Strategies to change how science is conducted, reported, reviewed, and
disseminated. American Psychologist, 76(7), 1097–1112. https://doi-
org.unh.idm.oclc.org/10.1037/amp0000905.supp (Supplemental)
tasks: A review and quantitative synthesis. Psychological Bulletin, 132(3), 354–380.
Chamundeswari, S., Sridevi, V., & Kumari, A. (2014). Self-concept, study habit and academic
achievement of students. International Journal of Humanities Social Sciences and Education,
1(10), 47–55. https://www.arcjournals.org/pdfs/ijhsse/v1-i10/8.pdf
Chand, S. (2013). Study habits of secondary school students in relation to type of school and type of
family. International Journal of Social Science & Interdisciplinary Research, 2(7), 90–96.
Credé, M., & Kuncel, N. R. (2008). Study habits, skills, and attitudes: the third pillar supporting collegiate
academic performance. Perspectives on Psychological Science, 3(6), 425–453.
Dunlosky, J. (2013). Strengthening the student toolbox. American Educator, 37(3), 12–21.
https://files.eric.ed.gov/fulltext/EJ1021069.pdf
Ebele, U. F., & Olofu, P. A. (2017). Study habit and its impact on secondary school students' academic
performance in biology in the Federal Capital Territory, Abuja. Educational Research and
Reviews, 12(10), 583–588. https://doi.org/10.5897/err2016.3117
Ekuni, R., Souza, B. M. N., Agarwal, P. K., & Pompeia, S. (2020). A conceptual replication of survey
research on study strategies in a diverse, non-WEIRD student population. Scholarship of
Fakeye, D. O., & Amao, T. A. (2013). Classroom participation and study habit as predictors of
achievement in literature-in-English. Cross-Cultural Communication, 9(3), 18–25.
https://doi.org/10.3968/j.ccc.1923670020130903.2741
Farah, M. J. (2017). The neuroscience of socioeconomic status: correlates, causes, and consequences.
Neuron, 96(1), 56–71. https://doi.org/10.1016/j.neuron.2017.08.034
324
Fiorella, L. (2020). The science of habit and its implications for student learning and well-being.
Educational Psychology Review, 32(3), 603–625. https://doi.org/10.1007/s10648-020-09525-1
Geller, J., Toftness, A. R., Armstrong, P. I., Carpenter, S. K., Manz, C. L., Coffman, C. R., & Lamm, M. H.
(2017). Study strategies and beliefs about learning as a function of academic achievement and
achievement goals. Memory, 26(5), 683-690. https://doi.org/10.1080/09658211.2017.1397175
Gettinger, M., & Seibert, J. K. (2002). Contributions of study skills to academic competence. School
Psychology Review, 31(3), 350–365. https://doi.org/10.1080/02796015.2002.12086160
Gibson-Davis, C., & Hill, H. D. (2021). Childhood wealth inequality in the United States: Implications for
social stratification and well-being. RSF: The Russell Sage Foundation Journal of the Social
Sciences, 7(3), 1-26. https://doi.org/10.7758/RSF.2021.7.3.01
Gurung, R. A. R. (2005). How do students really study (and does it matter)? Teaching of Psychology,
32(4), 239–241.
scheduling related to achievement? Psychonomic Bulletin & Review, 19(1), 126–134.
Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world? Behavioral and
Brain Sciences, 33(2), 1–23. https://doi.org/10.1017/S0140525X0999152X
Johnson, J. L., Greaves, L., & Repta, R. (2007). Better science with sex and gender: A primer for health
research. Vancouver: Women's Health Research Network, 8(14). https://doi.org/10.1186/1475-
9276-8-14
Kang, S. H. K., Lindsey, R. V, Mozer, M. C., & Pashler, H. (2014). Retrieval practice over the long term:
Should spacing be expanding or equal-interval? Psychonomic Bulletin & Review, 21(6), 1544–
Karpicke, J. D., Butler, A. C., & Roediger, H. L. (2009). Metacognitive strategies in student learning: Do
students practice retrieval when they study on their own? Memory, 17(4), 471–479.
Karpicke, J. D., & Roediger, H. L. (2008). The critical importance of retrieval for learning. Science,
319(5865), 966–968. https://doi.org/10.1126/science.1152408
study strategy choice. Cognitive psychology, 115(article 101237), 101-237.
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., . . . Nosek, B. A. (2018).
Many Labs 2: Investigating Variation in Replicability Across Samples and Settings. Advances in
Methods and Practices in Psychological Science, 1(4), 443–490.
Koriat, A., & Bjork, R. A. (2005). Illusions of competence in monitoring one’s knowledge during study.
Journal of Experimental Psychology: Learning, Memory, and Cognition, 31(2), 187–194.
Kornell, N., & Bjork, R. A. (2007). The Promise and Perils of self-regulated study. Psychonomic Bulletin &
Review, 14(2), 219–224. https://doi.org/10.3758/BF03194055
325
Kornell, N., Castel, A. D., Eich, T. S., & Bjork, R. A. (2010). Spacing as the friend of both memory and
induction in younger and older adults. Psychology and Aging, 25(2), 498–503.
Kornell, N., & Son, L. K. (2009). Learners’ choices and beliefs about self-testing. Memory, 17(5), 493–501.
Lawson, G. M., Hook, C. J., & Farah, M. J. (2018). A meta-analysis of the relationship between
socioeconomic status and executive function performance among children. Developmental
Science, 21(2), 1–22. https://doi.org/10.1111/desc.12529
Leonard, J. A., Mackey, A. P., Finn, A. S., & Gabrieli, J. D. (2015). Differential effects of socioeconomic
status on working and procedural memory systems. Frontiers in human neuroscience, 9(554).
https://doi.org/10.3389/fnhum.2015.00554
McDermott, K. B. (2021). Practicing retrieval facilitates learning. Annual Review of Psychology, 72, 609–
633. https://doi.org/10.1146/annurev-psych-010419-051019
Miyatsu, T., Nguyen, K., & McDaniel, M. A. (2018). Five popular study strategies: their pitfalls and
optimal implementations. Perspectives on Psychological Science, 13(3), 390–407.
O’Dea, R. E., Lagisz, M., Jennions, M. D., & Nakagawa, S. (2018). Gender differences in individual
variation in academic grades fail to fit expected patterns for STEM. Nature Communications,
9(1), 1-8. https://doi.org/10.1038/s41467-018-06292-0
OECD. (2019). PISA 2018 Results (Volume II): Where all students can succeed. In OECD Publishing.
https://www.oecd.org/pisa/publications/PISA2018_CN_IDN.pdf
Palmer, S., Chu, Y., & Persky, A. M. (2019). Comparison of rewatching class recordings versus retrieval
practice as post-lecture learning strategies. American Journal of Pharmaceutical Education,
83(9), 1958–1965. https://doi.org/10.5688/ajpe7217
synthesis. Psychological Bulletin, 144(7), 710–756. https://doi.org/10.1037/bul0000151
Persky, A. M., & Hudson, S. L. (2016). A snapshot of student study strategies across a professional
pharmacy curriculum: Are students using evidence-based practice? Currents in Pharmacy
Teaching and Learning, 8(2), 141–147. https://doi.org/10.1016/j.cptl.2015.12.010
Poscia, A., Frisicale, E. M., Parente, P., Milia, D. I. La, Waure, C. de, & Pietro, M. L. Di. (2015). Study habits
and technology use in Italian university students. Annali Dell’Istituto Superiore Di Sanità, 51(2),
126–130. http://dx.doi.org/10.4415/ann_15_02_10
Rodriguez, F., Rivas, M. J., Matsumura, L. H., Warschauer, M., & Sato, B. K. (2018). How do students
study in STEM courses? Findings from a light-touch intervention and its relevance for
underrepresented students. PLoS ONE, 13(7), 1–20.
https://doi.org/10.1371/journal.pone.0200767
Saez, E., & Zucman, G. (2016). Wealth inequality in the United States since 1913: Evidence from
capitalized income tax data. The Quarterly Journal of Economics, 131(2), 519–578.
https://doi.org/10.1093/qje/qjw004
326
Sirin, S. R. (2005). Socioeconomic status and school outcomes: A Meta-Analytic review of research.
Review of Educational Research, 75(3), 417–453. https://doi.org/10.3102/00346543075003417
Smolen, P., Zhang, Y., & Byrne, J. H. (2016). The right time to learn: mechanisms and optimization of
spaced learning. Nature Reviews Neuroscience, 17(2), 77–88.
https://doi.org/10.1038/nrn.2015.18
Son, L. K. (2004). Spacing one’s study: evidence for a metacognitive control strategy. Journal of
Experimental Psychology. Learning, Memory, and Cognition, 30(3), 601–604.
Susser, J. A., & McCabe, J. (2013). From the lab to the dorm room: Metacognitive awareness and use of
spaced study. Instructional Science, 41(2), 345–363. https://doi.org/10.1007/s11251-012-9231-8
Thompson, S. (2018) Achievement at school and socioeconomic background—an educational
perspective. npj Science of Learning, 3(5). https://doi.org/10.1038/s41539-018-0022-0
van den Broek, G.,Takashima, A., Wiklund-Hörnqvist, C., Karlsson Wirebring, L., Segers, E., Verhoeven, L.,
& Nyberg, L. (2016). Neurocognitive mechanisms of the “testing effect”: A review. Trends in
Neuroscience and Education, 5(2), 52–66. https://doi.org/10.1016/j.tine.2016.05.001
Vaughn, K. E., & Kornell, N. (2019). How to activate students’ natural desire to test themselves. Cognitive
Research: Principles and Implications, 4(35), 1-16. https://doi.org/10.1186/s41235-019-0187-y
Voyer, D., & Voyer, S. D. (2014). Gender differences in scholastic achievement: A meta-analysis.
Psychological Bulletin, 140(4), 1174–1204. https://doi.org/10.1037/a0036620
Wood, E., Motz, M., & Willoughby, T. (1998). Examining students’ retrospective memories of strategy
development. Journal of Educational Psychology, 90(4), 698–704. https://doi.org/10.1037/0022-
0663.90.4.698
Yan, V. X., Thai, K. P., & Bjork, R. A. (2014). Habits and beliefs that guide self-regulated learning: Do they
vary with mindset. Journal of Applied Research in Memory and Cognition, 3(3), 140–152.
327
In Their Own Words The Learning Lab: An Experiment in Student-
Centered Teaching and Learning
to Apply the Science of Learning in Marlon Kuzmick, Lauren Davidson, Jordan Koffman, and
Tamara J. Brenner
Editors
Harvard University
Lauren L. Kordonowy
Victor A. Benassi
Introduction to the Learning Lab

As an administrative unit devoted to faculty development, graduate student training, and instructional
design, the relationship between the Derek Bok Center for Teaching and Learning and the science of
learning is in some sense a subordinate one. That is, our mission lies not so much in producing new
scientific discoveries, but rather in supporting our faculty and students as they bring their teaching and
learning into alignment with research-based ideas and best practices.
As a result, a major feature of our work involves disseminating these ideas and best practices through
workshops, online resources, and one-on-one consultations with faculty. But frequently, especially in
instances where faculty seek to transform their teaching quite radically, the dissemination of ideas and
best practices is not enough to achieve the faculty member’s goals. Even if one knows “how people
learn,” or that students excel in equitable, inclusive classrooms, this knowledge alone is insufficient. One
still needs to overcome the many challenges that arise when attempting to put this knowledge into
practice—challenges related to, for example, design, engineering and implementation, space, and
staffing.
The entire Bok Center is devoted to helping faculty overcome these challenges. In this chapter we report
on one particular initiative we undertook to achieve this goal: our “Learning Lab,” which we launched in
2015 to support faculty as they develop innovative new assignments, activities, and materials. For
reasons we will outline below, a particularly significant focus in the first years of the Learning Lab has
been multimodal academic communication, as faculty and students alike have been drawn increasingly
towards media for developing and displaying their work that go beyond academic writing alone.
This Learning Lab is both a space and a group of people. The space is a mixed-use media and design
studio: part classroom and part makerspace, part computer lab and part film-studio. The group of
people is an intergenerational team composed of Bok Center staff, faculty clients, graduate student
design fellows, and, crucially, undergraduate student collaborators who are simultaneously user testers
and co-designers.
In what follows, we tell the story of how and why we set up the Learning Lab, and we outline its
structure and processes, calling specific attention to the ways our work aligns with the science of
learning. We are not presenting everything we have done as novel or ground-breaking; in many cases
we have been deeply influenced by partners on campus and at other institutions. The goal of this
chapter, rather, is to reflect on the role that teaching and learning centers play in “putting knowledge
into practice,” to share some of the ways we have tried to fulfil this role, and to report on some of the
emergent phenomena we encountered along the way.
328
How It All Started
To begin our story, we need to take you back in time to before the Learning Lab existed. Between 2012
and 2014, there was a great deal of excitement surrounding the Harvard/MIT co-creation, EdX (and its
component parts, HarvardX and MITx). On Harvard’s campus, the goal of bringing Harvard courses to
online students around the world through Massively Open Online Courses (MOOCs) energized our
faculty and spurred countless innovations. The Bok Center team was well positioned to help. Bok Center
staff possessed not only expertise in helping faculty develop their teaching, but also, almost accidentally,
the technical skills required to capture teaching on video, as we had been recording teachers teaching
since our inception (originally as the Danforth Video Laboratory) in 1974.
It was, perhaps, another accident of history that in 2013, the inaugural Faculty Director of the Bok
Center, Robert Lue, was simultaneously serving as the Faculty Director of HarvardX, and in this dual
capacity he was invested in cultivating potential synergies between the organizations. HarvardX had
become an incubator for innovative online teaching and learning, and Lue encountered frequent faculty
requests for a similar incubator to support brick and mortar teaching innovations.
It was with all this in mind that the Bok Center’s Learning Lab was launched in 2015 to support Harvard
faculty as they explored new activities, assignments, and media in the on-campus courses they taught.
The Learning Lab was designed to provide the same sort of media support that faculty might receive
through HarvardX, but also to use its expertise, technology, and space to support Harvard’s
undergraduates. Hence, the space was designed not as a video studio optimized for a single performing
professor, but rather as a mash-up of a video studio and a classroom. (Sometimes we refer to the
Learning Lab as a makerspace wrapped in a video studio.)
In the early days of MOOCs, one of the principal uses of technology was to amplify the voice of the
professor and share content with a broad audience. As we developed the Learning Lab, we sought to
reverse that paradigm—to create a space where we could leverage the power of media and technology
to facilitate student communication and project-based learning, with the goal of helping students think,
learn, and express their own ideas. In effect, we aimed to help faculty shift from instructor-centered
practices to learner-centered teaching (Weimer, 2013).
Furthermore, the Learning Lab aimed to encourage faculty to approach their teaching with the same
spirit of experimentation and creativity with which they approach their research. At the same time, its
creation was grounded in a commitment to evidence-based practices. Increasing research had
demonstrated the importance of active learning (Freeman et al., 2014; cf. Newcombe, this volume), and
the fields of discipline-based educational research (DBER), cognitive science, educational psychology,
and the scholarship of teaching and learning also provided myriad insights for higher education (e.g.,
Ambrose, 2010; Bransford et al., 2000; Handelsman, 2004; Slater et al., 2010).
The Case of MCB81: A Support Stack for Multimodal Composition

The initial projects of the Learning Lab frequently lay at the intersection between HarvardX and the
brick-and-mortar classroom.
David Cox, the faculty member behind the innovative HarvardX course “Fundamentals of Neuroscience,”
requested our support for the analogous on-campus course, MCB81. Cox sought help for a new
assignment that was based on a practice he had encountered in graduate school, in which PhD students
presented a summary of a scientific article from memory, hand-drawing the figures in real time. Cox
spoke of how this ability to draw the figures (and the practice required to do so) aligned with the sort of
deep understanding he wanted his students to achieve.
329
Just a few years prior, Harvard had hosted Felice Frankel from MIT for a “Drawing to Learn” event. Cox’s
assignment aligned well with the ideas of Frankel and others, who argue that drawing helps students
develop a deeper understanding of science, organize their knowledge, develop reasoning skills, and
learn how to represent ideas (Ainsworth et al., 2011; Frankel, 2005; Quillin & Thomas, 2015).
Researchers have found that asking students to draw a diagram leads to deeper understanding than
producing a written summary (Gobert & Clement, 1999). Furthermore, drawing has been shown to
increase students’ motivation and engagement (Ainsworth et al., 2011).
Cox also saw a connection between the drawing-centric talks for which he was aiming and the content
he had been producing for his HarvardX course. So deep was the connection that he felt the best of the
students’ assignments could be incorporated into optional modules for the online course. Thus, the
Learning Lab developed a strategy to record videos of students explaining concepts and figures from
scientific papers, using hand-drawn illustrations. In a sense, the Learning Lab was taking the production
plan used to support faculty as they made MOOCs and offering it at a considerably greater scale to each
of the students in Cox’s class.
The Learning Lab offered students not just technical video production support, but perhaps more
importantly, support for the process of developing their presentations. The Bok Center’s staff frequently
included former instructors from the Harvard College Writing Program, all of whom had been heavily
influenced by research on the importance of the many cornerstones of college composition courses: the
draft-revision cycle, low-stakes assignments, scaffolding, and so on.
While our institution possessed a robust and well-integrated support system for students as they
learned academic writing, no equivalent system existed for the drawing-based, visual and oral
communication assignment Cox had in mind. There was no equivalent of the introductory writing
course, writing-across-the-curriculum program, Writing Center, and peer writing tutors. But we were
able to look to this existing system for a well-tested, evidence-based model of the support system we
would need to design for MCB81. With this in mind, the Learning Lab created a similarly structured stack
of support for this course as a prototype. What the Learning Lab attempted to do for Cox’s students was
to provide a combination of the pre-production–production–post-production cycle it offered Cox’s
online course and the low-stakes brainstorming, draft, feedback, revision cycle that composition courses
offer beginning writers.
Students in the course visited the Learning Lab studio multiple times to practice articulating ideas on
camera and drawing concepts using simple materials, and to receive feedback as they iteratively revised
their ideas. By providing appropriate scaffolding for this drawing and speaking activity, the Learning Lab
helped students approach what could be an unfamiliar project with a positive attitude and the
confidence that they could learn and apply new skills for explaining scientific concepts (Quillin &
Thomas, 2015).
Why It Works
The instructors we support, like many instructors, do not ground every single one of their classroom
experiments in the research on teaching and learning from the very start. Indeed, experienced
instructors often have tacit knowledge of their disciplines and their classrooms that gives them strong
intuitions that can be quite productive.
As a faculty support team, we saw our role as providing the course leads with highlights from the
science of learning as they co-developed their ideas with us. While not every project was grounded in
the research from the start, the projects proceeded in an “iterative dialogue” with the research. This
meant, ultimately, that we achieved alignment between instructors’ innovative work and the research,
330
but without ever too quickly “trumping” their tacit knowledge with jargon, facts, and figures. In the end,
our projects “work” because they are in alignment with what we know about project-based learning,
multimedia learning and communication, student engagement, and constructivist pedagogies, to name
only a few.
When students draw key illustrations while explaining a concept on camera, they must identify what
information is important to convey, build connections between new and existing knowledge, and
organize this knowledge into a coherent story. The MCB81 student videos are just one example of a
generative learning activity, through which learners integrate new knowledge with existing knowledge
and apply what they are learning to new situations (Fiorella & Mayer, 2016). Moreover, students learn
through the process of articulating ideas, which are often still unformed (Sawyer, 2014); the Learning
Lab responded to the benefits of generative learning by designing a set of experiences for MCB81 that
offered students repeated opportunities to practice articulating their ideas over the semester, prior to
completion of the final project.
In the years since we first worked with MCB81, we have supported scores of courses in developing
assignments in which students explain their ideas visually, graphically, and orally. These generative
assignments have included infographics, podcasts, virtual 3D galleries, animations, timelines, and more.
Although the Learning Lab often introduces students to activities that may be unfamiliar, which could
perhaps be intimidating, we are intentional about increasing students’ self-efficacy, or belief in their
ability to accomplish a goal (Bandura, 1997; Schwartz et al., 2016). We guide students through the
process of using new media or developing new literacies through scaffolded assignments that build skills
for a larger project. We also provide workshops, online resources, and hackathons where staff and
student fellows can provide individual support. Additionally, students have the opportunity at
workshops to observe their peers perform and model their thinking in front of others, providing
examples of what success can look like.
By any measure, students at the Learning Lab are actively engaged in doing things—from designing a
game, to creating an infographic, to building a virtual gallery. As such, the Learning Lab is based on the
fundamental principle of constructivism—that people learn through actively constructing knowledge
(Bransford et al., 2000; Eyler, 2018). Several meta-analyses have found that active learning, as compared
to passive learning, leads to superior knowledge gains, and is particularly valuable for underrepresented
minority and low-income students (Freeman et al., 2014; Theobald et al., 2020). Moreover, activities in
the Learning Lab often involve students working collaboratively, learning together and from each other.
We foster interactive activities because they can lead to higher learning gains than situations where
students work independently (Chi, 2009; Eyler, 2018; Wiggins et al., 2017).
We frequently encounter faculty who invite a range of creative final assignments—from interpreting a
biological process through poetry or artwork, to producing films, songs, or plays that mirror or rework
the canonical works students study in humanities courses. To reinforce and accentuate the learning in
these assignments, we encourage faculty to include an “artist’s statement” as part of the assignment, in
which students analyze their work and draw connections with concepts and readings from the course,
often through orally recording their reflection on camera in our studio. This type of reflection builds
students’ metacognitive awareness, giving students evidence-based opportunities to reflect on their
knowledge (Bransford et al., 2000).
Echoes of the Research in the Practices

If the tacit knowledge of our academic clients conformed in many respects to what we knew from the
research on teaching and learning, it perhaps should not have surprised us that the artists and
331
technologists with whom we collaborated—coming from backgrounds ranging from dance to graphic
design to virtual reality development—also came with skills, practices, and intuitions that frequently
aligned with the research on teaching, learning, and communication.
To take one example, in one of our earliest workshops for faculty preparing to make the leap to online
teaching for HarvardX, we planned to make use of Richard Mayer’s chapter, “Research-Based Principles
for Designing Multimedia Instruction” (2014; see also, Mayer, this volume), along with some quick tips
on slide design from a staff member with graphic design training. In planning for the workshop, it
became clear that the graphic designer’s tips corresponded to an uncanny degree with Mayer’s
principles. Mayer’s “spatial contiguity” principle, for example, calls for close relationships between
related words and images of the sort that the designer was demonstrating, and Mayer’s “coherence
principle” is based on evidence that learners do best when extraneous visual information is excluded
rather than included; modeling Mayer’s principles gave the graphic designer a way to explain the value
of the clean, minimal design they championed.
In effect, the two approaches complemented each other perfectly. Mayer’s “Research-Based Principles”
gave us a way to explain what we initially thought were merely aesthetic preferences by grounding them
in research, and the graphic designer was able to give us ideas on how to put Mayer’s principles into
practice. It is one thing to demonstrate, as Mayer does, that “the signaling principle” calls for visual and
auditory cues about the content’s conceptual structure; it is quite another to accomplish this practically,
especially when at times it seems that these additional cues might end up becoming the sort of
“extraneous information” that violate the “coherence principle.” In the visual domain, graphic designers
have deep experience with these sorts of tensions between conflicting goals and entire toolboxes full of
strategies for resolving them.
This scenario has repeated itself across a wide array of media over the years as the Learning Lab has
worked to bring the science of learning together with the skills of artists, designers, and technologists.
We know from the literature that students learn best from interactive rather than passive experiences
(Chi, 2009; Freeman et al., 2014), but the research cannot provide the thousands of interactive game
mechanics that we find in the work of game-designers. We know, again from Mayer, that students learn
best from natural, informal, personalized vocal communication that uses oral cues to highlight key
information (Mayer, 2014), but the expert skills required to train faculty in vocal presentation are more
likely to exist in the theatre department (and, indeed, in our case collaboration with vocal coaches from
our theatre program have been transformative for faculty).
Since our goal has been not to produce new research findings, but, rather, to find innovative and
effective ways of putting research into practice, we have developed a community, not only of scholars,
teachers, and learners, but also of artists, designers, and technologists whose practices can help
instructors bring their ideas to life.
Discovering a Student-Centered Process

The model of support we developed for the oral and visual presentations of MCB81 worked well
because it leveraged the skills of our Center’s internal staff. But it was daunting to imagine creating that
entire range of support, from assignment design through implementation in a course, for every new
medium or unusual project that faculty brought to our door. In order to scale beyond the initial
HarvardX-inspired video assignments, we needed to develop a dynamic, fast-moving team.
The key development that helped us survive was the introduction of two groups of student fellows:
● Media and Design Fellows (MDFs) who, as PhD students, combine their disciplinary expertise
with expertise in a specific communication medium, and
332
● Learning Lab Undergraduate Fellows (LLUFs), who serve as usability testers, all-purpose
assistants and, as we will outline below, co-designers of activities and assignments.
The graduate fellows program launched first, with PhD students from an array of departments joining us
for ten hours per week to develop their skills in a particular medium of communication and to work on
connecting this medium to their academic disciplines. The program unleashed the creativity of the
graduate students involved in energizing and generative ways. We discovered that our PhD students in
neuroscience, anthropology, and Romance Languages have backgrounds as graphic designers, dancers,
and video game developers. Where we did not find students already possessing these skills, we found
many willing to learn and highly capable of doing so. Philosophers learned to become web developers
and biologists developed graphic design skills. The ideas that they developed at the intersection of their
academic and media skills were remarkable.
These graduate student fellows became essential partners in all stages of the Learning Lab process.
Oftentimes, faculty would come in with an idea for a course project but not have the time to be present
for every step of the iterative design process. Graduate students who were familiar with the course or
had relevant disciplinary expertise took on the role of the instructor if we were testing components of
workshops or materials that would be presented to students; they developed student-facing materials
for the assignments, as well as grading rubrics; and they often had ideas for course innovations
themselves and could connect new faculty members to the Learning Lab.
As for the Learning Lab Undergraduate Fellows (LLUFs), our initial plan was simply to hire a group of
students as user testers. This prompted us to create a more structured and systematic workflow for
project development:
1. Our team would have initial meetings with faculty to get a high-level sense of the request.
2. In conjunction with course staff, our graduate fellows would work to design prototypes of the
assignments, activities, or other interventions the course requested.
3. We would offer the prototypes to the LLUFs for testing and feedback.
4. We would revise our work based on that feedback and perform further test and revision cycles
as needed.
5. We would submit the final designs to the faculty for approval.
6. Finally, we would offer extensive support for the implementation of the new assignment or
activity when the course ran in a subsequent term through tutorials, workshops, hackathons,
and end-of-term showcases of student work.
This process, influenced heavily by design thinking (Luka, 2020), and more specifically by the Agile
software development methodology (Beck et al., 2001), allowed us to iterate through prototypes and
test at a pace that is rather unusual at our institution. In contrast to a typical scenario in which faculty
try something new one year and look at the evaluations at the end of the term in order to revise for the
following year, in the Learning Lab, the graduate fellows could design something on Monday, test it on
Tuesday, revise it on Wednesday, then test again on Thursday. We should emphasize that this frequency
of iteration departed from the norm by orders of magnitude. Our intergenerational team, with the
undergraduates at the core, could iterate weekly or even daily, dramatically speeding the
experimentation process. Moreover, the speed at which we worked (and the fact that we were working
with test students) made failure less frightening, and faculty who would not risk a particular innovation
in their course of 300 could try it out in a test group of ten students first—and they could try it out
multiple times.
333
We found the undergraduate testers (or LLUFs) to be invaluable. The LLUFs would help us determine
ostensibly simple things, such as whether the assignment was logistically feasible in the time allotted.
But they would also help us discover more subtle pitfalls, such as identifying common student
misconceptions; understanding misconceptions is important for designing successful learning
experiences (Bransford et al., 2000). The LLUFs’ feedback was crucial to our experimental and iterative
approach to developing the scaffolding and assessment of new modes of academic communication.
While their role as user testers made the LLUFs essential to our quality of work, it became clear to us
that their contributions went beyond the constraints of the user tester role. They were still crucial to the
‘testing’ step in the workflow mentioned above, but they were also beginning to help out in all the other
steps as well. They had become something more than just testers: they were collaborators, co-
designers, and co-creators. It is true that they were not yet experts on the content we helped our faculty
teach, but they were experts on their own experiences as learners, and, working from this expertise,
their contributions went far beyond simple user feedback. Frequently, they would contribute new and
more effective ways of structuring activities, ways of incorporating examples and cultural references
more appropriate for their peers, and approaches to make sure that the activities were equitable and
inclusive. As they developed their skills through testing and co-developing these activities, they became
capable of leading workshops, office hours, and hackathons, during which they would teach their peers.
The LLUFs’ expanded portfolio provided them with a sense of agency, as they took an active role in
determining the course of their own learning and in shaping the curriculum (National Academies of
Sciences, Engineering, and Medicine, 2018b).
Just as we have found the presence of the LLUFs to be essential to the process and ecosystem of the
Learning Lab, others have identified numerous benefits of partnering with students to develop curricula.
For example, student partners develop a greater metacognitive awareness of the learning process; they
exhibit increased engagement and motivation, and they take more responsibility for their own learning
(Bovill et al., 2011). Moreover, encouraging LLUFs to lead workshops for their peers aligns with research
that shows that students who are responsible for teaching others make a greater effort to learn and
show greater retention of the material (Chase et al., 2009; Fiorella, this volume; Fiorella & Mayer, 2016).
Teaching requires students to organize, integrate, and generate knowledge, and engage in retrieval
practice (National Academies of Sciences, Engineering, and Medicine, 2018b).
Within our intergenerational team, the boundaries between teachers and learners became more fluid.
We were serving the faculty, but in the context of the Learning Lab, senior faculty would sit at our
worktables and learn about undergraduates’ experiences, and fifth-year PhD students would collaborate
with freshmen on augmented reality projects. These collaborations yielded insights that would have
been harder to achieve if everyone involved had adhered strictly to their institutional roles and
identities. With the arrival of the LLUFs, something had changed. It broadened and energized the
intergenerational team we were developing to an extent we had not yet observed, and it began
producing emergent phenomena for which we had not planned, but which were valuable and
transformative to our work.
Intergenerational Collaboration and Inclusion

What began to emerge within the Learning Lab was a thriving intergenerational community of teachers
and learners. We were cultivating a space where faculty, graduate students, and undergraduates could
work collaboratively at a slight distance from their assigned roles in their lives outside our studio. This
yielded an array of unplanned developments and opportunities, not the least of which surrounds our
core goal of inclusive and equitable learning.
334
Centers for teaching and learning have long worked with instructors to create inclusive classrooms and
equitable learning outcomes (Ouellett, 2010). But, perhaps inevitably given the goal of faculty
development, some of the training and programming tends to put the instructor at the center. In
workshops on “handling hot moments in the classroom,” for instance, it is difficult to avoid seeing the
classroom from the instructor’s perspective, casting the student as “other,” the object representing a
challenge to be overcome. Indeed, creating a safe space for instructors to share their difficult
experiences with each other is and always will be an essential component of the work of teaching and
learning centers, and continues to be essential across the Bok Center’s work.
But within the Learning Lab’s Studio, something changed when the identity of our core team shifted
away from being centered solely in teaching experience and expertise. It became increasingly
unreasonable to drift into an “us against them” mentality, because the “us” now included both teachers
and students, sixty-year-olds and twenty-year-olds, experts and non-experts. At times this was
challenging, because our studio—packed with undergraduates and graduate students—was no longer a
sanctuary for faculty visitors, no longer a safe space for hearing confidential concerns or offering
confidential advice.
In place of confidential advice on what to do with their students, we were able to offer actual students
as partners with whom faculty could think through their teaching challenges collaboratively. No matter
how much our expert staff reads or studies or reflects, we cannot completely place ourselves in the
position of the student (Camerer et al., 1989; Xiong et al., 2020), so there was an authenticity to these
encounters that we could not match. Students might not be “experts” on teaching and instructional
design, or on various academic disciplines, but they are experts of their own experiences and
communities and, ultimately, it was frequently this expertise that our faculty were seeking. Each year
students arrive on our campus with newly emergent investments, concerns, and languages for
articulating their goals and desires and identities. And without this expertise, it is difficult to design fully
inclusive learning environments and experiences.
Ultimately it became clear that students were not merely assisting our Center’s staff as we offered
professional development opportunities for faculty. The students themselves were offering us—the
Center’s staff as well as the faculty—our own professional development opportunities.
The Future of Academic Work

Not only does the Learning Lab enhance the learning experience of today’s students and support faculty
in the courses they are teaching now, but the Learning Lab also presents a model for preparing future
faculty. At the center of that model are the Learning Lab’s own graduate student fellows. These
graduate students play an active role in the ongoing pedagogical innovation that we have described
above, often serving as the point of contact for faculty in their departments and the Learning Lab. As
fellows, they develop skills across an array of technologies that are relevant for the future of their
disciplines, and they critically engage questions about multimodal communication and future directions
for scholarship with our intergenerational and multidisciplinary team. Through this process, graduate
students are partners in creating the future of their disciplines.
We see the impact of this training when graduate students pursue pedagogical projects that extend
beyond their fellowship year and contribute in significant ways to their future careers. Graduate
students from the music department have proven particularly adept at leveraging the framework of the
Learning Lab: A founding graduate fellow from the music department developed podcasting
assignments for music theory courses that allowed students to analyze scores, used the video studio to
record feedback on student compositions, and now leads a faculty working group on digital project
design at the institution where he is currently on the faculty. Another graduate fellow from music took a
335
different approach, focusing on databases and code as a way to collect and analyze student interactions
with music. It was through his work that the Learning Lab came to build custom tools for courses—web-
based games, widgets, and other interfaces—on a regular basis. Of course, graduate student innovations
were not limited to music theorists. Graduate fellows in biology worked on assignments that asked
students to communicate science to a public audience in a range of media, from drawing to video
production to game design. Several of these students have gone on to faculty positions with a focus on
STEM pedagogy. A graduate fellow in Southeast Asian Studies received a grant to develop a 3D
interactive exploration of historic sites in India, and a philosophy graduate fellow, who later joined the
Learning Lab’s full-time staff, leveraged work on the use of visualization to teach logic into professional
skills in web development.
In addition to training our cohort of fellows, the work of the Learning Lab contributes to training
graduate students more broadly across the university, in alignment with recommendations from the
National Academies that graduate education teach transferable skills, such as communication and
collaboration (National Academies of Sciences, Engineering, and Medicine, 2018a, p. 107). Materials
developed at the Learning Lab have resulted in new Bok Seminars, short courses open to all graduate
students, which can be taken to earn a teaching certificate. These include sessions on multimedia
presentation, theater in the classroom, game design principles for teachers, and more. The hope is that
every graduate student who encounters the Bok Center will appreciate that developing their teaching
skills results in their developing an array of transferable skills, from communication to collaboration to
multimedia creation, that will give them the ability to shape the future of scholarship and pedagogy in
their disciplines and beyond the academe.
21st Century Media, Authenticity, Equity, and Justice

So far, we have presented a rationale for multimodal communication that is grounded in its importance
as knowledge-construction, and, indeed, this alone is all the justification many faculty need. But we are
also encountering an increasing number of faculty and students for whom it is not enough to simply
describe the world accurately; they hope to intervene in the world to change it (Marx et al., 1972, p.
145).
Whether it is a course on the science of climate change, the sociology of higher education, or gender
and politics in American culture, we find faculty and students wishing not merely to learn within the
University, but also to bring some of what they have learned into the world, beyond the ‘gates’ of
academia. When faculty and students look to the modes of education and persuasion that have impact
outside of higher education, they find YouTube channels and video games and museum installations and
infographics rather than academic papers. These are the modes of public education they seek our help
in practicing, and the Learning Lab has responded by developing robust strategies to support them.
When students undertake projects in which they address issues of concern to the broader community,
or share their learning with a public audience, they feel that their education matters. Personal relevance
can increase students’ interest in a subject, along with their performance in a course (Hulleman &
Harackiewicz, 2009). Connecting one’s education with the community—and having some choice in the
implementation—capitalizes on students’ intrinsic motivation, which is greatest when students perceive
a connection to other people, that they have some autonomy over a project, and that they are
competent at achieving a task (Ryan & Deci, 2000). Perhaps not surprisingly, community engagement, or
service learning, was listed as one of ten “High Impact Practices,” based on data from the National
Survey of Student Engagement (Kuh, 2008).
336
Conclusion
Our iterative and agile approach to design has allowed us to offer our faculty maximum freedom to
explore their inspirations while simultaneously bringing these inspirations into alignment with the
science of learning in a relatively organic way. Our approach—particularly our collaboration with the
Learning Lab Undergraduate Fellows—has allowed us to rediscover something hidden in plain sight: the
student’s centrality to the teaching and learning process. We often find ourselves returning to a
particular passage from Dewey’s How We Think (1997):
Teaching and learning are correlative or corresponding processes, as much so as selling and
buying. One might as well say he [or she] has sold when no one has bought, as to say that he [or
she] has taught when no one has learned. And in the educational transaction, the initiative lies
with the learner even more than in commerce it lies with the buyer. (p. 29)
This passage sounds relatively obvious, but is, on further examination, quite radical. For if the instructor
cannot say they have taught if no one has learned, then we could argue that a center for teaching and
learning cannot say that it has supported teaching if it has only involved itself in the professional
development of the teacher in isolation. Meetings with faculty and trainings for teaching assistants can
produce the conditions for great teaching, and great teachers in potentia, but without learners learning,
there is not yet any actual teaching.
Essentially, we are arguing that experiments in teaching are simultaneously experiments in learning, and
that the learners need to be deeply involved—and central—in the beginning, the middle, and the end.
By locating students at the center of the process, in the center for teaching and learning, we have
created a community where teaching and learning are happening each day. This is a virtuous circle
within which teachers practice teaching and learners practice learning, where the lines between
teachers and learners are not drawn once and for all.
We created the Learning Lab, in part, based on inspiration from the MOOC, but with a goal of moving
away from a one-way monologue and instead fostering learner-centered teaching environments. Rather
than primarily recording instructors on camera, we created a process for capturing students and their
ideas on camera, and soon students were integral to all aspects of our work. From this start, we have
created an ongoing dialogue with students, which has helped us understand their aspirations: their
career goals; their need to become superb visual narrators of their lives, careers, and relationships on
social media; their hope to connect what they do in college to the world beyond; and their desire to
change the world with what they learn.
While our learners are learning new academic content and new modes of communication, our teachers
are learning too. They are learning to teach, but they are also teaching to learn. It is through teaching
that they learn ways of making their work meaningful to the students in their classes and even to the
wider world beyond. The reciprocal relationship between our learners’ desires and the work of our
teachers has created a generative dialogue between the University itself and the “real world” it hopes to
instruct and serve. Just as we earlier mentioned the way an iterative dialogue between teaching and the
research on learning can bring the two into alignment, here too we see the possibility of achieving
greater alignment between the knowledge generated by the University and the world that exists beyond
its gates.
337
Author Note
We are grateful for the leadership of Robert Lue (1964-2020), Richard L. Menschel Faculty Director of
the Derek Bok Center for Teaching and Learning (2013-2020), whose vision led to the creation of the
Learning Lab.
Corresponding author: Tamara Brenner. Email: [email protected]
References
Ainsworth, S., Prain, V., & Tytler, R. (2011). Drawing to learn in science. Science, 333(6046), 1096–1097.
Ambrose, S. A., Bridges, M. W., DiPietro, M., Lovett, M. C., & Norman, M. K. (2010). How learning works:
Seven research-based principles for smart teaching. John Wiley & Sons.
Bandura, A. (1997). Self-efficacy: The exercise of control. W.H. Freeman.
Beck, K., Beedle, M., Van Bennekum, A., Cockburn, A., Cunningham, W., Fowler, M., Grenning, J.,
Highsmith, J., Hunt, A., Jeffries, R., Kern, J., Marick, B., Martin, R.C., Mellor, S., Schwaber, K.,
Sutherland, J., & Thomas, D. (2001). Manifesto for agile software development.
https://agilemanifesto.org/
Bovill, C., Cook‐Sather, A., & Felten, P. (2011). Students as co‐creators of teaching approaches, course
design, and curricula: Implications for academic developers. International Journal for Academic
Development, 16(2), 133–145. https://doi.org/10.1080/1360144X.2011.568690
Bransford, J. D., Brown, A. L., & Cocking, R. R. (2000). How people learn: Brain, mind, experience, and
school: Expanded Edition. National Academies Press. https://doi.org/10.17226/9853
Camerer, C., Loewenstein, G., & Weber, M. (1989). The curse of knowledge in economic settings: An
experimental analysis. Journal of Political Economy, 97(5), 1232–1254.
Chase, C. C., Chin, D. B., Oppezzo, M. A., & Schwartz, D. L. (2009). Teachable agents and the protégé
effect: Increasing the effort towards learning. Journal of Science Education and Technology,
18(4), 334–352. https://doi.org/10.1007/s10956-009-9180-4
Chi, M. T. H. (2009). Active-Constructive-Interactive: A conceptual framework for differentiating learning
activities. Topics in Cognitive Science, 1(1), 73–105 https://doi.org/10.1111/j.1756-
8765.2008.01005.x
Dewey, J. (1997). How we think. Courier Corporation.
Eyler, J. (2018). How humans learn: The science and stories behind effective college teaching. West
Virginia University Press.
Fiorella, L., & Mayer, R. E. (2016). Eight ways to promote generative learning. Educational Psychology
Review, 28(4), 717–741. https://doi.org/10.1007/s10648-015-9348-9
Frankel, F. (2005). Translating science into pictures: A powerful learning tool. Invention and impact:
building excellence in undergraduate science, technology, engineering, and mathematics (STEM)
education, 155-158. AAAS. https://www.aaas.org/sites/default/files/05_Vis_Ed_Frankel.pdf
338
mathematics. Proceedings of the National Academy of Sciences, 111(23), 8410–8415.
Gobert, J. D., & Clement, J. J. (1999). Effects of student-generated diagrams versus student-generated
summaries on conceptual understanding of causal and dynamic knowledge in plate tectonics.
Journal of Research in Science Teaching, 36(1), 39–53. https://doi-
org.unh.idm.oclc.org/10.1002/(SICI)1098-2736(199901)36:1<39::AID-TEA4>3.0.CO;2-I
Handelsman, J. (2004). Scientific teaching. Science, 304(5670), 521–522.
Hulleman, C. S., & Harackiewicz, J. M. (2009). Promoting interest and performance in high school science
classes. Science, 326(5958), 1410–1412. https://doi.org/10.1126/science.1177067
Luka, I. (2020). Design thinking in pedagogy. Journal of Education Culture and Society, 5(2), 63–74.
https://doi.org/10.15503/jecs20142.63.74
Kuh, G. D. (2008). Excerpt from high-impact educational practices: What they are, who has access to
them, and why they matter. Association of American Colleges and Universities, 14(3), 28-29.
https://www.aacu.org/publication/high-impact-educational-practices-what-they-are-who-has-
access-to-them-and-why-they-matter
Marx, K., Tucker, R. C., & Engels, F. (1972). The Marx-Engels reader. Norton.
Mayer, R. E. (2014). Research-based principles for designing multimedia instruction. In V. A. Benassi, C.
E. Overson, & C. M. Hakala (Eds.), Applying science of learning in education: Infusing
psychological science into the curriculum (pp. 59–70). Society for the Teaching of Psychology.
https://teachpsych.org/ebooks/asle2014/index.php
National Academies of Sciences, Engineering, and Medicine. (2018a). Graduate STEM education for the
21st century. The National Academies Press. https://doi.org/10.17226/25038
National Academies of Sciences, Engineering, and Medicine. (2018b). How people learn II: Learners,
contexts, and cultures. The National Academies Press. https://doi.org/10.17226/24783
Ouellett, M. L. (2010). Overview of diversity issues relating to faculty development. In K. J. Gillespie, D. J.
Robertson, & W. H. Berquist (Eds.), A guide to faculty development (pp. 185–202). Jossey-Bass.
Quillin, K., & Thomas, S. (2015). Drawing-to-Learn: A framework for using drawings to promote model-
based reasoning in biology. CBE Life Sciences Education, 14(1). https://doi.org/10.1187/cbe.14-
08-0128
Ryan, R. M., & Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic motivation,
social development, and well-being. American Psychologist, 55(1), 68–78. https://doi-
org.unh.idm.oclc.org/10.1037/0003-066X.55.1.68
Sawyer, R. K. (2014). The Cambridge handbook of the learning sciences (Second edition). Cambridge
University Press.
Schwartz, D. L., Tsang, J. M., & Blair, K. P. (2016). The ABCs of how we learn: 26 scientifically proven
approaches, how they work, and when to use them. W.W. Norton & Company, Inc.
Slater, S. J., Slater, T. F., & Bailey, J. M. (2010). Discipline-based education research: A scientist’s guide.
W.H. Freeman.
339
Theobald, E. J., Hill, M. J., Tran, E., Agrawal, S., Arroyo, E. N., Behling, S., Chambwe, N., Cintrón, D. L.,
Cooper, J. D., Dunster, G., Grummer, J. A., Hennessey, K., Hsiao, J., Iranon, N., Jones, L., Jordt, H.,
Keller, M., Lacey, M. E., Littlefield, C. E., . . . Freeman, S. (2020). Active learning narrows
achievement gaps for underrepresented students in undergraduate science, technology,
engineering, and math. Proceedings of the National Academy of Sciences, 117(12), 6476–6483.
Weimer, M. (2013). Learner-centered teaching: Five key changes to practice (Second edition). John Wiley
& Sons.
Wiggins, B. L., Eddy, S. L., Grunspan, D. Z., & Crowe, A. J. (2017). The ICAP Active Learning Framework
Predicts the Learning Gains Observed in Intensely Active Classroom Experiences. AERA Open,
3(2), 2332858417708567. https://doi.org/10.1177/2332858417708567
Xiong, C., Van Weelden, L., & Franconeri, S. (2020). The curse of knowledge in visual data
communication. IEEE Transactions on Visualization and Computer Graphics, 26(10), 3051–3062.
https://doi.org/10.1109/TVCG.2019.2917689
340
In Their Own Words Considerations for Academic Support Centers
Regarding Learning Strategy Recommendations
to Apply the Science of Learning in Jennifer A. McCabe
Goucher College
Editors
Lauren L. Kordonowy
Victor A. Benassi
Academic support centers (ASCs), embedded within many institutions of higher education, typically
serve as both the front line of help for students struggling with course demands and for the larger
student body in support of academic success. Under a variety of names, ASCs offer many services,
including workshops, videos, peer tutoring, and individual appointments with staff members. ASCs can
take both proactive (e.g., skills training or study tips for all students) and reactive (e.g., targeting support
to students experiencing academic difficulties, connecting them with specific resources) approaches.
Thus, the responsibilities of ASCs are complex and multi-faceted, requiring skilled and knowledgeable
practitioners to deliver effective services.
What Memory Researchers Know About Learning

In view of the placement of ASCs in higher education and given that students are frequently directed (or
self-direct) to their services, it is essential that they offer evidence-based advice on how to learn.
Memory and education researchers have accumulated decades of research to support several major
categories of learning strategies that have been consistently demonstrated as effective in the laboratory
and in the classroom. Yet, at least historically, there has often been a disconnect between this research
and ASCs.
Part of the disconnect is that some of the strategies that are most effective for durable learning tend to
be unintuitive or counterintuitive. Also problematic is that some popular strategies, such as re-reading,
underlining, and highlighting, are less beneficial for learning (for a review, see Dunlosky et al., 2013;
Roediger & Pyc, 2012). Cognitive psychologists, and memory researchers specifically, endorse a number
of strategies that can be grouped into three categories: spacing, elaboration, and testing. These can be
remembered using the mnemonic “SET” or the phrase “SET for success” (McCabe et al., 2021).
The strategy of spacing is based on the well-established principle that, given the same amount of study
time, memory is improved if the learner processes the material in shorter sessions with breaks in
between (what researchers refer to as distributed practice), compared to one long study session
(commonly called cramming, and what researchers call massed practice) (e.g., Rohrer & Pashler, 2008,
cf. Rohrer & Hartwig, this volume). Although spacing in this form is a relatively intuitive strategy, a
survey study I conducted suggests that students’ reports of ideal or intended spacing of their study time
are more distributed than their actual study schedule (Susser & McCabe, 2013). In other words, students
seem to understand that spacing is better for learning, and they want to use it, but they run out of time
and end up cramming. I imagine this story is quite familiar to ASC staff and others reading this chapter.
This highlights the critical importance of teaching students time management skills, such that they have
a schedule that will allow for spaced or distributed learning.
341
A special case of spacing is interleaving, or mixing up the order of topics being studied into small chunks
instead of blocking out extended time for one topic before moving on to the next (e.g., Kornell & Bjork,
2008; Kang, this volume; Rohrer & Hartwig, this volume). This strategy is counterintuitive, and research
suggests that many students and instructors are unaware of the benefits of interleaving. Indeed, many
believe the opposite to be true – that a large block of study time focused on one topic is better than
using that same block of time to study various topics, switching back and forth (McCabe, 2011;
Morehead et al., 2016). Blocking can play a role, however, in the overall scheme of mastering course
content. When a student is just beginning to learn new material, initial blocking of study before moving
to distributed, interleaved study may produce better learning (Kang, this volume).
Elaboration is an umbrella term for several strategies that can boost learning through meaning-based
connections with to-be-learned material. Specific examples of elaboration include the use of concrete or
real-world examples, self-explanation, asking and answering “why” and “how” questions, visual imagery
(pictures, charts, graphs), and mnemonic devices that provide an organizational scheme for otherwise
hard-to-organize information. Common examples of mnemonics include ROY G BIV, Please Excuse My
Dear Aunt Sally, and the alphabet song. More elaborate tools like the keyword method and the memory
palace (or method of loci) have been shown to support learning as well (for reviews, see Belleza, 1996;
McCabe et al., 2013). Several elaborative techniques that use visual imagery (such as imagining a hippo
walking through a college campus and thinking, “I’ll definitely remember that!” to learn that the
hippocampus is involved in long-term memory formation) are effective due to dual-coding, which
creates multiple codes (or pathways) in varying modalities to improve the chance that the memory can
be retrieved later (Paivio et al., 1968).
The learning strategy with the most robust body of evidence is testing, or retrieval practice. Given the
same amount of time to study material, it is far better to spend that time retrieving the recently learned
information from long-term memory (e.g., a quiz) than it is to spend the time re-reading the material
(e.g., Roediger & Karpicke, 2006; Yang et al., this volume). Retrieving information from memory is more
effortful, which makes learning more durable, because retrieval strengthens the pathways to access the
information in the future. As a side benefit, testing is also a metacognitive tool in that it can provide
learners with feedback on how much they have learned and areas in need of continued study. Testing is
an example of the broader memory principle of generation. When learners create or effortfully initiate
processes to study to-be-learned materials (e.g., make a study guide, teach another person from
memory), learning will tend to be more durable (Slamecka & Graf, 1978). You may notice that some
examples of generation tap both testing and elaborative processes, which further supports the power of
generation as a learning tool.
The “SET” strategies are examples of desirable difficulties (Bjork, 1994; Bjork & Bjork, this volume), in
that they are more effortful and challenging (and can take more time and lead to errors) in the short
term, but they lead to far more durable learning in the long term. Unfortunately, there is a
metacognitive disconnect because students show poor understanding of interleaving and testing
(McCabe, 2011), even though they have better knowledge of spacing and generation. Although
instructors are generally more knowledgeable of these learning strategies, there is still room for
improvement (Morehead et al., 2016). Thus, two challenges emerge: 1. educating students on the value
of these strategies and 2. motivating them to actually implement the strategies in their studying. The
former is happening more and more in higher education (including at ASCs), but the latter is quite
difficult. (Refer to chapters in Part 4 of this volume for detailed consideration of this issue.)
To take one example of classroom research to address these questions, my colleagues and I recently
conducted an intervention study in an introductory psychology laboratory course in which students
received training on the “SET” strategies. Findings revealed that after training, students’ demonstrated
342
an increased knowledge of the strategies and modestly increased self-reported use of elaboration and
testing by the end of the course (McCabe et al., 2021). Not surprisingly, the big picture from this and
other research suggests that even when knowledge about effective strategies improves, students would
often prefer to fall back on strategies that feel familiar, easier, or make the material feel more fluent
more quickly. It is therefore critical for students to know that, although strategies such as re-reading and
highlighting can give the illusion of learning, their actual learning is likely of a shallow nature, and
therefore not likely to last. Equally important is to emphasize to students the value of understanding
how memory works (e.g., basic cognitive processes involved in effective encoding, storage, and
retrieval), both in academic settings and for lifelong learning.
What Academic Support Centers Know (and Share) About Learning

To better understand whether ASC staff are aware of the utility of the evidence-based strategies
described above, and to what extent they recommend these strategies to the students they serve, I
conducted a survey study of heads of ASCs from higher education institutions across the U.S. (McCabe,
2018). Of the 400 ASC heads invited, 77 completed the survey. Participants responded to open-ended
and closed-ended questions about the learning strategies they endorse and recommend to students.
When asked in a free-response question about the top three learning strategies ASC heads recommend,
by far the most reported strategy was “time management” (58%). Focusing on the evidence-based
strategies discussed above, 23% commented on spacing, and only 10% reported that they
recommended organizational outlines, asking and answering questions, flashcards, multiple modes of
studying, and mnemonics.
In the next part of the survey, the ASC heads rated (on a 5-point scale) how much they believe 36
specific strategies to be effective and how often they recommend these strategies to the students they
serve. As effectiveness and recommendation ratings were strongly correlated, I will focus on
summarizing recommendation ratings for the evidence-based (“SET”) strategies (see Table 1, organized
in columns by scale intervals from “never” to “always”).
Table 1
Recommendation Frequency Ratings for Learning Strategies Relevant to Evidence-Based and Non-
Evidence-Based Principles
Never – Rarely Rarely – Sometimes Sometimes – Often Often – Always
Recommend Recommend Recommend Recommend
Write notes verbatim Plan for longer study Incorporate a variety of Discuss course
from course sessions focused on one modalities (e.g., visual, materials with
materials (1.81) subject (2.41) auditory (3.97) another person (4.34)
Focus on one type of Use pictures/diagrams Answer questions
course material for an (3.97) about the material
extended time (M = (4.31)
Teach material to a real or
2.22)
imagined other (3.96) Self-test (4.27)
Use mnemonics (memory Plan for shorter study
aides such as acronym, sessions with breaks
keyword, or song) (3.60) in between (4.16)
Study in a way that is
consistent with the
343
student’s learning style Use concrete real-life
(e.g., visual auditory) (3.55) examples (4.13)
Review highlighted or Create study
underlined passages from materials (4.12)
text or notes (3.55)
Use outlines or other
Use flashcards (3.53) organizational
materials (4.04)
Highlight or underline
important parts of text or
notes (3.49)
Study the same course
material in at least two
sessions (3.44)
Mix up the order of related
materials that are studied
during a single session
(3.34)
Note. Strategies are listed in descending order by mean in each scale-interval column. Means in
parentheses represent the 1-5 scale (1 = never, 2 = rarely, 3 = sometimes, 4 = often, 5 = always) for
recommendation frequency. Regular font indicates strategies not empirically supported or otherwise
counter to evidence-based strategies, whereas bold font indicates evidence-based strategies. Adapted
from “What learning strategies do academic support centers recommend to undergraduates?” by J. A.
McCabe, 2018, Journal of Applied Research in Memory and Cognition, 7, p. 146
(https://doi.org/10.1016/j.jarmac.2017.10.002)
Summary Discussion of ASC Head Ratings in Table 1
Spacing-related strategies
ASC heads reported more frequently recommending plan for shorter study sessions with breaks in
between than study the same course material in at least two sessions. Therefore, the former may be a
more obvious or intuitive spacing strategy, even though the memory benefit from spaced, or
distributed, practice comes from the latter iteration. More important, and encouragingly, the ratings for
plan for longer study sessions focused on one subject were far lower than the two spacing items above.
ASC heads demonstrated moderate endorsement of interleaving strategies. First, there were higher
ratings for mix up the order of related materials that are studied during a single session compared to
focus on one type of course material for an extended time. Thus, the overall picture is quite positive with
regard to ASC heads recommending spacing strategies to students.
Elaboration
Several relevant strategies were frequently recommended to students; in descending order of ratings:
discuss course materials with another person, answer questions about the material, use concrete real-life
examples, create study materials, use pictures/diagrams, teach material to a real or imagined other
(Fiorella, this volume), and use mnemonics. A takeaway from these results is that, although there is
some variation in frequency of recommendation, many elaborative strategies grounded in meaning-
based connections are recommended by ASCs.
344
Testing strategies
ASC heads heavily recommended the most obvious item, self-test. Another item, use flashcards, was less
frequently endorsed. Flashcards, when used correctly to encourage effortful retrieval from long-term
memory, can be an excellent way to implement the testing principle (cf. Dunlosky et al., this volume).
However, flashcards used more casually, such as skimming or checking the answer without memory
retrieval, would not incur the benefits of testing. Therefore, it is not fully clear how to interpret the
lower mean for this item. The important takeaway for ASC staff is to teach students to use flashcards in
a way that requires retrieval from long-term memory, such that this strategy will strengthen memory for
the material; they should also convey to students that any type of self-test can give metacognitive
feedback about their current state of learning. This double-benefit for learning is part of what makes
testing, or retrieval practice, so powerful. It is also worth noting that the highly-rated item discussed
above for elaboration, answer questions about the material, could also be relevant to testing principles
if the learner were to try to answer the questions from memory without referring to the material.
This optimistic picture of endorsement and recommendation for some of the core evidence-based
learning strategies was further bolstered by the fact that none of the more shallow strategies (that
would NOT be endorsed by memory researchers as a means to deep and durable learning) had
particularly high ratings: highlight or underline important parts of text or notes, re-read course materials
for review, write notes verbatim from course materials. It is important to point out that, although several
of these strategies were not very often recommended, they were reported at similar levels to some
evidence-based strategies; specifically, review highlighted or underlined passages from text or notes was
equivalent or numerically higher than several effective strategies (Table 1). In summary, and as
discussed at more detail in my article (McCabe, 2018), there is much to commend with regard to
evidence-based practices at ASCs, but there is also room for improvement on some important fronts.
The Pervasive Myth of Learning Styles

The idea of learning styles is that individuals may report that they have a specific modality or means
through which they learn best and, by extension, that they should learn better when the manner of
presentation matches their learning style. There are many variations on learning styles, and several ways
to group them, although perhaps the most common is to divide students into those who self-identify as
‘visual learners,’ ‘auditory learners,’ and ‘kinesthetic learners.’
Although researchers have failed to find evidence in support of the benefits of matching individual
learning styles to the mode of to-be-learned material presentation (Pashler et al., 2009; Kirshner, 2017;
Willingham et al., 2015), 9% of the ASC heads in my study (McCabe, 2018) reported that they
recommend that students know and apply learning styles as a top recommendation; this was mentioned
far more often than several evidence-based strategies described above. In the closed-ended ratings
section, the results were more encouraging, in that the strategy to study in a way that is consistent with
the students’ learning style (e.g., visual, auditory) was less frequently recommended than what could be
considered the alternative, incorporate a variety of modalities (e.g., visual, auditory). Yet the learning
styles strategy was rated fairly strongly given the lack of empirical evidence; learning styles received an
equivalent or higher frequency rating than evidence-based strategies such as use flashcards and mix up
the order of related materials that are studied during a single session.
It turns out that the learning styles idea is one of the most pervasive myths in education, with one study
reporting that 58% of students believed they have a specific learning style, and that 91% of university
instructors endorsed the idea (Morehead et al., 2016). Clearly there is a need to better communicate
that the notion of learning styles does not exist, at least not in a way that matters for actual learning. A
recently released “viral video” communicates that learning styles are a myth in an easily digestible,
345
accurate, and entertaining way. Several other videos by leaders in memory and education research help
to combat the learning styles myth – one featuring Dr. Daniel Willingham, who is also interviewed in the
video, and another featuring Dr. Robert Bjork. Two TED Talks can be useful to help educate ourselves,
faculty, and our students – “Learning Styles and the Importance of Critical Self-Reflection” featuring Dr.
Tesia Marshik, and “Misconceptions of Learning Styles” featuring Dr. Anita Acai.
One method of making the myth-busting information about learning styles more palatable to students
(the idea is often reinforced in K-12 schooling, and to which students may staunchly cling to this part of
their identity) is to emphasize that although they may certainly have learning preferences, these do not
guarantee better learning. Indeed, the best advice is to “go wide” and incorporate as many modalities as
appropriate (Brown et al., 2014), considering the nature of to-be-learned material with the goal of
choosing a modality that makes sense (e.g., auditory mode for a music class, visual mode for an art
class). Ideally, higher education stakeholders – from ASCs to instructors to other student-facing staff –
can develop a consistent, fact-based framework to counteract misconceptions about learning styles.
Learning Strategy Information on Academic Support Center (ASC) Web

Sites
As a new initiative for this chapter, I worked with two research assistants to locate the ASC websites of
the original 400 higher education institutions contacted for the original survey study (McCabe, 2018).
After omitting three institutions that had either closed or did not have available websites, we assessed
each site using a qualitative coding scheme to determine the presence of information about general
learning support, specific evidence-based learning strategies (“SET”), language indicating to users that
they derived recommendations from cognitive science or memory research, and endorsement of
learning styles. The goal was to take a snapshot of the outward-facing web presence from these centers
with regard to evidence-based practices.
As displayed in Table 2, two categories of general learning support came up most frequently – learning
strategies/study skills and time management. Neither of these is particularly surprising given the survey
results discussed above (McCabe, 2018). Around half of the ASC sites claimed to support students in test
taking (often including management of test anxiety), note taking, and reading. Fewer sites described
goal setting, organization, or support specific to online learning. With regard to the last category, it is
notable that even 29.7% of ASC sites provided such advice, given that almost certainly this percentage
would have been quite small prior to the COVID-19 pandemic. This suggests that many ASCs gathered
and organized this specific content rather quickly in light of the emergency shift to distance learning in
March 2020. Other topics that were not part of our coding scheme, but we noticed arising multiple
times, included motivation, study environment, control of distractions, group work, writing, and
wellness/self-care. We only noted seven instances of use of “metacognition” (1.7%), an important but
relatively technical term referring to the ways in which learners understand their own learning and
make strategic decisions to improve.
346
Table 2
Academic Support Center (ASC) Web Site Content Analysis Results
Content Percentage
General Learning Support Categories
Learning Strategies/Study Skills 74.6%
Time Management 67.8%
Test Taking 52.9%
Note Taking 49.1%
Reading 44.6%
Goal Setting 32.5%
Organization 31.5%
Online/Distance Learning 29.7%
Specific “SET” Learning Strategy Categories
At least one: 27.7%
Elaboration 22.7%
Testing 20.7%
Spacing 17.4%
Learning styles 9.6%
Note. Percentages computed out of 397 websites and listed in descending order by sub-category.
Turning to the “SET” strategies, 27.7% of ASC sites included at least one specific evidence-based
recommendation from the domains discussed above. Strategies relating to the principles of spacing,
elaboration, and testing were presented at around the same rate, although elaboration (e.g., outlines,
mnemonics, asking questions, making connections) was highest, and spacing (e.g., short study sessions
with breaks, distribute studying over many days) was lowest. Additional analyses revealed that only
11.6% included recommendations related to all three “SET” strategies. We also coded for language
along the lines of the recommendations being “evidence-based” or “based on cognitive science
research” or “based on what scientists know about memory,” and only 13.4% met this criterion.
What about learning styles? Keeping in mind that 9% of ASC heads listed learning styles as a top-three
recommended strategy in my survey study (McCabe, 2018), it is interesting that learning styles were
mentioned on ASC web sites at almost exactly that same rate (9.6%). The good news is that
recommendations based on learning styles were found at a far lower rate compared to evidence-based
strategies. The not-so-encouraging news is that nearly one out of ten institutions we assessed still
endorse this concept; based on the presence of learning style inventories on many of these sites,
presumably they recommend that students find out their learning style and then adjust their study
strategies to match their style. A few interchanged the language of learning “styles” vs. “preferences,”
but, in my opinion, this still indicates to students that the discovery of their unique way of learning is
important in their study choices and, ultimately, to their academic outcomes. As discussed above, there
is a lack of evidence supporting learning styles (e.g., Pashler et al., 2009), so it is important for ASCs to
examine their sites for learning styles language and, ideally, replace it with recommendations supported
by empirical research.
347
Our approach to this website analysis was certainly coarse and cannot capture the entire nuanced
picture of what is happening at ASCs. Many ASCs may be recommending strategies not described on
their websites, and some of the sites appeared to have not been recently updated. We also found a few
institutions where content was password-protected, restricted to members of the institution.
Nonetheless, these data suggest some interesting patterns of overlap (and non-overlap) when it comes
to the types of supports and strategies offered to students.
Based on these analyses, I believe that ASCs could more consistently embrace the disbursement of
evidence-based learning recommendations on their public sites. This can be a component of what I
referred to earlier as a proactive approach to student support – promoting information relevant to
student success early and often to the entire community.
Academic Support Centers as Learning Strategy Ambassadors

ASCs are critical in bridging the gap between theory, evidence, and practice – that is, they are uniquely
situated to interpret and translate what memory research has demonstrated into education and, on
behalf of the common goal of everyone in higher education, to support students in maximizing learning
and academic achievement. After all, use of ASC services has been associated with GPA, graduation rate,
and academic skills (Grillo & Leist, 2013; Perin, 2004), and ASCs may be particularly supportive of
historically underserved and less-prepared students (Rheinheimer et al., 2010). In addition, combining
the use of a variety of strategies that work together in a manner that can potentially boost long-term
retention takes both knowledge and planning. Given that not all ‘desirably difficult’ strategies are
particularly intuitive or well-known to students and instructors alike (e.g., McCabe, 2011; Morehead et
al., 2016), ASCs have an even broader role in conveying the importance of understanding human
learning and memory as a basis for applying effective strategies.
That being said, not every strategy will benefit every student in every situation. Further, there are
certainly some known and as-yet-unknown boundary conditions for each strategy (e.g., some may be
better suited to specific course structures or topics). This is an important area of continued research. To
give two examples, one study demonstrated that undergraduates with ADHD showed an equivalent
testing effect (improved memory with increased retrieval from long-term memory) as those without
ADHD (Knouse et al., 2016), and another suggested that students with lower working memory had an
even larger testing effect benefit than those with higher working memory (Agarwal et al., 2017).
Is it true that where students have the “will” (in this case, possessing knowledge of the best study
strategies for each circumstance), they also have the “way”? Research and anecdotal experience suggest
not. Students may need support in finding the “way.” Some may not even really know what this thing
called “studying” is supposed to be. Ideally, ASCs can help students develop a ‘cognitive toolbox’
(Benassi, et al., 2014; Dunlosky, 2013) of effective techniques that can be strategically applied during
study sessions scheduled throughout the week. The ASC heads’ popular recommendation to focus on
time management skills (McCabe, 2018) aligns very well with the evidence-based strategies discussed
above. For example, spaced studying cannot happen without effective time management to allow for
shorter bouts of focused learning with breaks in between. Targeted support of time management skills,
therefore, will continue to play an appropriately central role in ASC interventions. All these supports will
ideally combine to improve students’ self-regulated learning, so that they can function (and are
motivated to do so) in learning environments without ready assistance. Chapters in Part 4 of this volume
address this issue in depth.
The call to ASCs to ensure that they possess the most up to date information regarding strategies that
research has shown that do (and do not) promote learning is essential. Using their student interaction
skills to both convey the information and convince students to change behaviors that do not promote
348
learning is key. Counteracting prevalent myths and misconceptions (e.g., learning styles, the value of re-
reading and underlining/highlighting) will likely need repeated and consistent messaging. Just as it is
said “it takes a village to raise a child,” I would say it takes a campus to empower a student with the best
tools for lifelong learning. ASCs can help advocate for institution-wide investment in the endorsement
and integration of effective strategies. Staff at ASCs also know the unique character and values of their
institution, along with the specific needs of the student population, so they are in a position to craft
messaging and interventions most likely to be positively received by and helpful to the students they
serve.
If you are reading this as an ASC staff member and you are unsure how to proceed with growing your
understanding of learning and memory research relevant to academic success, you have already found
one resource (this and other chapters in this e-book) to get started. I also recommend seeking out high-
quality media in which research is translated into everyday recommendations and actions, including
websites (e.g., The Learning Scientists; Lasting Learning; Retrieval Practice), books (e.g., Make It Stick by
Brown et al., 2014; How We Learn by Carey, 2014; Powerful Teaching by Agarwal & Bain, 2019; Small
Teaching by Lang, 2016), and other resources such as those mentioned above. And please consider
reaching out to your local cognitive psychologist, and more broadly, to the faculty in your psychology
and education departments, and staff in your teaching and learning centers. Speaking personally, it is
one of my favorite things to talk with people about how to communicate learning strategies research
effectively to students. Ideally, these conversations could even grow into a space for collaborative
research, such as designing and evaluating outcomes of innovative ASC initiatives. These are the types of
connections that will solidify the bridge from research to evidence-based practice.
Author Note
Thank you to Layla Murray and Alexander Steitz for assistance with Academic Support Center (ASC) web
site coding.
References
Agarwal, P., & Bain, P. (2019). Powerful teaching: Unleash the science of learning. San Francisco, CA:
Jossey-Bass.
Agarwal, P. K., Finley, J. R., Rose, N. S., & Roediger, H. L. (2017) Benefits from retrieval practice are
greater for students with lower working memory capacity, Memory, 25(6), 764-771.
http://dx.doi.org/10.1080/09658211.2016.1220579
Bellezza, F. (1996). Mnemonic methods to enhance storage and retrieval. In E. L. Bjork & R. A. Bjork
(Eds.), Memory: Handbook of perception and cognition (pp. 345-380). Academic Press.
https://doi.org/10.1016/B978-012102570-0/50012-4
Benassi, V.A., Overson, C. E., Tappin, E., Lee, M.O'Brien, E., Stiegler-Balfour, J. J. & Hakala, C. (2014).
Applying the science of learning: The Cognition Toolbox. In V.A. Benassi, C.E. Overson, & C.M.
Hakala (Eds.), Applying science of learning in education: Infusing psychological science into the
curriculum (pp. 194-205). Society for the Teaching of Psychology.
Bjork, R. A. (1994). Memory and metamemory considerations in the training of human beings. In J.
Metcalfe & A. Shimamura (Eds.), Metacognition: Knowing about knowing (pp. 185–205). MIT
Press.
349
The Belknap Press of Harvard University.
Carey, B. (2014). How we learn: The surprising truth about when, where, and why it happens. Random
House.
Educator, 37, 12-21.
psychology. Psychological Science in the Public Interest, 14(1), 4 –58.
Grillo, M. C., & Leist, C. W. (2013). Academic support as a predictor of retention to graduation: New
insights on the role of tutoring, learning assistance, and supplemental instruction. Journal of
College Student Retention: Research, Theory & Practice, 15(3), 387-408.
https://doi.org/10.2190/CS.15.3.e
Kirshner, P. A. (2017). Stop propagating the learning styles myth. Computers & Education, 106, 166-171.
https://doi.org/10.1016/j.compedu.2016.12.006
Knouse, L. E., Rawson, K. A., Vaughn, K. E., & Dunlosky, J. (2016). Does testing improve learning for
college students with attention-deficit/hyperactivity disorder? Clinical Psychological Science,
Kornell, N., & Bjork, R. A. (2008). Learning concepts and categories: Is spacing the "Enemy of Induction"?
Psychological Science, 19(6), 585-592. http://dx.doi.org/10.1111/j.1467-9280.2008.02127.x
Lang, J. M. (2016). Small teaching. Jossey-Bass.
Cognition, 39, 462-476. https://doi.org/10.3758/s13421-010-0035-2
McCabe, J. A. (2018). What learning strategies do academic support centers recommend to
undergraduates? Journal of Applied Research in Memory and Cognition, 7(1), 143-153.
McCabe, J. A., Friedman-Wheeler, D. G., Davis, S. R., & Pearce, J. (2021). SET for success: Targeted
instruction on learning strategies and behavior change in Introductory Psychology. Teaching of
McCabe, J. A., Osha, K. L., Roche, J. A., & Susser, J. A. (2013). Psychology students’ knowledge and use of
mnemonics. Teaching of Psychology, 40(3), 183-192.
strategies. Memory, 24(2), 257-271. https://doi.org/10.1080/09658211.2014.1001992
Paivio, A., Smythe, P. C., & Yuille, J. C. (1968). Imagery versus meaningfulness of nouns in paired-
associate learning. Canadian Journal of Psychology, 22(6), 427-441.
http://dx.doi.org/10.1037/h0082782
Psychological Science in the Public Interest, 9(3), 105-119. http://dx.doi.org/10.1111/j.1539-
6053.2009.01038.x
350
Perin, D. (2004). Remediation beyond developmental education: The use of learning assistance centers
to increase academic preparedness in community colleges. Community College Journal of
Research and Practice, 28(7), 559-582. http://dx.doi.org/10.1080/10668920490467224
Rheinheimer, D. C., Grace-Odeleye, B., Francois, G. E., & Kusorgbor, C. (2010). Tutoring: A support
strategy for at-risk students. Learning Assistance Review, 15(1), 23-34. Retrieved from
https://eric.ed.gov/?id=EJ886384
Roediger, H. L., & Karpicke, J. D. (2006). Test-enhanced learning: Taking memory tests improves long-
term retention. Psychological Science, 17(3), 249-255. http://dx.doi.org/10.1111/j.1467-
9280.2006.01693.x
Roediger, H. L. & Pyc, M. A. (2012). Inexpensive techniques to improve education: Applying cognitive
Cognition, 1(4), 242-248. http://dx.doi.org/10.1016/j.jarmac.2012.09.002
Rohrer, D., & Pashler, H. (2007). Increasing retention without increasing study time. Current Directions in
Psychological Science, 16(4), 183-186. http://dx.doi.org/10.1111/j.1467-8721.2007.00500.x
Slamecka, N. M., & Graf, P. (1978). The generation effect: Delineation of a phenomenon. Journal of
Experimental Psychology: Human Learning and Memory, 4(6), 592–604.
spaced study. Instructional Science, 41(2), 345-363. http://dx.doi.org/10.1007/s11251-012-
9231-8
Willingham, D. T., Hughes, E. M., & Dobolyi, D. G. (2015). The scientific status of learning styles theories.
Teaching of Psychology, 42(3), 266–271. https://doi.org/10.1177/0098628315589505
351
In Their Own Words Applying the Science of Learning: The Cognition
Toolbox at 15 Years and Counting
to Apply the Science of Learning in Victor A. Benassi, Catherine E. Overson, Lauren L.
Kordonowy, Elizabeth M. Tappin
Editors
Lauren L. Kordonowy
Victor A. Benassi

Michael C. Melville
Introduction
Relatively few faculty enter the profession of college and university teaching with much more than a
smattering of knowledge of the science of learning or of its applications in the courses they will teach,
never mind a deep understanding of the large and growing body of scholarship in this area (cf. Chew,
this volume; Rowe & Hattie, this volume). The amount and depth of understanding runs the gamut, from
new faculty who had no formal or informal instruction on the science of learning and its applications to
those who completed formal programs in college teaching that included instruction in science of
learning (for example, the University of New Hampshire Academic Program in College Teaching. We
hear anecdotally from many faculty that they modeled their early approaches to teaching based on
favorite teachers they had as students.
Staff of college and university teaching and learning centers provide an array of services to faculty,
graduate students, and staff involved in providing instruction to students with the goal of fostering their
academic performance. As such, these centers are in a unique position to work directly with teachers to
assist them in designing and implementing instruction informed by evidence-based practice, including
the science of learning.
In this chapter, we provide an update on the current status of a science of learning focused program
that operates within the University of New Hampshire’s (UNH) Center for Excellence and Innovation in
Teaching and Learning (CEITL). The program formally began in January 2007 and has grown
incrementally over the past 15 years. We provided our first description of the program in 2014 based on
the work we had completed to date (Benassi, Overson, et al., 2014).
Before proceeding, we offer a caveat. As noted elsewhere (Overson, Hakala et al., this volume;
Newcombe, this volume), the science of learning is a large field of study (e.g., Sawyer, 2014). As will
become clear in the remainder of this chapter, our work has focused on science of learning from a
cognitive perspective. Also, our applications with teachers and students have mostly, but not entirely,
centered on instructional interventions informed by cognitive principles of learning (Benassi, et al.,
2014). Examples of the instructional interventions our teacher collaborators have implemented in their
courses include:
• Test-enhanced Learning (Zang et al., this volume; Griswold et al., 2017)
• Self-Explanation (Chi & Boucher, this volume)
• Elaborative Interrogation (Pyburn et al., 2014)
• Distributed Practice and Interleaving of practice (Kang, this volume; Rohrer & Hartwig, this
volume)
• Worked Examples (Renkl, this volume)
352
• Multimedia Principles (Mayer, this volume)
• Successive Relearning (Dunlosky et al., this volume)
• Peer Instruction (Schell & Butler, this volume)
• Team-Based Learning (Collins et al., 2020)
• Teaching Cognitive Principles Directly to Students (chapters in Part 4, this volume)
• Cognitive Load Theory (Ashman & Sweller, this volume)
• Guiding Questions (Stiegler-Balfour & Benassi, 2015)
About CEITL at UNH

CEITL is a teaching and learning center situated within Engagement and Faculty Development in
Academic affairs. The Center has a full-time Director, Associate Director, Science of Learning Project
Coordinator, Student Learning Outcomes Assessment Coordinator, Learning Development and
Innovation (LDI) Manager, and three Learning Architects. CEITL also has an academic year 50%-time
Graduate Assistant (an advanced doctoral student). One or two grant-funded research associates are
also usually on staff. The Center Director coordinates programming with members of the CEITL Steering
Council and the CEITL Advisory Committee.
CEITL’s programming is fairly typical. Staff offer workshops, institutes, conferences, book clubs,
midcourse assessments, individual consultations with instructors, and more. The Center also oversees
two Academic Programs in College Teaching available to graduate students, faculty, and others, from
both UNH and beyond. Participants earn either a Cognate (minor for UNH graduate students) or a
Certificate in College Teaching. UNH faculty, graduate students, and staff can also earn CEITL
Participation Points for participation in CEITL programming, leading to different tiers of Certificates of
Participation. The LDI team offers an annual Faculty Instructional Technology Summer Institute. A
distinctive feature of CEITL programming is the Cognition Toolbox Program. The remainder of this
chapter focuses on this program, which has been funded by grants from the Davis Educational
Foundation.
Overview of Cognition Toolbox Program

Our grant work completed to date under the general heading of the Cognition Toolbox Program has
followed the same general process. We begin each of our grant foci by identifying topics, learning issues,
and instructional interventions that we are interested in exploring in the context of academic courses
(or, occasionally, in the context of a curriculum). Prior to academic terms, we advertise a list of potential
related projects to UNH course instructors. As an example, with our first grant, we were interested in
working with instructors on implementing practice quizzing and spacing/interleaving of study (among
other interventions).
Our project work with instructors follows a general pattern:
1. Exploration. Our early discussions focus on a learning issue(s) the instructor wishes to address.
Through these early meetings, we learn more about the course, such as course goals/objectives,
current learning activities, and the nature of major assessments. We then suggest one or two
potential cognitively-supported instructional interventions that both meet the instructor’s
learning issue and are consistent with their learning objectives. For example, if an instructor in
an introductory biology course wants students to learn and retain the definitions of key terms,
we might propose that they develop a regimen of frequent practice quizzing spread out over
multiple study sessions.
353
2. Design. The CEITL Director and Science of Learning Project Coordinator design the intervention
in the context of the instructor’s course. Our collaboration with the instructor is then focused on
implementation details with corresponding plans for a strong learning outcomes assessment
aligning with their course learning goals. We then work with the instructor to create exam
questions that target the content related to their instructional intervention (in the above
example, students’ mastery of key terms).
3. Deployment: A CEITL staff person works with the instructor to ensure the proper
implementation of the cognitive principle and assessment tool. Whether the project is an
implementation-only or formal research study, we always strongly encourage instructors to
assign ‘low stakes’ course credit for completing the project activities (e.g., completing assigned
quizzes on several consecutive days). We have found that if project activities are merely
‘encouraged’ or ‘optional,’ completion rates are consistently low. With making project activities
required, completion rates are high.
For instructional interventions or assessments administered online (e.g., via the Canvas©
learning management system), CEITL staff complete the necessary work to set up the protocol.
Contact between the CEITL staff person throughout the project can range from minimal to
considerable, depending on circumstances.
4. Assessment, Analysis and Follow-Up: CEITL’s assessment coordinator often conducts a
midcourse assessment to determine the extent to which students perceive the benefits from
the cognitive tool(s) used in their course. For information about our midcourse assessment
process, link to our Mid-Course Assessment Process.
As mentioned in the Design phase, we include an assessment protocol for each course project.
For our formal research projects, we design a research protocol to assess the impact of the
instructional intervention investigated in the course (e.g., see Griswold, this volume; Overson,
2014). In other instances, when an instructor consults with CEITL to address a learning issue that
results in an implementation project, we will work with them to develop course assessments
linked to the learning activities. If the instructor wishes, we meet with them to review course
learning outcomes and any plans for next steps.
Upon completion of a major assessment or course, we provide feedback to instructors in the
form of verbal and written reports. In some instances, we continue to work with a course
instructor during subsequent semesters. After we analyze the data, we often modify
instructional interventions for future course implementations.
The overall rationale for our work on the science of learning and full details on our approach are
described in Benassi, Overson, et al. (2014).
The Cognition Toolbox Project

CEITL began its Cognition Toolbox initiative in 2007 with initial grant support (2009 – 2012) from the
Davis Educational Foundation. The focus of that first effort was general in nature: assist faculty (and
others involved in fostering student learning) in implementing instructional interventions to improve
student learning, retention, and transfer of material in academic courses. Based on learning issues that
faculty wanted to address, we designed interventions grounded in principles from cognitive psychology
research in learning and memory. General details on the process we followed in working with faculty are
described above and examples of the projects that we undertook are provided in Benassi, Overson, et al.
(2014). Our overarching goal of this initiative was to design, implement, and evaluate the impact of the
Cognition Toolbox instructional interventions in a broad range of academic courses. We continue to
354
disseminate our work with UNH colleagues and with colleagues beyond the university (through offering
conferences, giving presentations at scholarly meetings, and meeting with individual faculty).
One project completed after the end of the Cognition Toolbox grant funding illustrates our sustainability
efforts to continue working with colleagues who want to incorporate instructional interventions in their
courses informed by the science of learning. A faculty member in Occupational Therapy collaborated
with CEITL on a study which examined the benefits of embedding quiz questions in online course-related
lecture presentations on midterm exam performance (Griswold et al., 2017). A discussion of that work
appears in Griswold (this volume). Additional published studies conducted with faculty include Collins et
al. (2020) and Pyburn et al. (2014).
Subsequent Projects
Since the completion of the Cognition Toolbox grant in 2012, we have received ongoing support from
the Davis Educational Foundation in the form of three consecutive grants: Teaching and Learning with
Multimedia; Applying the Science of Learning across the Biological Science Curriculum to Increase
Persistence and Enhance Early Engagement in STEM; The Student Cognition Toolbox. We next briefly
describe each project.
Teaching and Learning with Multimedia
Student learning from multimedia instruction is greatly affected by the way the instruction is structured.
You only have to think back to slide presentations through which you have had to suffer. These
presentations may have included densely populated slides, spoken and written words that conflict with
one another, etc. Our overall goal in the Teaching and Learning with Multimedia project (2012 – 2015)
was to design, implement, and evaluate the impact of powerful principles of multimedia instruction in a
broad range of undergraduate courses. Our work has been informed by Richard Mayer’s theory of, and
research on, learning with multimedia instruction. A systematic review of much of that work is included
in Mayer (this volume).
We have worked with dozens of faculty with a focus on improving their use of multimedia instruction. In
one project, Overson and Melville worked with a teacher of an introductory psychology course. They
applied the Multimedia Principle (Mayer, this volume) to PowerPoint© slide presentations viewed by
students as homework assignments. Some students viewed content presented through text-only slides
and others viewed slide content consistent with the Multimedia Principle. Based on prior research, we
predicted that students would score better on a post-presentation quiz when the slides were created
consistent with the Multimedia Principle—students learn better with pictures and words than words
alone. Results showed, in all comparisons, that students performed better, on average, on quizzes after
they viewed slides consistent with the Multimedia Principle.
Overson (2014) worked with four instructors, each of whom taught a different course: introduction to
justice studies, science and nature of stress, introduction to child life, and community and epidemiology.
These instructors had each taught their course previously using their standard set of PowerPoint© slide
show presentations, with one set of slides presented during each classroom lecture across the semester.
Prior to the start of the semester, Overson requested and received each instructor’s set of slide shows
for the semester. She then randomly selected half of the slide show presentations and modified them to
adhere as close as possible to Mayer’s multimedia principles of learning (Mayer, this volume). She
returned all of the slide shows (half modified and half original) to the instructors with the instructions to
teach their course in the usual manner using the slide shows. In Study 1, the instructors gave a quiz at
the end of each class period on the material covered during that session. In Study 2, students completed
questions on a major exam that were associated with either modified or unmodified slideshows. In both
355
studies, students scored higher, on average, on questions that were associated with the modified
slideshows that incorporated Mayer’s Multimedia Principle. Details on this project are provided in
Overson (2014).
Occasionally, graduate student teachers have collaborated with CEITL on designing and implementing
science of learning projects. The most ambitious of these projects to date was a dissertation conducted
by Dr. Michael Melville (PhD, Psychology). Melville completed a dissertation that examined Mayer’s
Personalization Principle in laboratory and course-level experiments.
Mayer (this volume) proposed that people learn better when words in a multimedia presentation
appear in a conversational rather than a more formal style. For example, instead of textual material
being presented in a formal, passive style, it could be presented in a conversational style by including
first-person and second person constructions (e.g., you, your, I).
Over the years, CEITL staff have worked with a UNH lecturer in Psychology (Dr. Kari Dudley) to assess the
impact of science of learning informed instructional practices in her large-enrollment introductory
psychology courses (over 200 students per course). Melville (2016) tested the personalization principle
in a section of her course. Students completed an online learning module outside of class. The module
consisted of two separate lessons, one on Gestalt psychology and one on signal detection theory. Each
lesson consisted of a series of slides that the students read at their own pace. Students were randomly
assigned to either the personalization condition or the non-personalization condition. Below is an
example of how the material was presented in the two conditions:
Non-personalization Condition Personalization Condition

There are certain cues that are used to Believe it or not, there are certain cues that you use
organize input from a stimulus. These cues to organize input from a stimulus whenever you
include figure/ground, similarity, proximity, see something. These cues include figure/ground,
good continuation, and closure, and are similarity, proximity, good continuation, and
known as Gestalt principles. closure. All of these cues that you use are known as
Gestalt principles.
As soon as students read the final slide in the series for each lesson, they answered a measure of their
perceptions of the lesson, and then they completed a 10-item multiple choice quiz. These questions
were transfer questions (Mayer, this volume) based on the material covered on the slides. Melville
found that the differences between the textual material (personalized vs. not personalized) was
associated with performance on the post-instruction quizzes. For the Gestalt lesson, the personalized
group scored higher on the quiz than the non-personalized group, d = .32. For the signal detection
theory lesson, there was the same predicted effect, d = .34. The results from this study showed that a
small change to otherwise identical reading material (personalization) produced improved student
performance on post-instruction quizzes relative to students who read non-personalized passages. For
material created by an instructor, it would take no more time to prepare personalized materials than
non-personalized materials. The boost in transfer of learning (as measured by the quizzes) is a benefit to
students. Full details on this study and other related studies can be found in Melville (2016, Experiment
4).
For the interested reader, additional information on the Teaching and Learning with Multimedia
initiative is available.
356
Applying the Science of Learning across the Biological Science Curriculum
Our next Davis Educational Foundation grant (2015 – 2018) initiative focused on applying the science of
learning in the biological sciences curriculum. One focus of this work involved collaborating with biology
faculty on improving performance in STEM gateway courses in the biological sciences. We will describe a
project that we completed several times in different sections of an introductory biology course.
The course instructor (in addition to previous instructors of the course) noticed that quite a few
students performed relatively poorly on the first major course exam, which focused on knowing the
meaning of key terms and understanding important concepts. We knew informally from instructors of
this course that many students reported trying to memorize the material that would be tested on an
exam, often right before the exam (cramming). The implementation of retrieval practice (quizzing) and
distributed practice (spacing) seemed to be a potentially impactful intervention to address the
instructor’s learning issue. We designed an intervention (based on Kornell’s 2009 laboratory study) for
the students enrolled in the course. They completed online flashcards (administered via Canvas©) on
four consecutive days leading up to a major course exam. Students responded to some of the items in
the deck multiple times on a given day (massed), but not on the other three days. Other items were
quizzed fewer times (spaced) each of the four days. Overall, students responded to the same number of
questions in the massed and spaced study conditions. We found that students performed over 15%
better on exam questions for content that was repeatedly quizzed across four days than material on
which they were quizzed on one day (even though the total number of exposures to the material was
the same in both conditions).
Our work on the STEM grant has had a number of positive additional benefits. We were successful in
recruiting four faculty from biology, chemistry, mathematics, and physics, who then created a series of
instructional videos that each showed the connections to a particular phenomenon among these four
disciplines (e.g., The Biology, Chemistry, Physics and Mathematics of the Nuclear Pore). We also
collaborated with three faculty on an early version of our Student Cognition Toolbox Program (Overson,
Kordonowy, et al., this volume). They recruited upper-division undergraduate honors students to serve
as study skills tutors for students enrolled in lower-division biology courses. What we learned from that
initiative led directly to the development of our Student Cognition Toolbox Program.
The Student Cognition Toolbox: Teaching Students Study Skills Informed by the Science
of Learning
Each of our previous grants focused on working directly with course instructors, who then implemented
(with our assistance) instructional interventions (in-class and/or online) designed to promote student
learning. In our Student Cognition Toolbox (SCT) grant (2018 – 2022), we developed and have been
providing an online instructional tool directly to students that is designed to inform them about how
they can use various study strategies to improve their academic performance. This ongoing work is
described fully in the chapter by Overson, Kordonowy, et al. (this volume).
Self-Directed Learning: Promoting Students’ Persistent Use of Effective Learning
Strategies
Our latest grant (2022 – 2024) follows directly from the previous one. In the SCT project, we
documented that students can be taught to develop effective learning strategies informed by the
science of learning. We found that students benefit on subsequent exams when these strategies are
used. However, when students are left to their own devices, many of them do not persist in using these
effective learning strategies. Thus, the challenge is to increase the likelihood that students
357
spontaneously apply them as part of their regular protocol for studying academic material. Our current
project addresses this challenge. Details on this work will be forthcoming as the project progresses.
Who Has Participated in Our Projects?

We reach out to course instructors from across the university, from all fields and disciplines. We connect
with them through a variety of means, as shown in Figure 1.
Figure 1
Forming Collaborations With Course Instructors
Faculty attend a
workshop which
leads to further
Faculty respond to collaboration
an announcement Department request
placed by CEITL in an for CEITL
area of work/grant presentation
interest
Faculty respond to
Faculty come to
CEITL website
CEITL with a Forming
identifying several
teaching/learning Collaborations potential project
question
possibilities
If an instructor has a learning issue that they want to address with assistance from CEITL, we do our best
to accommodate them. As a result, we have worked with instructors in a wide variety of courses on a
wide range of learning issues, representative course examples of which are shown in Table 1 below. This
table does not include the dozens of UNH instructors with whom we have worked on our Student
Cognition Toolbox (The courses they teach are listed in our chapter on the Student Cognition Toolbox;
Overson, Kordonowy et al., this volume).
Table 1
Examples of Course Collaborations Between Instructors and CEITL Staff
Health Care
American Energy and the Introduction to Principles of
Financial
Government Environment Physics Coaching
Management
Anatomy and English Introductory Psychology of
Health Economics
Physiology I & II Composition Anthropology Consciousness
Human Introductory Research
British Literature Epidemiology
Occupation Psychology Methods
358
Evolution, Science and
Human
Business Statistics Biodiversity and Justice Studies Nature of Human
Reproduction
Ecology Stress
Science and
Cardiopulmonary Introduction to
First Year Writing Macroeconomics Practice of
Pathology Child Life
Strength Training
Introduction to
Chemistry for General Molecular and
Economics Social Psychology
Engineers Chemistry Cellular Biology
(Macro)
Introduction to
Clinical
General Geographic Nanoscience in
Immunology and Soils Science
Microbiology Information Energy
Serology
Systems
Community Introduction to OT Therapy for
General Physics Zoology
Psychology Justice Studies Adults
Dissemination
Our efforts to share our experience and expertise regarding our applications of the science of learning in
education occur at two levels: at UNH and beyond UNH.
UNH
A considerable amount of CEITL’s programming is derived from knowledge and experience gained
through work on the grant initiatives described above. Below we list examples of how we continue our
science of learning work.
• Talk About Teaching Workshops. A brief scan of the workshops offered will show that a number
of them have a science of learning theme.
• January Workshops on College Teaching. Over the years, many of these half-day workshops have
focused on a science of learning intervention, often with participants working with workshop
presenters to develop a project they will implement in an upcoming course.
• Summer Pedagogy Institute. CEITL staff offer an annual institute during which they present
evidence-based teaching strategies (informed by science of learning). Participants arrive at the
institute with a course they want to work on with institute staff, with the goal of implementing a
project in an upcoming offering of the course. In addition, the Institute also presents on what it
is like to collaborate with CEITL on a classroom-based research project.
• Cognition, Teaching, and Learning Course. Each summer, CEITL offers a two-credit graduate
course on applying the science of learning in academic courses at the college and university
level. Most enrollees are UNH graduate students working toward earning the Cognate in College
Teaching, but UNH faculty and staff as well as others outside UNH complete the course.
• Fundamentals of Active Learning Course. CEITL’s LDI team leads an asynchronous online short
course on active learning, with an optional synchronous hands-on component. The course is
offered three times per year and is required for instructors who will teach using one of the
university’s Teaching-Enhanced Active Learning classrooms.
• Themed Special Interest Group (T-SIG). T-SIGs are semester-long opportunities for faculty,
graduate students, and staff. Small cohorts meet four times during the semester to discuss a
topic related to teaching and learning and perhaps develop a project they will implement in an
359
academic course. For example, a recent T-SIG focused on learning assessment techniques
(Barkley & Major, 2016). Creating assessment tools that align with learning outcomes is a key
component of the instructional protocols that we design for use by instructors.
• CEITL Book Club. The book club members read and discuss a current book—for example, Brown,
et al., Make It Stick (2014)—on a topic selected by a CEITL staff member who organizes and
leads the club.
Beyond UNH
We disseminate beyond UNH what we have learned through our work in three main ways—
presentations at regional, national, and international conferences, site visits to other colleges and
universities, and conferences that we develop and host. Over the past 15 years, we have broadly
disseminated information and recommendations regarding our approach to apply the science of
learning in higher education.
• Publications. CEITL staff and faculty collaborators have published articles and book chapters
over the years that relate to our Cognition Toolbox work (e.g., Collins et al., 2020; Griswold, et
al., 2017; Overson & Benassi, 2021; Seidel, et al., 2008; Stiegler-Balfour & Benassi, 2015).
• Conference Presentations. CEITL staff have given presentations at a wide range of conferences,
institutes, and workshops—for example, American Psychological Association; Annual Lilly
Conference on College Teaching – West; Society for the Teaching of Psychology Conference on
Teaching; Association for Psychological Science; Psychology One Conference (Stanford
University); EdCog Conference (McMaster University); National Institute on the Teaching of
Psychology; New England Association of Schools and Colleges; New England Psychological
Association; Open Learning Initiative Community (Carnegie Mellon University).
• Visits to Regional Colleges and Universities. Beginning with our first grant (2009 – 2012), we
have visited colleges and universities across the New England region to share our work and
recommendations with colleagues (e.g., Clark University, Lesley University, Massachusetts
College of Pharmacy and Health Sciences, Merrimack College, Middlebury College, Springfield
College, University of New England, U.S. Naval War College, Wheaton College). These visits have
ranged from discussions among a small group of faculty to a keynote presentation given to over
250 faculty and administrators.
• Hosting Conferences. With each of our four Davis Educational Foundation grants, we have
hosted two one-day conferences (followed by an intensive hands-on workshop during two of
these conferences for selected participants). In addition to highlighting our own work at these
events, we invited internationally known science of learning scholars to present their work
related to the conference themes. All but the two most recent conferences were held on the
UNH campus. Participants included faculty, teaching and learning staff, administrators, and
others from institutions located across the New England region of the USA. Due to the COVID-19
pandemic, our two Student Cognition Toolbox grant conferences were held as Zoom Webinars.
A major benefit resulting from these circumstances was that we were able to invite a large
number of people to participate in the conferences. Our advertisements resulted in over 1,000
and 900 registrants, respectively, for the two Webinars from across the USA and from several
dozen other countries. The conference presentations are accessible at the CEITL website.
• Sharing resources. CEITL makes available on its website a variety of resources that are available
to people from UNH and to others external to the university. The Teaching and Learning
Resource Hub is a recent addition.
360
Further details on these activities and resources are available on various webpages at the CEITL website
• https://www.unh.edu/professional-success/ceitl/science-learning-grants
• https://www.unh.edu/professional-success/ceitl
Conclusion
Now that we have a decade and a half of experience working with course instructors to create and
implement instructional interventions informed by the science of learning, we can speak to a concern
that we addressed in our first update on the Cognition Toolbox program:
. . . most of these [science of learning] principles have a strong base of support in tightly
controlled laboratory experiments. Few of them have been systematically examined in actual
academic courses (Dunlosky, Rawson, Marsh, Nathan, & Willingham, 2013). Daniel (2012)
recently argued that much additional course-level translational research is needed before
“promising principles” should be promoted to teachers as “best practices” (p. 251). Our work at
UNH over the past six years has focused on evaluating a variety of science of learning principles
in a wide range of academic disciplines and course formats.” (Benassi, Overson, et al., 2014, p.
205).
Both Chew (this volume) and Koedinger et al. (this volume) have raised similar issues.
We have completed dozens of informal and formal collaborative projects with course instructors in
authentic academic courses. For projects in which we have collected assessment of impact data, most
have provided results that support the efficacy of the instructional interventions we have implemented.
In some projects, we did not find the desired effect, but even in those cases, we invariably learned
something that could be applied in a further investigation. For example, in a project during the early
days of our program, a CEITL staff member worked with a faculty member who was interested in
improving students’ performance on fact-based exam questions. They designed an intervention in which
some content would be quizzed leading up to a major exam and other content would be reread. The
staff member let the teacher create two conditions of study, assuming the teacher would randomly
select which content was quizzed and which content was reread. On the exam there was no difference
between the two conditions. This result was surprising, as we had found reliable testing effects in other
projects. During an informal chat one day, the teacher mentioned in passing that, in preparing for our
project, he had examined previous exam results in his course and selected the items that students often
answered incorrectly for inclusion in the quizzing condition; the items in the reread condition were
those items that most students answered correctly. Finally, we had the answer. There was a testing
effect! The practice quizzing boosted performance on questions that had previously been answered
incorrectly by many students. This lesson led us to ensure that we would attend to and monitor all
aspects of a project, including implementation and assessment.
Our work over the past 15 years offers support for the assertion that teaching and learning centers can
play an important role in helping faculty to develop and provide instruction that is informed by the
science of learning. Programs such as the Cognition Toolbox are not built without resources, staff
expertise, and adequate funding. The Center’s director, associate director, science of learning project
coordinator, and research associates have engaged in ongoing professional development in the form of
working on the Cognition Toolbox projects, attending and participating in professional conferences,
publishing in scholarly publications, and preparing for and leading workshops and presentations (see
also the approaches developed and implemented by Lovett et al. [this volume] at Carnegie Mellon
University and by Kuzmick et al. [this volume] at Harvard University).
361
Depending on staffing, resources, and interest, a center’s staff may develop a smaller or larger focus on
science of learning initiatives. Regarding funding, we would not have been able to build the Cognition
Toolbox program to its present level without the support of the Davis Educational Foundation. That
support has allowed us to devote staff time toward accomplishing grant-specific goals, share what we
have learned with colleagues at UNH and beyond, and create the structures and protocols to continue
after grant funding ends. At the same time, with a little ingenuity and consideration of current resource
deployment even a small center could be refocused to include a science of learning component. The
opportunities are there. If our experience over the past decade and a half is any indication of what to
expect, the direct and indirect benefits to faculty and students, respectively, could be substantial.
Author Note
Our work has been supported by grants from the Davis Educational Foundation (The Cognition Toolbox
(2009 – 2012); Teaching and Learning with Multimedia (2012 – 2015); Applying the Science of Learning
across the Biological Science Curriculum to Increase Persistence and Enhance Early Engagement in STEM
(2015 – 2018); The Student Cognition Toolbox: Teaching Students Study Skills Informed by the Science of
Learning (2018 – 2022); Self-Directed Learning: Promoting Students’ Persistent Use of Effective Learning
Strategies (2022 – 2024). The Foundation was established by Stanton and Elisabeth Davis after Mr.
Davis's retirement as chairman of Shaw’s Supermarkets, Inc. The support of the Foundation has been
key to the successful development, implementation, assessment, and dissemination of our work. Leanne
Greeley Bond (Director of Grants and Programs) has worked with us on all of our Davis Education
Foundation grants. We thank her for her support and guidance.
We also acknowledge the ongoing support of the UNH Office of the Provost and Vice President for
Academic Affairs, University of New Hampshire. We also recognize with appreciation the contributions
to our work of many UNH faculty, staff, and students.
Dr. Michael Melville (Carnegie Mellon University) participated in this work when he was first a doctoral
student and then a Research Associate in CEITL.
References
Barkley, E. F., & Major, C. H. (2016). Learning assessment techniques: A handbook for college faculty.
Jossey-Bass.
Benassi, V.A., Overson, C. E., & Hakala, C. (Eds.) (2014). Applying science of learning in education:
Benassi, V.A., Overson, C. E., Tappin, E., Lee, M. O'Brien, E. J., B. P. White, Stiegler-Balfour, J. J. & Hakala,
C. (2014). Applying the science of learning: The Cognition Toolbox. In V.A. Benassi, C.E. Overson,
& C.M. Hakala (Eds.), Applying science of learning in education: Infusing psychological science
into the curriculum (pp. 194-205). Society for the Teaching of Psychology.
Brown, P.C., Roediger, H. L, & McDaniel, M.A. (2014). Make it stick: The science of successful learning.
Harvard University Press.
Collins, D. P., Rasco, D., & Benassi, V. A. (2018). Test-Enhanced learning: Does deeper processing on
quizzes benefit exam performance? Teaching of Psychology, 45(3), 235–238.
362
Collins, K. E., Overson, C. E., & Benassi, V. A. (2020). Team-Based Learning in a Coaching Education
Course: Impact on Student Learning. Journal of Teaching in Physical Education, 39(1), 28–35.
https://doi.org/10.1123/jtpe.2018-0223
Griswold, L. A., Overson, C. E., & Benassi, V. A. (2017). Embedding questions during online lecture
capture to promote learning and transfer of knowledge. American Journal of Occupational
Therapy, 71(3), 1-7. https://doi.org/10.5014/ajot.2017.023374
Kornell, N. (2009). Optimising learning using flashcards: Spacing is more effective than
cramming. Applied Cognitive Psychology, 23(9), 1297–1317. https://doi.org/10.1002/acp.1537
Melville, M. C., The role of social agency in student learning. (2016). Doctoral Dissertations.
1361. https://scholars.unh.edu/dissertation/1361
Overson, C. E. (2014). Applying multimedia principles to slide shows for academic presentation. In V. A.
Benassi, C. E. Overson, & C. M. Hakala (Eds.), Applying science of learning in education: Infusing
psychological science into the curriculum (pp. 252 – 258). Society for the Teaching of Psychology.
Overson, C. E., & Benassi, V. A. (2021). Backward design, the science of learning, and the assessment of
student learning. In S. A. Nolan, C. M. Hakala, & R. E. Landrum (Eds.), Assessing undergraduate
learning in psychology: Strategies for measuring and improving student performance (pp. 95–
107). American Psychological Association. https://doi.org/10.1037/0000183-008
Pyburn, D. T., Pazicni, S., Benassi, V. A., & Tappin, E. E. (2014). The testing effect: An intervention on
behalf of low-skilled comprehenders in general chemistry. Journal of Chemical Education.
91(12), 2045-2057. http://pubs.acs.org/doi/pdfplus/10.1021/ed4009045
Sawyer, R. K. (Ed.). (2014). Cambridge handbook of the learning sciences (2nd ed.). Cambridge University
Press. https://doi.org/10.1017/CBO9781139519526
Seidel, L. F., Benassi, V. A., & Lewis, J. (2008). The Testing Effect: Teaching to Enhance Learning in Health
Administration Education. Journal of Health Administration Education, 25(1), 63-72.
PMID: 19655619
Stiegler-Balfour, J. J., & Benassi, V. A. (2015). Guiding questions promote learning of expository text for
less-skilled readers. Scholarship of Teaching and Learning in Psychology, 1(4), 312-325.
363
In Their Own Words
Part 4
Preparing Students to Apply the
Science of Learning
Editors
Lauren L. Kordonowy
Victor A. Benassi

364
In Their Own Words How to Teach Powerful Strategies so That
Students Self-Regulate Their Use: The KBCP
Framework
Mark A. McDaniel
Editors
Washington University in St. Louis
Lauren L. Kordonowy
Victor A. Benassi

Gilles O. Einstein
Furman University
In many ways, this is an exciting time to be a student. A plethora of engaging materials are available to
stimulate students’ thinking and to capture their interest. For example, elementary, middle school, and
even college students can learn about radiocarbon dating by reading instructional comic books (Hunt &
Swogger, 2016). Students can interact with a teachable agent, Betty’s Brain (a computer program), to
learn neuroscience; the student teaches “Betty” concepts about her brain and then poses questions to
determine if Betty accurately understands those neuroscience concepts (Leelawong & Biswas, 2008).
College students can enroll in flipped classrooms in which cooperative learning occurs in small
discussion groups within the context of a larger class (e.g., Talanquer & Pollard, 2010). Yet, being a
student may also be more challenging than ever. We hear students complain that their courses are
crammed with information that they are expected to memorize, learn, and master. The situation is that
teachers are working assiduously to design state-of-the-art courses, but once students leave the
classroom, they are largely responsible for figuring out how to learn the information presented in their
classes. From our perspective, this state of affairs is troubling because research shows that students do
not have a great understanding of how their learning and memory work. Indeed, middle- and high-
school students (Agarwal et al., 2014), as well as college students (Dunlosky et al., 2013; Karpicke et al.,
2009; Kornell, 2009; Putnam et al., 2016), often rely on ineffective learning strategies.
Why Do Most Students Not Use Optimal Learning Strategies?

Given the extensive experience that students have studying for and taking tests in school, you might
think that they would eventually learn which study strategies are effective and which are not. Research
shows, however, that relying on personal experience can be a very difficult way to discover powerful
learning strategies.
Most memory scientists (e.g., Koriat, 1997; Kirk-Johnson et al., 2019) believe that we do not have direct
access to the strength or durability of the memories that we are forming. That is, when learning
something new, we do not inherently know whether it has been sufficiently stored in our brain to
enable us to successfully access and use that information in the future. Instead, we rely on a variety of
cues and guesses to infer whether we are likely to remember that information later on, and many
studies have shown that our inferences are often incorrect.
One factor that undermines our ability to determine whether or not a learning strategy is powerful is
that we tend to use immediate retrieval (i.e., whether we can recall something immediately after
studying it) to judge the effectiveness of a strategy. The problem here is that immediate judgments of
learning are often inaccurate predictors of good long–term retention (Bui et al., 2017; Dunlosky &
Nelson, 1994; Dunlosky & Tauber, 2016), and nearly all study strategies, including relatively poor ones
365
like rote repetition, produce good immediate memory. Thus, these immediate judgements of memory
can lead us to embrace learning strategies that are relatively ineffective (e.g., Shaughnessy, 1981).
We also have biases that undermine our ability to accurately judge the effectiveness of learning
strategies. For example, learners tend to assume that effortful strategies (those that seem to require
high mental effort during study) must not be very effective (Kirk-Johnson et al., 2019). The idea is that if
the strategy does not feel fluent, we must not be learning much. The researchers asked participants to
learn some bird categories using blocked presentation (presenting all of the instances from one category
before presenting instances from another category) and some using interleaved presentation (randomly
mixing up the presentation of instances from the different categories). Participants correctly judged the
interleaved presentation to be more effortful but incorrectly judged it to be the less effective learning
strategy. Importantly, participants were also less likely to choose to use this more powerful learning
strategy for subsequent study. Because many powerful learning strategies (strategies that produce good
long-term learning) are relatively effortful (Bjork, 1994), this “misinterpreted-effort hypothesis” can lead
us astray.
Another factor that interferes with students’ ability to rely on personal experience to determine which
strategies produce the best long-term learning is that we are unable to control all of the relevant
variables that affect performance. Thus, under certain circumstances (e.g., an easy test), using relatively
poor learning strategies like rote repetition can lead to good performance. This is likely what happened
when the 8th-grade daughter of one of the authors confidently told her dad that she had discovered that
she scores better on tests when she doesn’t study. Conversely, in some situations—for instance, taking a
very difficult test—even using an effective learning strategy may not lead to high performance. This
inability to conduct controlled experiments with our study strategies severely limits our ability to rely on
personal experience to figure out which study strategies maximize learning.
Compounding the problem, even if learners are aware that the strategy is ineffective, they can persist in
using the ineffective strategy for the expediency of getting the job done (Garner, 1990). Indeed,
according to Hattie et al. (1996), “It is very difficult to change the study skills that students have
acquired … and older students [college] are more resistant to change” (p. 122). Thus, it is not surprising
that “many students are committed to ineffective strategies” (Pressley et al., 1989, p. 302; see also
Dunlosky et al., 2013).
In conjunction with the challenges of relying on personal experience to develop effective learning
strategies, students unfortunately receive little or no comprehensive instruction in learning strategies
(e.g., as noted in self-reports, Hartwig & Dunlosky, 2012; see also Dunlosky et al., 2013; Pomerance et
al., 2016). Educational and cognitive psychologists have identified key study strategies that have
substantial effects on learning and long-term retention (Dunlosky et al., 2013). These strategies have
been shown to have considerable utility across academic domains and across individuals, and yet, most
students will not likely develop such optimal study strategies on their own (e.g., Een, 2021) nor will most
students receive instruction in how to learn. Accordingly, we have advocated that learning-strategy
training should be incorporated into the curriculum (McDaniel & Einstein, 2020). Here we provide a
concrete framework for doing so, develop how the components of the framework meet several
challenges for getting students to broadly apply effective learning strategies in school, and describe two
classroom examples of learning-strategy training guided by our framework.
366
The Knowledge, Believe, Commitment, and Planning (KBCP)
Framework
Table 1 provides an overview of the framework. The first component, knowledge about the strategy and
how to use it, is a self-evident feature of strategy training. The knowledge component includes
conveying knowledge about the strategy (what the strategy is) and knowledge about how to implement
the strategy for the students’ learning tasks (how to apply). When study strategies are taught, albeit
rarely, the training typically tends to rely exclusively on these knowledge components (but there are
recent exceptions—see chapters in this volume, as well as Bernacki et al., 2020; Cogliano et al., 2021;
McCabe et al., 2021). Efforts focusing on knowledge alone have typically been disappointing in that
students will generally not sustain use of the instructed strategies or transfer them to appropriate new
learning tasks (e.g., Borkowski et al., 1987; McDaniel & Einstein, 2020). Our first critical take-home
message then is that strategy training that focuses exclusively on telling students about particular
learning strategies may not be especially effective for a large number of students. Additional training
components are needed.
We suggest that students are more likely to transfer strategies to their own study practices when they
directly experience the benefits of strategy use. That is, a second important component of strategy
training is to ensure that students believe that the strategies are effective for them. The scientific
evidence indicates, however, that such belief can be difficult to instill. When students are instructed in a
strategy, presented research evidence that it works, and even use the strategy in a practice learning
task, students are still reluctant to endorse the strategy for their own use (Yan et al., 2016). Yan et al.
(2016) concluded that students have an “eagerness to believe that one is unique as a learner-- that what
enhances others’ learning differs from what enhances one’s own learning” (p. 918) and that learners
were most likely to endorse a strategy if the learner experienced the strategy’s benefits (relative to
trying to learn the material with their usual strategy). The idea here is that the direct experience has a
powerful influence on belief in the strategy’s effectiveness. As we develop in the next section,
participatory demonstrations can be an extremely useful method to provide that direct experience (and
hence, belief) to students.
Students can believe that an effective strategy works for them but may lack the motivation to use the
strategy. We are certain that teachers have heard students complain that they do not like to study, and
one teacher told us that in her middle school, students have the idea that only poor students need to
study. Thus, the third component is to foster students’ commitment to use the strategy. There are
various methods that could be used to stimulate commitment, including operant conditioning—having
the student set up rewards for studying (McCabe et al., 2021), making strategy use a class assignment
(McDaniel et al., 2021), and increasing the student’s perceived value of the positive outcomes
associated with effective strategy use (see McDaniel et al., 2020). We amplify on these two latter
methods below.
Finally, there is a voluminous social/motivational literature establishing that commitment is not
sufficient for effective follow-through (e.g., Gollwitzer, 1999). In terms of learning strategies, for
instance, students can intend to space their studying over the days before an exam; however, the
demands of daily life and school can derail this commitment (Susser & McCabe, 2013). Accordingly, we
include a fourth component in our strategy-training framework: teaching students to form an action
plan for implementing the strategy in their school learning. Research has shown that such action
planning effectively helps even young students (e.g., fifth graders) in their classroom learning
(Duckworth et al., 2013).
367
Table 1
Overview of the Knowledge, Belief, Commitment, Planning (KBCP) Framework
KBCP Component Function

Students must understand the strategy, the evidence behind its effectiveness,
Knowledge
and how to apply the strategy to their educational demands.
Students must be convinced that the strategy works for them. Students need to
Belief appreciate the relationship between their strategy use and learning outcomes,
thereby giving them a sense of self-efficacy over their learning outcomes.
Students must be sufficiently committed to using the strategy. Commitment
Commitment helps students exert the effort required to implement and adopt the strategy to
their coursework.
Students must have a personalized blueprint for how to incorporate the strategy
Planning
into their studying. This also may increase accountability for using the strategy.
Examples of How to Teach Learning Strategies Using the KBCP

Framework in a Classroom Setting
In this section, we describe how one could teach students to use an effective study strategy for self-
regulated learning using the KBCP framework. We want to underscore that there are many ways to
implement the central components of the KBCP framework. As such, we present two possible concrete
examples, one a brief training protocol and one a more extended semester-long course.
A Brief Training Module
Assume that we want to teach students to use a self-explanation study strategy and that we can allot
parts of two class sessions out of the normal curriculum. Such a strategy module might be useful in
science, social studies, history, and mathematics courses, among others. Self-explanation involves
learners generating meaningful explanations for the material they are reading or studying. This is
thought to enhance learning and retention by encouraging integration of new learning with one’s
knowledge base (Tse et al., 2007). Laboratory and classroom research show that when students use the
strategy of generating understanding or self-explanations, they substantially enhance their learning and
memory with a wide variety of materials, including mathematical and reasoning problems, and across
content areas (see Dunlosky et al., 2013, and Pashler et al., 2007, for reviews).
Knowledge (Session 1—About 15 Minutes)
One could initially address the knowledge component by presenting a lecture that emphasizes general
points about strategy and learning outcomes, such as that good learners are those who use good
strategies and that students often use ineffective strategies and often do not use effective strategies
(e.g., Dunlosky et al., 20013, Karpicke, et al., 2009; Kornell, 2009). The teacher could then make the case
that that deep understanding of material is critical for good learning and retention and that self-
explanation is a great strategy for generating understanding. Next, the teacher could illustrate these
points with the methods and results of several relevant studies (e.g., Bransford & Johnson, 1973; Seifert,
1994).
368
Belief (Session 1—About 10 Minutes)
Our KBCP framework assumes that students are more likely to transfer strategies to their own learning
objectives when they also directly experience the benefits of using a particular strategy. The idea here is
that the direct experience has a potent influence on belief in the strategy’s effectiveness. A powerful
demonstration that we have used to instill belief in the usefulness of self-explanation is based on the
research of Stein et al. (1982; see also Pressley et al., 1987). This demonstration involves having
students first learn a set of materials using their own strategy (or a commonly used strategy that clearly
does not involve understanding like rote repetition) and then learn an equivalent set of materials using
the self-explanation strategy. Students are tested after each set of materials, and then receive feedback
on their test performance in each condition. The materials and instructions for implementing this
demonstration are presented in the Appendix. (A teacher with whom we shared the demonstration
found it successful with students as young as 5th grade.)
More Knowledge (Session 2—About 20 Minutes)
As described earlier, the knowledge component includes not only knowledge about what the strategy is,
but also knowledge about how to apply the strategy to authentic learning tasks. After refreshing
students’ memory of the lecture and demonstration in the previous session, teachers could ask students
to practice using the self-explanation strategy with actual course material. This could be done using the
next assigned reading for the class. Specifically, you could ask all the students to read a designated
section of the assignment (perhaps the material under the first subheading) and then to stop and ask
themselves “What is the gist of this section?” and “What new information have I learned and how does
it relate to what I already know?” In the beginning, it may be helpful to ask the students to write down
their explanations and to share their explanations and implications with each other. Students can then
practice using this strategy with several more sections of the reading and then be encouraged to use this
powerful learning strategy for all of their learning materials.
Commitment (Session 2—About 10 Minutes)
Augmenting students’ commitment to using the self-explanation strategy for students’ schoolwork can
be fostered with a utility value intervention, which has been supported by a large number of
randomized controlled studies (see Harackiewicz & Priniski, 2018, for a review). This intervention helps
enhance motivation for changing behavior (i.e., using an effortful new learning strategy) by increasing
the perceived value of engaging in an activity (e.g., using the self-explanation strategy will help me
achieve my short- and long-term goals of making good grades and getting a better job or getting into
graduate school). Utility value interventions have repeatedly been shown to be associated with
continued motivation and persistence (Wigfield & Cambria, 2010). A utility value intervention could
involve asking students to write an essay (e.g., two paragraphs) that addresses (1) how using the self-
explanation learning strategy will influence their learning of their course material, and (2) how their
learning in this course will influence their life (cf. Hulleman et al., 2017).
Planning (Session 2—About 10 Minutes)
Successfully following through and actually using the self-explanation strategy while studying involves
overcoming vulnerabilities to things like distractions and tendencies to use routine study techniques,
which could derail one’s plan to use the strategy. To help ensure that students think of using the self-
explanation strategy at the appropriate times, you can encourage them to create implementation
intentions. Implementation intentions take the form of “When situation x arises, I will perform response
y” (Gollwitzer, 1999). The key principle underlying the use of an implementation intention is to
associatively link situational cues with intended actions, such that encountering the situational cues will
369
trigger retrieval of the intended actions. For example, instead of forming the general intention to use a
self-explanation strategy, using implementation intentions will force students to think through exactly
when, where, and how they will use the strategy (e.g., in the evening in the library, when I read my
textbook for this course, I will use a self-explanation strategy). You should ask students to imagine their
future study opportunities and use the form, “When I __________, I will _________,” to write out
specific and concrete action plans for using the self-explanation strategy.
A Semester-Long Course
One of us designed and taught a course for college students based on the KBCP framework, titled
“Applying the Science of Learning” (McDaniel et al., 2021; we have also talked with several high school
teachers about implementing a similar course). The course was a small-enrollment course (about a
dozen students) and was designed for first-year students and sophomores. It covered a set of strategies
that we judged could be applied to an array of our students’ college courses (e.g., recalling to learn,
generating understanding/self-explanation, organization, mnemonics, spacing study, and intermixing
study of similar concepts). Each strategy was taught in turn, with the KBCP components integrated into
the instructional approach. For each strategy, knowledge about the strategy was conveyed through
lectures, readings, and pertinent review of the evidence supporting the strategy’s effectiveness.
To instill belief, for each strategy we designed an in-class demonstration that included the following
elements. Every student learned some material with their usual learning strategy and learned other
material with the new strategy. Additionally, across the two learning episodes, critical dimensions were
kept constant: the nature of the materials, the study time, the retention interval, and the test task (see
the Appendix for one of the demonstrations). Students then shared how the new strategy worked
relative to their usual strategy, their experience with each strategy, and their reflections on the
demonstration. These discussions and reflections reinforced for students (and the instructor) the power
of the strategies. For instance, a typical reaction from students after the recall-to-learn demonstration
(attempt to recall a text after reading it vs. rereading the text) was, “I wouldn’t have appreciated the
power of recall for learning—my experience convinced me how much more I could remember after
attempting recall than after rereading.” After the mnemonics demonstration, one student reflected that
he felt that using imagery encoding to try to learn definitions of unknown words (the keyword
mnemonic) was only producing fleeting memories. He was astonished to find that on the test, upon
identifying the keyword (from the vocabulary item), the image popped into mind. He directly
experienced that imagery supports good recall, thereby countering his subjective impression (and
reinforcing for the students in the class that subjective impressions of learning can be extremely
misleading).
Commitment and planning were accomplished through a course requirement. All students were
required to apply each strategy to at least one other course for which they were enrolled that semester.
Specifically, we concluded each strategy “unit” with a “learning project assignment,” in which students
constructed a plan that specified the course(s) in which the strategy would be applied, the learning
objective associated with the strategy (e.g., improve learning for the next exam; improve understanding
of the course material), and the details of when and how the strategy would be applied. After turning in
the plan, each student attempted to implement their plan. A week or two was allowed for the students
to apply the strategy per their plan. Students then reflected on the success of the strategy through in-
class discussion and written reflection (part 2 of the learning project assignment).
To get a sense of whether this college course changed students’ approach to learning and studying, and
if so how, we administered a learning-strategies survey on the first and last days of class (available from
the authors upon request). Table 2 displays the change in self-reported learning strategy use over the
370
semester for strategies that for many students are implemented in a relatively superficial way (i.e., little
generation of new connections, insights, and elaboration), and thus are relatively ineffective for forging
good retention and deep understanding of material. At the outset of the semester rereading the
textbook and rereading and rewriting notes were reported to be used in the range of “sometimes” to
“often.” By the end of the semester, there was a notable drop in the frequency in which these relatively
passive strategies tended to be used (between “rarely” and “sometimes” on average).
Table 2
Reported Use of Non-generative (“Superficial”) Strategies at the Beginning and the End of the Learning
Strategies Course
Pre-Course Post-Course
Read slides 3.4 3.1
Reread textbook 3.6 2.3**
Reread notes 4.2 2.5**
Rewrite notes 3.1 2.5*
Rehearse/memorize 3.4 3.2
Note. 1 = never; 2 = rarely; 3 = sometimes; 4 = often; 5 = very often
*p < .10; ** p < .05
By clear contrast, as shown in Table 3, students reported substantially increasing their use of a range of
effective (generative, elaborative) strategies. That is, the students embraced the evidence-based
strategies covered in the course and appeared to be replacing their relatively ineffective strategies with
more powerful and potent learning strategies.
Table 3
Reported Use of Generative (“active”) Strategies at the Beginning and the End of the Learning Strategies
Course
Pre-Course Post-Course
Drawing 1.8 2.2**
Use Flashcards 2.3 3.2**
Generate 2.9 3.7*
Teaching 3.0 3.8*
Outline 2.0 3.9**
Mnemonics 2.8 4.0**
Self-quizzing 3.1 4.6**
Self-explanation 3.5 4.5**
Note: 1 = never; 2 = rarely; 3 = sometimes; 4 = often; 5 = very often
*p < .10; ** p < .05
Reinforcing these quantitative results are a handful of comments that some students volunteered to the
instructor after they completed the course. The following semester, one student came into the office of
one of the authors in a very upbeat mood, saying that “I’m now confident that I can meet any challenge
371
presented in my classes…I no longer waste time on ineffective study techniques and use my time better
with effective study techniques.” Several years after completing the course, another student e-mailed “I
just wanted to let you know that I've been using spaced studying religiously over these past semesters
and it's been working beautifully.”
Summary
Given the problems inherent in using personal experience to discover optimal learning strategies and
given the lack of systematic training of effective strategies in the educational system, it is not surprising
that students at all levels often develop and use ineffective learning strategies. At first blush, it might
seem that training students to use more effective learning strategies is a simple matter of exposing
them to the strategies and how to use them (i.e., the knowledge component). Extensive research in the
education and psychology literatures, however, suggests that a multidimensional approach may be
needed to maximize the likelihood that students embrace the trained strategies and apply them to their
learning challenges. Our KBCP framework proposes four components, knowledge, belief, commitment,
and planning, that we believe are important for changing behavior and getting students to use new and
powerful strategies in a self-regulated manner.
Corresponding Author:
Mark A. McDaniel, Department of Psychological and Brain Sciences, Washington University in St. Louis,
One Brookings Drive, St. Louis, MO 63130, USA. Email: [email protected]
References
Agarwal, P. K., D’Antonio, L., Roediger, H. L., III, McDermott, K. B., & McDaniel, M. A. (2014). Classroom-
Bernacki, M. L., Vosicka, L., & Utz, J. C. (2020). Can a brief, digital skill training intervention help
undergraduates “learn to learn” and improve their STEM achievement? Journal of Educational
Psychology, 112(4), 765–781. https://doi.org/10.1037/edu0000405
Bjork, R. A. (1994). Memory and metamemory considerations in the training of human beings. In J.
Metcalfe & A. P. Shimamura (Eds.), Metacognition: Knowing about knowing (pp. 185–205). The
MIT Press.
Borkowski, J. G., Carr, M., & Pressley, M. (1987). “Spontaneous” strategy use: Perspectives from
metacognitive theory. Intelligence, 11(1), 61–75. https://doi.org/10.1016/0160-2896(87)90027-
4
Bransford, J. D., & Johnson, M. K. (1973). Considerations of some problems of comprehension. In Visual
information processing. Academic.
Bui, Y., Pyc, M. A., & Bailey, H. (2018). When people’s judgments of learning (JOLs) are extremely
accurate at predicting subsequent recall: The “Displaced-JOL effect.” Memory, 26(6), 771–783.
Cogliano, M., Bernacki, M. L., & Kardash, C. M. (2020). A metacognitive retrieval practice intervention to
improve undergraduates’ monitoring and control processes and use of performance feedback
372
for classroom learning. Journal of Educational Psychology, 113(7), 1421-1440.
Duckworth, A. L., Kirby, T. A., Gollwitzer, A., & Oettingen, G. (2013). From fantasy to action: Mental
contrasting with implementation intentions (MCII) improves academic performance in
children. Social Psychological and Personality Science, 4(6), 745–753.
Dunlosky, J., & Nelson, T. O. (1994). Does the sensitivity of judgments of learning (JOLs) to the effects of
various study activities depend on when the JOLs occur? Journal of Memory and
Language, 33(4), 545-565. https://doi.org/10.1006/jmla.1994.1026
Dunlosky, J., & Tauber, S. (2016) (Eds). Oxford handbook of metamemory. Oxford.
Een, E. (2021). Changes in usage and perceptions of effectiveness of learning strategies of high school
students during a rigorous academic experience. Unpublished Master’s Thesis, Washington
University in St. Louis. https://openscholarship.wustl.edu/art_sci_etds/2281
Garner, R. (1990). When children and adults do not use learning strategies: Toward a theory of
settings. Review of Educational Research, 60(4), 517–529. https://doi.org/10.2307/1170504
Gollwitzer, P. M. (1999). Implementation intentions: Strong effects of simple plans. American
Psychologist, 54(7), 493–503. https://doi.org/10.1037/0003-066X.54.7.493
Harackiewicz, J. M., & Priniski, S. J. (2018). Improving student outcomes in higher education: The science
of targeted intervention. Annual Review of Psychology, 69, 409–435.
scheduling related to achievement? Psychonomic Bulletin & Review, 19(1), 126–134.
Hattie, J., Biggs, J., & Purdie, N. (1996). Effects of learning skills interventions on student learning: A
meta-analysis. Review of Educational Research, 66(2), 99-136.
http://www.jstor.org/stable/1170605
Hulleman, C. S., Kosovich, J. J., Barron, K. E., & Daniel, D. B. (2017). Making connections: Replicating and
extending the utility value intervention in the classroom. Journal of Educational
Psychology, 109(3), 387–404. https://doi.org/10.1037/edu0000146.supp (Supplemental)
Hunt, A.M.W., & Swogger, J.G. (2016). Unlocking the past! Radiocarbon dating. Carbon Comics Issue 1:
Bi-lingual STEM Comic Book Series.
Karpicke, J. D., Butler, A. C., & Roediger, H. L., III. (2009). Metacognitive strategies in student learning:
Do students practice retrieval when they study on their own? Memory, 17(4), 471–479.
study strategy choice. Cognitive Psychology, 115.
373
Koriat, A. (1997). Monitoring one's own knowledge during study: A cue-utilization approach to
judgments of learning. Journal of Experimental Psychology: General, 126(4), 349.
Kornell, N. (2009). Optimising learning using flashcards: Spacing is more effective than cramming.
Applied Cognitive Psychology, 23(9), 1297–1317. https://doi.org/10.1002/acp.1537
Leelawong, K., & Biswas, G. (2008). Designing learning by teaching agents: The Betty’s brain system.
International Journal of Artificial Intelligence in Education, 18(3), 181-208.
McCabe, J. A., Friedman-Wheeler, D. G., Davis, S. R., & Pearce, J. (2021). “Set” for success: Targeted
instruction in learning strategies and behavior change in introductory psychology. Teaching of
McDaniel, M. A., & Einstein, G. O. (2020). Training learning strategies to promote self-regulation and
transfer: The knowledge, belief, commitment, and planning framework. Perspectives on
Psychological Science, 15(6), 1363–1381. https://doi.org/10.1177/1745691620920723
McDaniel, M. A., Einstein, G. O., & Een, E. (2021). Training college students to use learning strategies: A
framework and pilot course. Psychology Learning & Teaching, 20(3), 364–382.
Pashler, H., Bain, P., Bottge, B., Graesser, A., Koedinger, K., McDaniel, M., & Metcalfe, J. (2007)
Organizing Instruction and Study to Improve Student Learning (NCER 2007-2004). Washington,
Education. https://files.eric.ed.gov/fulltext/ED498555.pdf
Pomerance, L., Greenberg, J., & Walsh. K. (2016). Learning about Learning: What Every New Teacher
Needs to Know. National Council on Teacher Quality, 1-36.
Pressley, M., Goodchild, F., Fleet, J., Zajchowski, R., & Evans, E. D. (1989). The challenges of classroom
strategy instruction. The Elementary School Journal, 89(3), 301–342.
Pressley, M., McDaniel, M. A., Turnure, J. E., Wood, E., & Ahmad, M. (1987). Generation and precision of
elaboration: Effects on intentional and incidental learning. Journal of Experimental Psychology:
Learning, Memory, and Cognition, 13(2), 291–300. https://doi.org/10.1037/0278-7393.13.2.291
Putnam, A. L., Sungkhasettee, V. W., & Roediger, H. L., III. (2016). Optimizing learning in college: Tips
from cognitive psychology. Perspectives on Psychological Science, 11(5), 652–660.
Seifert, T. L. (1994). Enhancing memory for main ideas using elaborative interrogation. Contemporary
Educational Psychology, 19(3), 360–366. https://doi.org/10.1006/ceps.1994.1026
Shaughnessy, J. J. (1981). Memory monitoring accuracy and modification of rehearsal strategies. Journal
of Verbal Learning & Verbal Behavior, 20(2), 216–230. https://doi.org/10.1016/S0022-
5371(81)90389-3
Stein, B. S., Bransford, J. D., Franks, J. J., Owings, R. A., Vye, N. J., & McGraw, W. (1982). Differences in
the precision of self-generated elaborations. Journal of Experimental Psychology:
General, 111(4), 399–405. https://doi.org/10.1037/0096-3445.111.4.399
spaced study. Instructional Science, 41(2), 345–363. https://doi.org/10.1007/s11251-012-9231-8
374
Talanquer, V., & Pollard, J. (2010). Let’s teach how we think instead of what we know. Chemistry
Education Research and Practice, 11, 74-83. https://doi.org/10.1039/C005349J
Wigfield, A., & Cambria, J. (2010). Students’ achievement values, goal orientations, and interest:
Definitions, development, and relations to achievement outcomes. Developmental
Review, 30(1), 1–35. https://doi.org/10.1016/j.dr.2009.12.001
Psychology: General, 145(7), 918–933. https://doi.org/10.1037/xge0000177
Appendix
Self-Explanation Demonstration
First, present the sentences below for about 5 seconds each (materials adapted from Pressley et al.,
1987). Present one sentence at a time and instruct students to try to learn the sentences for a later test.
If you want a more striking effect, and to more clearly show the effects of a strategy that does not
involve self-explanation and understanding, we suggest that you instruct students to use a rote
repetition strategy (i.e., instruct students to rotely repeat each sentence to themselves, over and over
again, during its 5-second presentation).
1. The short man bought the broom

2. The brave man gave the money to the robber
3. The fat man read the sign
4. The tall man bought the crackers
5. The thin man found the scissors
6. The rich man picked up the chair
7. The dying man used a feather
8. The kind man ate dinner
9. The bald man used the phone
10. The frightened man ironed the sheet
11. The dishonest man looked closely at the wrapper
12. The smart man went to work
Second, present the test below and ask students to write their answer on a sheet of paper.
1. Who ironed the sheet? (frightened man)
2. Who bought the broom? (short man)
3. Who read the sign? (fat man)
4. Who found the scissors? (thin man)
5. Who ate dinner? (kind man)
6. Who went to work? (smart man)
7. Who used the phone? (bald man)
8. Who looked closely at the wrapper? (dishonest man)
375
9. Who gave the money to the robber? (brave man)
10. Who bought the crackers? (tall man)
11. Who picked up the chair? (rich man)
12. Who used a feather? (dying man)
Third, give students the answers, and have them score their responses.
Fourth, give students instructions for how to use the self-explanation strategy for these types of
sentences. Specifically, encourage students to ask themselves how the action relates to the
characteristics of the person, and illustrate this with several sample sentences. For example, for the
practice sentence, “The hungry man got into the car,” ask students to come up with possible
explanations for why this particular person (a hungry man) would perform this particular action (get into
a car). One reasonable explanation is to drive to the restaurant. For the practice sentence, “The brave
man ran into the house,” probe students for an explanation that ties together brave man and running
into the house (e.g., to save the boy from the fire). For the practice sentence, “The tall man used the
paintbrush,” ask students to discover an explanation that connects painting to a tall man (e.g., to paint
the ceiling).
Fifth, present the next set of sentences for about 5 seconds each and remind students to search for a
meaningful explanation—and not to try to memorize the sentences.
1. The sad man looked at his new boat
2. The artistic man put down the knife
3. The sleepy man bought the mug
4. The evil man wound up the clock
5. The clever man hit the flea
6. The bearded man threw out the coupon
7. The friendly man flicked the switch
8. The religious man used the saw
9. The long-haired man looked for the pole
10. The Irish man counted the leaves
11. The weak man thanked the check-out woman
12. The patriotic man memorized the words
Sixth, present the test below and ask students to write their answer on a sheet of paper.
1. Who wound up the clock? (evil man)
2. Who put down the knife? (artistic man)
3. Who hit the flea? (clever man)
4. Who looked for the pole? (long-haired man)
5. Who thanked the check-out woman? (weak man)
6. Who memorized the words? (patriotic man)
7. Who threw out the coupon? (bearded man)
8. Who flicked the switch? (friendly man)
9. Who looked at his new boat? (sad man)
10. Who bought the mug? (tired man)
11. Who used the saw? (religious man)
376
12. Who counted the leaves? (Irish man)
Seventh, give students the answers, and have them score their responses. Then ask students to compare
their learning using the self-explanation strategy with using the rote repetition (or their own) strategy.
Our experience with this demonstration with college and high-school students is that nearly everyone
remembers substantially more items with the self-explanation strategy. This demonstration could be
followed by discussion that relates these results to the lecture emphasizing the importance of
understanding and self-explanation for optimizing learning.
377
In Their Own Words Adaptive Blended Learning to Foster Self-
Regulated Learning – A Principle-Based
Explanation of a Self-Regulated Learning
Your Academic Setting Training
Editors
Lauren L. Kordonowy
Tino Endres
Victor A. Benassi

University of Freiburg
If They Would Just Do What the Good Learners Do

Not all learning from a class or course occurs within the classroom. Some exercises and learning are
done at home and in individual, self-regulated settings. Some learners will likely struggle substantially
with these more self-regulated tasks than others. In light of this, an important question instructors
might ask is, "How can I help these struggling learners to learn more like their more successful
counterparts?" In other words, how can I help my students to become better self-regulated learners?
Unfortunately, it is not beneficial to simply tell learners what an effective strategy is and expect that
they act accordingly. Self-regulated learning (SRL) strategies must be flexibly applied according to the
current learning situation to be helpful, and then these strategies should get automatized to be most
effective.
But how can SRL be supported when teaching the course content alone is already very demanding?
Within this chapter I will offer one possible answer to this question and introduce a blended learning
(online learning with in-class support) SRL training we have developed at the University of Freiburg.
First, I will introduce the principles that, in our understanding, increase the probability that SRL training
will be successful. This theoretical overview will then be used to justify the design of our blended SRL
training program and should provide readers with the knowledge needed to adapt certain parts of our
training to their respective educational context. Readers interested in the scientific evaluation of the
blended SRL training program are invited to refer to Endres et al. (2021).
Principles of Our SRL training

When developing SRL trainings, it is important to keep in mind what we know from educational science
about what makes trainings focused on the strategies most effective. The following paragraphs outline
key principles, link them to specific components of our training program, and help teachers to make
their own adaptions to a similar training in an informed way. The different principles are not completely
distinct from each other and can overlap in certain areas. Still, the main idea of each principle can be
seen as a guideline for our SRL training (for an overview, see Figure 1). The list of principles is not meant
as a complete list but as a practical set of guidelines that can be used to understand and adapt our
training.
378
Figure 1
Overview of Our Principles Implemented in Our SRL Training
Principle 1: Cyclical Model of SRL

Prominent models of SRL focus on a three-phases cycle: a preparatory phase, a performance phase, and
an evaluation (or appraisal) phase (Panadero, 2017). Within each of these three phases, three groups of
strategies should be applied by students. First, cognitive strategies should help students acquire
knowledge and skills as effectively as possible. Examples for cognitive strategies are organizational
strategies (i.e., identifying main points) or elaborative strategies (e.g., providing an example). Second,
metacognitive strategies help students to plan their learning process, monitor what they have already
learned, what they still need to work on, and how they can address the knowledge gaps they have
discovered (Boekaerts, 1999). Third, support strategies comprise resource-based strategies (social
resources or time), volitional strategies (focus on motivation, emotion, and the physical environment),
or self-esteem protecting strategies (self-handicapping: Learners who create a handicap for themself
before an assessment situation (e.g., exam) that serves as an excuse in case of a possible failure. For
example a learner might start studying too late before an important exam to use it as an excuse that is
not directly tracked back to the general ability of that individual student (Weinstein & Mayer, 1986). This
three-phase, three-strategy model is broadly agreed upon by SRL researchers and emphasizes the
importance of the cyclic nature of the SRL process (Dignath & Büttner, 2008).
Principle 2: Focus on the Quality of Strategy Application
Quality of strategy application is one of the most important aspects of successful learning strategies.
Quality has two important aspects: conditional quality and performance quality. Conditional quality
describes the selection of a suitable learning strategy for a specific learning situation (Koedinger et al.,
this volume). This aspect is important because the effectiveness of a learning strategy depends on the
situation in which it is used (Endres et al., 2017, 2020; McCrudden & Schraw, 2007; Morris et al., 1977).
Effective self-regulated learners also adapt appropriate strategies to the learning situations at hand.
Studies show that high achievers do not just use more strategies in their SRL; rather, they make adaptive
use of a limited number of strategies (Heirweg et al., 2020). This finding highlights the need for
conditional knowledge, that is, knowledge about when and why to use strategies.
One way to convey such knowledge is to make the conditional knowledge explicit in an informed
training intervention (e.g., Hübner et al., 2010; Paris et al., 1983). Informed trainings teach students not
only about which strategies they can use, but they can tell them about the advantages, disadvantages,
and requirements of the strategies. Studies show that informed training interventions have the potential
to foster students’ strategy application and improve their learning (e.g., Ariel & Karpicke, 2018;
Carpenter et al., 2017; Hübner et al., 2010).
379
The second aspect of strategy application is performance quality, referring to students’ ability to
effectively use a strategy. Numerous studies show that the performance quality of a strategy application
is a driving force to increase the quality and quantity of learning (Endres et al., 2017; Glogger et al.,
2012; Rawson & Dunlosky, 2016; Roelle et al., 2017). An intervention that might help in this regard is the
implementation of cognitive modeling. In cognitive modeling, a more experienced learner explains
cognitive processes when applying a certain self-regulated learning strategy in a think-out-loud manner.
In this way the learner is provided with an overt model of a normally hidden cognitive process. In doing
so, learners are able to better understand the essence of the strategy and improve their own strategy
performance. Learners can transfer those strategies easier to their own learning (Zimmerman, 2013).
If students know when to use a strategy (conditional quality) and how to use a strategy (performance
quality), they still may fail to use it effectively. This problem could be due to a mediation deficiency, a
production deficiency, or a utilization deficiency. A mediation deficiency occurs when a learner is not
able to perform a certain strategy even when they are told how and when exactly to do it. This difficulty
could be due to a variety of reasons, including the necessary mental capacity (mediation deficiency).
Mediation deficiency should be kept in mind particularly when working with younger learners or
learners that have a lower working memory capacity. Production deficiency occurs when students fail to
use a strategy spontaneously. In this stage, explicit prompts can be used to encourage learners to use
the newly learned strategies (Nückles et al., 2020) or other instructions (e.g., implementation intensions
can be provided; Endres et al., 2021). Utilization deficiency occurs when learners use strategies
spontaneously and at an appropriate time, but they do not benefit from them (Hasselhorn, 1996,
Hasselhorn & Gold, 2017). This last deficiency usually results from a lack of automation because the
strategy execution absorbs most of the cognitive capacity. Learning strategies trainings should therefore
be implemented over a longer period of time to allow for automation. Only when these three
deficiencies are addressed can learners flexibly and effectively use new learning strategies.
Principle 3: Intertwined Use of Direct and Indirect Ways of Fostering SRL Strategies
There are two major ways to support SRL: direct and indirect (Dignath & Veenman, 2021). Direct
support refers to interventions that instruct learners directly to use certain learning strategies. For
example, interventions such as the previously introduced informed training interventions (see Principle
2) use direct support. Indirect support, on the other hand, describes interventions that give learners
opportunities to make use of their SRL strategies. Examples of indirect support include engaging learners
in complex and meaningful tasks and allowing them to make their own strategy choices while learning.
SRL support improves strategy application best when it employs both direct and indirect support in
tandem. If just direct support is used, learners may have few opportunities to make use of their newly
learned strategies. They might not see the value in these strategies and rapidly forget what they have
learned. Conversely, opportunities to engage in complex tasks without additional support from teachers
or peers do not provide students with the conditional and performance knowledge required to
effectively use appropriate strategies (Perry, 2013). It is crucial to find the right balance between
indirect and direct support, especially in difficult learning situations (Archer & Hughes, 2011).
Two instructional methods that are essential when combining direct and indirect interventions are
scaffolding and fading. Scaffolding involves performing some elements of the task for the learner or
providing hints to reduce overall cognitive load and to enable them to focus on the most essential
element(s) of a strategy. Fading involves progressive removal of these scaffolds over time. This gradual
release of responsibility to the learner enables them to make the transition from external- to self-
regulation without being overwhelmed in the process (Fisher & Frey, 2014; Friedrich & Mandl, 1997).
380
Another combination is described in the concept of constructive alignment (Biggs, 1996). This
framework highlights the match between direct support, the use of learning strategies by learners, and
exams. When a teacher uses a form of direct support that requests learners to engage deeply with the
topic, the teacher should also develop exam questions that require deep engagement and inform
students that they will be assessed on these skills. Such a procedure will increase the probability that
students will use more complex SRL strategies. Constructive alignment at the institutional level is even
more effective than at the class level, as it demonstrates strategy use expectations to students across
multiple classes and years (Bakkenes et al., 2010; De Smul et al., 2020).
Principle 4: Persistence to Change Strategy Application
Learners have their own learning experience that must be considered when changing ineffective
strategies. Consideration of students’ prior behavior and prior knowledge can influence the training
success. When deciding whether a new learning strategy makes sense to implement, students often
compare the new learning strategy to the strategies they usually use (Hui et al., 2021a). The problem
here is that new strategies are not performed in the same effective way as old and established
strategies. This is often because the old strategy is automated and can be carried out easily, but the new
strategy is still conscious and effortful, leading to a utilization deficiency (the strategy is used, but few
benefits are gained). This ‘trough of disillusionment’ (low initial strategy benefit) with new strategies can
keep many learners from changing their strategies, even when the new strategy would have led to
better learning. Therefore, a learning strategy implementation cannot be taught on a single occasion. It
must be reinforced persistently to reveal its full potential to students. Strategies should be implemented
in varying contexts, which include different topics with different structures and different application
situations that are familiar to the learners. In that way, learners can extend and practice their
conditional knowledge in their authentic learning environment. This additional enrichment of the
conditional knowledge will increase transfer of the learning strategies to further learning contexts
(Hasselhorn, 1996). Approaches to persistent reinforcement are presented in the following three
paragraphs.
Implement Strategies in the Daily Learning Routine
The chosen examples in the training should match the authentic learning situations of the respective
learners. This aspect embraces the connections to content taught in the curriculum as well as the
learning situations. Two benefits of making these connections are worth highlighting. First, there is a
motivational benefit, as the interest in the strategies can be triggered by the interest in the topic
(Renninger & Hidi, 2019). Second, it is easier for students to see potential application possibilities in
their own learning. This recognition facilitates the connection between the explanations in the training
and their own learning. Such connections foster long-term memory for these strategies (Mayer, 2014).
Connect SRL Training to the Curriculums’ Content
This aspect highlights that a SRL training should be connected to the curriculums’ content and be a
steady companion in different classes. This win-win situation allows instructors to be very time efficient
in the implementation of SRL trainings while teaching the curriculum’s content. In that way, no
additional time must be “wasted” on SRL trainings, and at the same time it conveys the need for proper
Spaced Repetition of Strategies to Change Habits
The foundation for which learning strategies learners use is developed early in life. In school, students
develop beliefs about learning and develop certain learning strategies and habits. These strategies and
habits may influence their approach towards learning in adult life. The longer a learner is using a certain
381
learning strategy in a specific learning situation, the harder it will be for the learner to change that
behavior. A problem, however, is that many or even most students in this early and decisive stage do
not primarily rely on learning strategies that are evidence-based (Bjork et al., 2013; Carpenter et al.,
2020; Jairam & Kiewra, 2009). Therefore, strategy training should also consider how to help change
these habits. One possibility is to form new connections between learning situations and the desired
learning behavior (Gollwitzer & Sheeran, 2006s). To reach sustainable behavior changes it is essential
that the behavior is repeated and spaced over time (Bakkour et al., 2018).
Principle 5: Changing Motivational Structures
Effective SRL training also addresses motivational factors. Learners usually already have beliefs about
what effective learning strategies are for them; thus, they may be unmotivated to attempt a new
strategy. This is because the cost of change is high (effort), and the benefit to their learning may be, as
yet, unclear. If we want to change behaviors, the theory of planned behavior is one of the most
influential frameworks (Ajzen, 1991; Ajzen & Schmidt, 2020), and it has already been adapted to SRL
research (Hui et al., 2021b). The theory highlights three important beliefs that must be considered, and
potentially changed, to boost students’ motivation to change. When combined, these three aspects of
motivation to change will lead to the desired future behavior (Ajzen, 2019).
Behavior Beliefs
These beliefs include erroneous beliefs about the likely consequences of the current learning behavior.
To motivate learners to change their old learning strategies, learners need to have the expectation that
the new strategy is more efficient than their old strategy. There are many direct and indirect ways to
address previous misconceptions. One direct way is to implement refutation texts (Lassonde et al., 2016;
Tippett, 2010). Refutation texts are known from the field of conceptual change and are short bits of
information that consider and value misbeliefs a learner might have. For example, an instructor could
explain that investing effort in a strategy is not necessarily a sign learners do something wrong, although
it might be understandable to think that an increased effort is problematic (valuing the misbelief). The
instructor could explain that effort actually helps students to learn more sustainable in many cases and
they should try to invest more effort in certain strategies. An indirect way to influence behavioral beliefs
is to show students that their current learning behavior is not sufficient for the current learning goals.
For example, an instructor could use a complex learning task in a preparatory exam. Students can then
self-evaluate that their current learning strategy seems to be unable to reach the level of learning that is
necessary for the demanded tasks and change the behavioral belief about that strategy.
Normative Beliefs of What Other Learners Do
Learners need to see that certain learning behaviors work for others who are like them in a similar
context. This peer-evaluation seems to be very important for leaners (Schunk, 1987). This is one reason
why social aspects should be included in SRL trainings (Friedrich & Mandl, 1997). For example, students
could work on their strategies in small groups in which they support each other and exchange their
experiences about which difficulties and achievements they made when implementing the new learned
Control or Self-Efficacy to Actually Change This Behavior
If learners do not believe in their ability to change learning strategy behaviors, their willingness to at
least try it will be low. One approach to increase this motivation is to tell them about other learners who
have successfully changed their learning strategies, or ask them to recall a time in the past when they
successfully changed their own behavior (e.g., Biwer et al., 2020).
382
The ELIS Training
Our SRL training is named ELIS (acronym for " erfolgreich Lernen im Studium" or "successful learning at
university" in English). In an adapted form it might also be useful for other educational levels. In this
chapter, the training will be used as an example to illustrate the introduced principles. It is not meant as
a prototypical training program that fulfils all criteria, but as a practical solution we are currently
successfully using at our university.
The ELIS training is focused on learners that are confronted with a high demand of SRL, as this is usually
the case for first-semester students who have moved from the structured school environment to the
less structured university environment. We are currently offering the training to our beginning
psychology students, with the training embedded within a first-semester developmental psychology
module. The ELIS learning strategy training includes several phases that have different goals, and
different principles are employed to help students to use learning strategies (for an overview see Figure
2). Over the years, the training program has been adapted and improved. In the remainder of this
chapter, I will briefly introduce the various aspects of the ELIS training. Each aspect will be addressed in
two parts. The first part will describe how the training reflects the theoretical principles introduced
above. This part should provide an understanding of what we are doing in the classroom. The second
part will propose potential adaptions for your own educational setting. This part should provide ideas
for how to apply what we have learned in different educational settings.
Figure 2
Overview of SRL Training Components
Indirect Ways of Supporting SRL Application

Our Goals and Implementation
One of the most important aspects of our training is that we also include indirect ways of supporting SRL
strategies. It might seem odd to describe this point first, but I want to highlight how important this
aspect is to the success of our training. We want to make sure that students who implement the
383
suggested strategies will have opportunities to apply them and will benefit from doing so. Therefore, the
training is embedded within our developmental psychology module. The module consists of a seminar
and a related lecture. By connecting the training to the module, we can make it clear to students that
the exam questions at the end of the unit will require the use of effective learning strategies (see
constructive alignment earlier in this chapter). Furthermore, included is the potential to make parts of
our training mandatory module work. This allows instructors to forego other academic tasks, such as
essays that might be obligatory in all modules. Because the strategy training is incorporated within the
curriculum’s content, the effort students invest in learning for the final exam is partly already included in
the strategy training. These two aspects should reduce the additional effort learners have to invest into
the training and take into consideration learners’ overall workload. The connection to the module also
helps us provide feedback and support when unexpected higher demands of support are needed.
Potential Adaptions for Alternative Settings
If instructors want to implement a similar SRL strategy training, it makes sense to think about how you,
as the instructor, could indirectly support students. The principle of constructive alignment is very
powerful in this respect. Teachers should ensure that students know from the start that passing exams
with good grades will require use of sophisticated learning strategies that might, for example, require
transfer to new problems or connections to examples from their own experience (Krathwohl, 2002).
Given that students will be instructed to self-regulate, it will be important that instructors do not overly
pre-structure the learning process. In a university module like ours, the demand to self-regulate is rather
high by nature, and sophisticated strategies are required. These SRL skills may not be as important in
more structured environments (e.g., high school). Ultimately, increasing the requirement for self-
regulation throughout instructors’ courses will foster learners' skills in SRL (see scaffolding and fading).
Another positive adaption would be to involve the whole institution in such a training, which represents
constructive alignment at an institutional level. When more teachers take part in a similar approach to
learning, this can lower the demand on individual teachers and can improve students’ SRL strategy
application, as students have more conditional knowledge on different subjects, and they can practice
their SRL skills in varying contexts.
Adaptive Learning of Conditional Knowledge and Motivation to Change
The main goals of this phase are to build motivation to change and to teach the needed conditional
knowledge about learning strategies. We use an online learning approach in this phase because we think
it can tackle most challenges in our context effectively.
The first part of our training includes different videos using the style of a sketched explanation video
(Endres et al., 2020). Sketched explanation videos are one way to implement emotional design, which
helps to motivate learners (Brom et al., 2018). Our emotional designed videos with engaging characters
have been shown to catch students’ interest in the topic of learning strategies and to start developing
interest (Endres et al., 2021). Our training includes a character called ELIS, who wants to change the SRL
strategies she is using and subsequently succeeds (control beliefs). In the training’s introduction video,
we highlight why ELIS’s current strategies are not working for her and why she wants to change her
behavior (behavioral beliefs). In the accompanying videos we teach conditional knowledge about the
strategies in an informed training. In the videos, common student misconceptions about learning are
directly addressed which aims to further boost students’ motivation to change (behavioral beliefs).
The content of the conditional knowledge phase contains the cyclic model of SRL introduced in Principle
1. This part is implemented online to give learners with varying learning prerequisites the opportunity to
384
engage with the content in an individual way. There are huge differences between learners’ knowledge
and skills about SRL strategies (Heirweg et al., 2019; Pintrich, 2004; Vandevelde et al., 2012). To give
every student the same chance to profit from a strategy training, teachers need to adapt their teaching
of SRL strategies to individual learners. However, without technological support, this adaptation is very
demanding when there is a bigger group of learners, which usually leads to a lower quality of SRL
instruction (Smul et al., 2018).
Our adaptive online system helps us to bring every learner to a similar level of conditional knowledge.
We implemented retrieval practice to improve conditional knowledge persistency. Retrieval practice
leverages the fact that recalling memories can strengthen them and provide a good additional
opportunity to elaborate on newly learned knowledge (see Rowland, 2014; Yang et al., 2021). Retrieval
practice is most effective when learners invest high mental effort in a retrieval task (Endres & Renkl,
2015; Pyc & Rawson, 2009) and, at the same time, succeed in recalling the respective contents
(Carpenter, 2009; Heitmann et al., 2021; Rowland, 2014). In an evaluation study, we found that learners
with different levels of prior knowledge benefit from different retrieval practice tasks (Endres et al.,
2021). For this reason, we use an adaptive algorithm based on learners’ prior conditional knowledge
about learning strategies. To assess prior conditional knowledge, we use items in which learners have to
judge how successful specific learning strategies will probably be in a specific learning situation. The
learning situations use in those items are learning situations that match a first-semester students
experiences in university (see Figure 3).
Figure 3
Example of Our One of Twelve Conditional Knowledge Tasks (Translated from German by the author.)
385
After a prior knowledge assessment, learners are assigned to a tailored progression of learning tasks. To
consolidate students’ conditional knowledge, retrieval tasks are increasingly spaced over time. We used
spacing intervals of one, three, and seven days based on an algorithm established in prior research
(Carpenter et al., 2012). The learners can access the learning content after that spacing period at any
time for remedial learning.
Currently we are evaluating a revised version of our conditional knowledge phase. We realized that the
previous version required high persistence over several weeks, which was especially problematic for
learners with low prior knowledge. This increased cognitive demand for low prior knowledge learners is
why we changed our adaptive phase to a criterion-based learning arrangement that adapts to the
learners’ prior knowledge in a shorter loop (see Figure 4). We split our content into twelve parts. Each
content part consisted of the learning material and a corresponding conditional knowledge task that
covered the knowledge of that one content part. To master one of the content parts, learners had to
answer the conditional knowledge task at least once correctly (Karpicke & Bauernschmidt, 2011;
Leonard et al., 2020). The content material was presented using sketched explanation videos and texts.
It consisted of conditional knowledge, refutation elements, and cognitive modeling of the specific SRL
strategies. The adaptive algorithm first presents a respective conditional knowledge task. If correctly
answered, the corresponding teaching material will not be provided. If answered incorrectly, the
content will be presented. The same task is interleaved with other conditional knowledge questions.
This is done to keep mental effort high. Initial results showed that adaptive learning on the criterion
level was especially helpful for low prior knowledge students (Endres et al., 2021).
Figure 4
Adaptive Mechanism Visualization Exemplary for Two of Twelve Conditional Knowledge Parts
386
The first part of our training focuses on motivation to change and adapt conditional knowledge. This
part might fit many contexts. If instructors think about using this part of our training in a university
module or with upper-level high school students, the content and examples should be adapted to be
suitable for such contexts as well. To make sure that learners change their beliefs and that they connect
the strategies to their own learning, we used content from late high school learning as well as the start
of university learning. The online parts are free to use. We are working on an English version that will be
available from Spring 2022.
If these situations are not authentic in your context, we recommend adapting the conditional knowledge
question and refutation parts of the text. Research shows that refutation works best when it is adapted
to the learners it refers to (Dersch et al., 2021). The same is true for the authentic contexts of SRL
strategies (Friedrich & Mandl, 1997).
On a more qualitative note, we heard from a few students in our program that they fully understood the
purpose of the training only after some additional time at university, that is, when they reflected on that
training in their 4th semester pedagogical psychology module. We assume this late awareness might be
due to the personal experience learners had with failure of their own old strategies for certain exams
(behavioral beliefs). It might be helpful to request this reflection a bit earlier. This should be helpful
especially for students who do not react to our explicit way of increasing their motivation to change (see
Adaptive Learning of Conditional Knowledge and Motivation to Change). For example, the inclusion of a
reflection lesson after the first exams they had after the training could increase changes in behavioral
believes.
Journal Writing to Maintain New Behavior
In the next part of our training, we use the method of learning journals from a Writing-to-Learn as self-
regulation-view. When using this perspective, learners are usually asked to write down their reflections
on previously learned contents in a written journal. They are prompted to ask themselves what they do
not understand (monitoring) and what they can do to close the detected knowledge gaps (regulation). In
doing so, learners can apply beneficial cognitive and metacognitive strategies in a slowed down
cognitive way, that supports their application quality (e.g., Nückles et al., 2020). After stimulating a
sufficient motivation to change and giving learners the conditional knowledge of when to use the
strategies, we focus on overcoming production deficiency and utilization deficiency of learners. We do
this in three parts. First, learners choose specific implementation intentions for learning strategies they
want to apply. Second, they repeatedly write learning journal entries on current topics of the
developmental psychology module. Finally, they engage in a peer feedback activity on journal writing.
1. Setting Implementation Intentions
In this phase, we tackle the aspects of production deficiency and habits. As mentioned previously,
learners usually have long histories of learning that resulted in certain habits of using strategies. These
habits can keep them from using the recently learned and more effective learning strategies. In our
training, after each lecture learners identify and document, in an online system, their learning goals for
the following week. In their journals, students formulate implementation intentions (Gollwitzer &
Sheeran, 2006) with respect to the strategies they intend to apply. We found that it is helpful to provide
learners with some guidance to set specific implementation intentions. More specifically, we
implemented a scaffolding procedure that helped students connect implementation intentions to
387
specific complex learning situations and provided contrasting cases of good and bad implementation
intentions (Endres et al., 2021).
Potential Adaptations for Alternative Settings
The implementation intention intervention is independent of the specific context of the SRL training. We
implemented the tool in our own adaptive learning module (Scheiter et al., 2017). If instructors use
other platforms to implement journal writing, the content can be found here: https://elis.vm.uni-
freiburg.de/demo/index.php.
2. Appling Learning Strategies in Learning Journals
After formulating their implementation intentions, learners engaged in journal writing at home. Our
online system provided learners with their previously chosen implementation intentions. The
implementation intentions should prompt learners to use the learning strategies they intended to use.
This prompting was meant to overcome the production deficiency within journal writing. The journal
writing itself was meant to overcome the utilization deficiency of strategies, because slower and more
reflective application of strategies helps learners to focus on the quality of the SRL strategies (Nückles et
al., 2020). The learners were required to write learning journals over several topics throughout the year.
This varied context spaced over time should enable them to transfer their strategies to other learning
situations and generate sustainable behavioral change.
If instructors want to implement a similar journal writing intervention without the use of our first phase
of setting implementation intentions, the use of prompts seems similarly effective (Endres et al., 2021).
On a more qualitative note, we experienced that it is important to not require a learning journal entry
for every topic of a weekly module, but instead, give learners a certain amount of freedom. The time
demand of an active reflection, together with the demand of the content, can be overwhelming for
students (see also Nückles et al., 2020). This is especially true when you want to include additional peer
feedback. The high time demand can reduce students’ motivation and persistence to work with learning
journals. This motivational decrease is why, in a further intervention, we included adaptive motivational
interventions based on the expectancy value theory (Eccles & Wigfield, 2020) to increase intrinsic
motivation in journal writing. This intervention showed that during the university term, intrinsic
motivation can be maintained and even slightly increased (Udvardi-Lakos et al., 2021).
3. Using Peer Feedback
For our learning journals we implemented a weekly, anonymous peer feedback intervention to support
the quality of the learning journals. To increase students’ feedback quality we included rubrics
(Panadero & Jönsson, 2013). Rubrics gave support on assessment of learning strategies as well as on the
formulation of the feedback (see Bürgermeister et al., 2021). Additionally, the peer feedback
components increased the social involvement of the students in the training. Learners had the
opportunity to observe both their peers’ successful and unsuccessful learning strategy behaviors,
thereby learning from their peers’ learning strategy application what could work for themselves
(normative believes). After midterms, learners had the option to request feedback from their instructor
on their learning journals. Instructors gave feedback on the quality the learning strategies were applied
and how students could improve their learning process even more.
388
In our experience, providing rubrics as well as teachers’ feedback on request are important aspects that
make the peer feedback component effective. Although we used an automatic system to distribute
anonymized peer feedback to the students, peer feedback can be implemented without technical
support. An analog distribution of papers to give peer feedback may reduce technical problems with
learners who are not as experienced in using an online learning system.
Current Use of the Learning Environment

Our SRL training has been implemented in the curriculum for first-semester students at the Department
of Psychology at the University of Freiburg (Germany) for five years now. Around 110 psychology
students take part in our program every year. The online components are used by an additional 60
students from different independent minors and majors each year.
In Closing
Our training, even in its current form, is not perfect. It is subject to financial, temporal, and social
resources we are able to invest in its development. One major limitation at the moment is that the
training is only connected to the content that our department teaches; other departments in our
institute do not explicitly connect their instructional approaches to the training. Differences in the
learning approach result in some exams using similar types of challenging questions while others do not.
An institute-wide or even university-wide approach (e.g., Maastricht University) would probably
increase the indirect support for our SRL training, as students could be sure what kind of learning
approach will be required in all their courses. We are currently developing an English version to allow
more learners to use our training. Further, we are working on adapting the program to a wider range of
fields of study. In this project we want to adapt the examples, refutations, and the cognitive modeling
included in the training more closely to the varying contexts of learners in different fields. These
contextual adaptations should increase learners’ motivation to work on their strategies persistently. All
parts of our SRL training that have increased learners’ performance are available to the public. You can
find the respective content in the links in the respective chapters (see Adaptive Learning of Conditional
Knowledge, and Motivation to Change, and Setting Implementation Intentions). We will continue to
improve our program and make evaluated parts widely available in order to allow as many learners as
possible to work with a state-of-the-art SRL training program.
Author Note
Tino Endres is a PostDoc at University of Freiburg. Correspondence concerning this article should be
addressed to Tino Endres, University of Freiburg, ORCID: 0000-0003-3334-4064, Department of
Psychology, Educational and Developmental Psychology, Engelbergerstraße 41, D-79085 Freiburg,
Germany. E-mail: [email protected].
References
Ajzen, I. (1991). The theory of planned behavior. Organizational Behavior and Human Decision
Processes, 50(2), 179–211. https://doi.org/10.1016/0749-5978(91)90020-T
Ajzen, I. (2019). Constructing a theory of planned behavior questionnaire. University of Massachusetts
Amherst, 1–7. http://people.umass.edu/~aizen/pdf/tpb.measurement.pdf
389
Ajzen, I., & Schmidt, P. (2020). Changing behavior using the theory of planned behavior. In M. S Hagger,
L. D. Cameron, K. Hamilton, & N. Han, The handbook of behavior change (pp. 17–31).
Archer, A. L., & Hughes, C. A. (2011). Explicit instruction: Effective and efficient teaching. Guilford Press.
Journal of Experimental Psychology: Applied, 24(1), 43–56. https://doi.org/10.1037/xap0000133
Bakkenes, I., Vermunt, J. D., & Wubbels, T. (2010). Teacher learning in the context of educational
innovation: Learning activities and learning outcomes of experienced teachers. Learning and
Instruction, 20(6), 533–548. https://doi.org/10.1016/j.learninstruc.2009.09.001
Bakkour, A., Botvinik-Nezer, R., Cohen, N., Hover, A. M., Poldrack, R. A., & Schonberg, T. (2018). Spacing
of cue-approach training leads to better maintenance of behavioral change. PLOS ONE, 13(7),
e0201580. https://doi.org/10.1371/journal.pone.0201580
Biggs, J. (1996). Enhancing teaching through constructive alignment. Higher Education, 32(3), 347–364.
Biwer, F., Egbrink, M. G. A. oud., Aalten, P., & de Bruin, A. B. H. (2020). Fostering effective llearning
strategies in higher education – A mixed-methods study. Journal of Applied Research in Memory
and Cognition, 9(2), 186–203. https://doi.org/10.1016/j.jarmac.2020.03.004
Annual Review of Psychology, 64(1), 417–444. https://doi.org/10.1146/annurev-psych-113011-
143823
Boekaerts, M. (1999). Self-regulated learning : Where we are today Self-regulated learning : where we
are today. International journal of educational research, 31(6), 445-457.
Brom, C., Stárková, T., & D’Mello, S. K. (2018). How effective is emotional design? A meta-analysis on
facial anthropomorphisms and pleasant colors during multimedia learning. Educational Research
Review, 25, 100–119. https://doi.org/10.1016/j.edurev.2018.09.004
Bürgermeister, A., Glogger-Frey, I., & Saalbach, H. (2021). Supporting peer feedback on learning
strategies: Effects on self-efficacy and feedback quality. Psychology Learning & Teaching, 20(3),
Carpenter, S. K. (2009). Cue strength as a moderator of the testing effect: The benefits of elaborative
retrieval. Journal of Experimental Psychology. Learning, Memory, and Cognition, 35(6), 1563–
Carpenter, S. K., Cepeda, N. J., Rohrer, D., Kang, S. H. K., & Pashler, H. (2012). Using spacing to enhance
diverse forms of learning: Review of recent research and implications for instruction.
Educational Psychology Review, 24(3), 369–378. https://doi.org/10.1007/s10648-012-9205-z
Carpenter, S. K., Endres, T., & Hui, L. (2020). Students’ use of retrieval in self-regulated learning:
Implications for monitoring and regulating effortful learning experiences. Educational
Psychology Review, 32(4), 1029–1054. https://doi.org/10.1007/s10648-020-09562-w
Carpenter, S. K., Rahman, S., Lund, T. J. S., Armstrong, P. I., Lamm, M. H., Reason, R. D., & Coffman, C. R.
(2017). Students’ use of optional online reviews and its relationship to summative assessment
outcomes in introductory biology. CBE Life Sciences Education, 16(2), 1–9.
390
De Smul, M., Heirweg, S., Devos, G., & Van Keer, H. (2020). It’s not only about the teacher! A qualitative
study into the role of school climate in primary schools’ implementation of self-regulated
learning. School Effectiveness and School Improvement, 31(3), 381–404.
De Smul, M., Heirweg, S., Van Keer, H., Devos, G., & Vandevelde, S. (2018). How competent do teachers
feel instructing self-regulated learning strategies? Development and validation of the teacher
self-ef fi cacy scale to implement self-regulated learning. Teaching and Teacher Education, 71,
214–225. https://doi.org/10.1016/j.tate.2018.01.001
Dersch, A.-S., Renkl, A., & Eitel, A. (2021). Do refutation texts foster teachers’ conceptual change? Only
when designed in a personalized way. Journal of Computer-Assisted Learning [Manuscript
Submitted for Publication].
Dignath, C., & Büttner, G. (2008). Components of fostering self-regulated learning among students. A
meta-analysis on intervention studies at primary and secondary school level. Metacognition and
Learning, 3(3), 231–264. https://doi.org/10.1007/s11409-008-9029-x
Dignath, C., & Veenman, M. V. J. (2021). The role of direct strategy instruction and indirect activation of
self-regulated learning—evidence from classroom observation studies. Educational Psychology
Review, 33(2), 489–533. https://doi.org/10.1007/s10648-020-09534-0
Eccles, J. S., & Wigfield, A. (2020). From expectancy-value theory to situated expectancy-value theory: A
developmental, social cognitive, and sociocultural perspective on motivation. Contemporary
Educational Psychology, 61, 101859. https://doi.org/10.1016/j.cedpsych.2020.101859
Endres, T ., Carpenter, S. , Martin, A. , & Renkl, A. (2017). Enhancing learning by retrieval: Enriching free
recall with elaborative prompting. Learning and Instruction, 49. 13-20.
Endres, T. , Kleinbub, J. D. , & Renkl, A. (2021). Adaptive interleaved retrieval practice – Usefulness of
adaptivity and interleaving intervals are depending on learner’s prior knowledge. [Manuscript in
Preparation].
Endres, T. , Kranzdorf, L. , Schneider, V., & Renkl, A. (2020). It matters how to recall – task differences in
retrieval practice. Instructional Science, 48, 699-728 https://doi.org/10.1007/s11251-020-09526-
1
Endres, T., Leber, J., Böttger, C., Rovers, S., & Renkl, A. (2021). Improving lifelong learning by fostering
students’ learning strategies at university. Psychology Learning and Teaching, 20(1), 144–160.
Endres, T., & Renkl, A. (2015). Mechanisms behind the testing effect: an empirical investigation of
retrieval practice in meaningful learning. Frontiers in Psychology, 6, 1–6.
Endres, T., Weyreter, S., Renkl, A., & Eitel, A. (2020). When and why does emotional design foster
learning? Evidence for situational interest as a mediator of increased persistence. Journal of
Computer Assisted Learning, 36(4), 514–525. https://doi.org/10.1111/jcal.12418
Fisher, D., & Frey, N. (2014, 2nd edition). Better learning through structured teaching: A framework for
the gradual release of responsibility, ASCD.
391
Friedrich, H., & Mandl, H. (1997). Analyse und Förderung selbstgesteuerten Lernens. In E. Weinert & H.
Mandl (Eds.), Psychologie der Erwachsenenbildung. Enzyklopädie der Psychologie (pp. 237-293).
Göttingen: Hogrefe). Hogrefe.
Glogger, I., Schwonke, R., Holzäpfel, L., Nückles, M., & Renkl, A. (2012). Learning strategies assessed by
journal writing: Prediction of learning outcomes by quantity, quality, and combinations of
learning strategies. Journal of Educational Psychology, 104(2), 452–468.
Gollwitzer, P., & Sheeran, P. (2006). Implementation intentions and goal achievement: A meta analysis
of effects and processes. Advances in Experimental Social Psychology, 38(2006), 69–119.
Hasselhorn, M. (1996). Kategoriales Organisieren bei Kindern: Zur Entwicklung einer
Gedächtnisstrategie. Hogrefe.
Hasselhorn, M., & Gold, A. (2017). Pädagogische Psychologie: Erfolgreiches Lernen und Lehren.
Kohlhammer Verlag.
Heirweg, S., De Smul, M., Devos, G., & Van Keer, H. (2019). Profiling upper primary school students’ self-
regulated learning through self-report questionnaires and think-aloud protocol analysis.
Learning and Individual Differences, 70, 155–168. https://doi.org/10.1016/j.lindif.2019.02.001
Heirweg, S., De Smul, M., Merchie, E., Devos, G., & Van Keer, H. (2020). Mine the process: investigating
the cyclical nature of upper primary school students’ self-regulated learning. Instructional
Science 48(4), 337-369. https://doi.org/10.1007/s11251-020-09519-0
Heitmann, S., Obergassel, N., Fries, S., Grund, A., Berthold, K., & Roelle, J. (2021). Adaptive practice
quizzing in a university lecture: A pre-registered field experiment. Journal of Applied Research in
Memory and Cognition, 10(4), 603–620. https://doi.org/10.1037/h0101865
Hübner, S., Nückles, M., & Renkl, A. (2010). Writing learning journals: Instructional support to overcome
learning-strategy deficits. Learning and Instruction, 20(1), 18–29.
Hui, L., de Bruin, A. B. H., Donkers, J., & van Merriënboer, J. J. G. (2021a). Does individual performance
feedback increase the use of retrieval practice? Educational Psychology Review, 33, 1835–1857.
Hui, L., de Bruin, A. B. H., Donkers, J., & van Merriënboer, J. J. G. (2021b). Stimulating the intention to
change learning strategies: The role of narratives. International Journal of Educational Research,
107. https://doi.org/10.1016/j.ijer.2021.101753
Jairam, D., & Kiewra, K. A. (2009). An Investigation of the SOAR Study Method. Journal of Advanced
Academics, 20(4), 602–629. https://doi.org/10.1177/1932202X0902000403
Karpicke, J. D., & Bauernschmidt, A. (2011). Spaced retrieval: Absolute spacing enhances learning
regardless of relative spacing. Journal of Experimental Psychology: Learning Memory and
Cognition, 37(5), 1250–1257. https://doi.org/10.1037/a0023436
Krathwohl, D. R. (2002). A revision of Bloom ’s taxonomy: Theory Into Practice, 41(4), 212–218.
https://doi.org/10.1207/s15430421tip4104_2
Lassonde, K. A., Kendeou, P., & O’Brien, E. J. (2016). Refutation texts: Overcoming psychology
misconceptions that are resistant to change. Scholarship of Teaching and Learning in
392
Leonard, L. B., Deevy, P., Karpicke, J. D., Christ, S. L., & Kueser, J. B. (2020). After initial retrieval practice,
more retrieval produces better retention than more study in the word learning of children with
developmental language disorder. Journal of Speech, Language, and Hearing Research, 63(8),
2763–2776. https://doi.org/10.1044/2020_JSLHR-20-00105
Mayer, R. E. (Ed.). (2014). The Cambridge handbook of multimedia learning (2nd ed.). Cambridge
University Press. https://doi.org/10.1017/CBO9781139547369
McCrudden, M. T., & Schraw, G. (2007). Relevance and goal-focusing in text processing. Educational
Psychology Review, 19(2), 113–139. https://doi.org/10.1007/s10648-006-9010-7
Morris, C. D., Bransford, J. D., & Franks, J. J. (1977). Levels of processing versus transfer appropriate
processing. Journal of Verbal Learning and Verbal Behavior, 16(5), 519–533.
Nückles, M., Roelle, J., Glogger-Frey, I., Waldeyer, J., & Renkl, A. (2020). The self-regulation-view in
writing-to-learn: Using journal writing to optimize cognitive load in self-regulated Learning.
Educational Psychology Review, 32(4), 1089–1126. https://doi.org/10.1007/s10648-020-09541-1
Panadero, E. (2017). A review of self-regulated learning: Six models and four directions for research.
Frontiers in Psychology, 8, 1–28. https://doi.org/10.3389/fpsyg.2017.00422
Panadero, E., & Jönsson, A. (2013). The use of scoring rubrics for formative assessment purposes
revisited: A review. Educational Research Review, 9, 129–144.
Paris, S. G., Lipson, M. Y., & Wixson, K. K. (1983). Becoming a strategic reader. Contemporary
Educational Psychology, 8(3), 293–316. https://doi.org/10.1016/0361-476X(83)90018-8
Perry, N. E. (2013). Understanding classroom processes that support children’s self-regulation of
learning. British Journal of Educational Psychology, Monograph Series II: Psychological Aspects of
Education—Current Trends. The British Psychological Society, 11(10), 45–68.
Pintrich, P. R. (2004). A conceptual framework for assessing motivation and self-regulated learning in
college students. Educational Psychology Review, 16(4), 385–407.
Pyc, M. A., & Rawson, K. A. (2009). Testing the retrieval effort hypothesis: Does greater difficulty
correctly recalling information lead to higher levels of memory? Journal of Memory and
Language, 60(4), 437–447. https://doi.org/10.1016/j.jml.2009.01.004
Rawson, K. A., & Dunlosky, J. (2016). How effective is example generation for learning declarative
concepts? Educational Psychology Review, 28(3), 649–672. https://doi.org/10.1007/s10648-016-
9377-z
Renninger, K. A., & Hidi, S. E. (2019). Interest development and learning. The Cambridge Handbook of
Motivation and Learning, 265–290. Cambridge University Press.
Roelle, J., Nowitzki, C., & Berthold, K. (2017). Do cognitive and metacognitive processes set the stage for
each other? Learning and Instruction, 50, 54–64.
393
Scheiter, K., Fillisch, B., Krebs, M.-C., Leber, J., Ploetzner, R., Renkl, A., Schmidt, H., Schüler, A., &
Zimmermann, G. (2017). How to design adaptive information environments to support self-
regulated learning with multimedia. In J. Buder & F. W. Hesse (Eds.), Informational
environments: Effects of use, effective Designs (pp. 203–223). Springer International
Publishing/Springer Nature. https://doi.org/10.1007/978-3-319-64274-1_9
Schunk, D. H. (1987). Peer models and children’s behavioral change. Review of Educational Research,
Tippett, C. D. (2010). Refutation text in science education: A review of two decades of research.
International Journal of Science and Mathematics Education, 8(6), 951–970.
Udvardi-Lakos, N. ., Endres, T. , Frey, I. , & Renkl, A. (n.d.). Motivating students to reflect - increasing the
motivation to write learning journals in classroom. [Manuscript in Preparation].
Vandevelde, S., Vandenbussche, L., & Van Keer, H. (2012). Stimulating self-regulated learning in primary
education: Encouraging versus hampering factors for teachers. Procedia - Social and Behavioral
Sciences, 69, 1562–1571. https://doi.org/10.1016/j.sbspro.2012.12.099
Weinstein, C. E., & Mayer, R. (1986). The teaching of learning strategies. In M. C. Wittrock (Ed.),
Handbook of research on teaching (pp. 315–327). Macmillan.
Zimmerman, B. J. (2013). From cognitive modeling to self-regulation: A social cognitive career path.
Educational Psychologist, 48(3), 135–147. https://doi.org/10.1080/00461520.2013.794676
394
In Their Own Words Cultivating Greater Spontaneity in Learning
Strategy Use
to Apply the Science of Learning in Emmanuel Manalo
Kyoto University
Editors
Lauren L. Kordonowy
Yuri Uesaka
Victor A. Benassi
The University of Tokyo
Clark A. Chinn
Rutgers University
Introduction
In the past one hundred years, the nature of education has changed dramatically, including the
expectations of what the provision of education ought to enable students to do. It is no longer
considered sufficient for students to simply follow their teachers’ instructions and retain—then
reproduce—the knowledge that those teachers impart. Instead, now, what matters is what students are
able to do with the knowledge they possess. They need to be able to scrutinize and draw inferences
from that knowledge, and to be able to use that knowledge creatively in familiar as well as novel
situations. They need to be able to see important connections in what they learn across different
subjects and knowledge domains, and across locations and time. In addition to these, they also need to
learn how to learn, to find and use information for themselves, to effectively communicate and work
with others, and to think and behave to match their requirements in the environments they occupy. A
fundamental assumption here is that learning is lifelong: it is not limited to the years that one spends in
formal education. Implicit in this assumption is a crucial goal that all students should aim to become self-
regulated learners during their years of schooling.
Self-regulated learning (SRL) refers to learners’ autonomous control in planning, monitoring, regulating,
and evaluating the actions they take toward achieving learning-related goals such as acquiring
information, developing expertise, and improving themselves (Paris & Paris, 2001). Zimmerman (1990)
pointed out that there are three essential features of SRL: the use of SRL strategies, responsiveness to
self-oriented feedback about the effectiveness of learning, and self-motivation. Learners, of their own
volition, need to be able to select and use learning strategies to match their learning requirements and
achieve their desired learning outcomes, taking into consideration feedback available to them about
their learning progress, effectiveness, and skills.
It is, therefore, a serious problem when learners lack spontaneity in using the learning strategies they
ought to use. However, the reality is that many students lack that spontaneity (Borkowski et al., 1987;
Garner, 1990; Manalo et al., 2018). When working on various learning tasks, they tend to use
inappropriate strategies—often ineffective, shallow processing strategies—unless their teachers prompt
them to do otherwise. For example, when preparing for a test, students may simply read and re-read
their textbooks and other learning materials instead of using more effective strategies such as self-
explanation (Chi & Boucher, this volume) and self-testing (Yang et al., this volume). When note taking,
they may just copy verbatim whole portions of reading materials they use with little or no consideration
of their purpose and how they may need to use what they have learned. Many students also take
information they encounter through the internet and other media sources at face value, without use of
critical evaluation and other thinking and reasoning strategies. As the well-known study by Marton and
395
Säljö (1976) showed, even at university, many students tend to rely on shallow-processing strategies in
learning, and consequently fail to grasp and/or retain important meanings of the materials they study.
Part of the problem is that many students lack sufficient knowledge and skills in using some effective
learning strategies: If they do not know enough about those strategies, it is unlikely that students will
even try to use them. There are also a number of other reasons that have been proposed to explain this
problem; we believe it is important for teachers to be aware of those reasons. In this chapter, we will
first summarize some of the key conclusions we drew in our 2018 book, Promoting Spontaneous Use of
Learning and Reasoning Strategies (Manalo et al., 2018), emphasizing suggestions of what teachers can
do to proactively promote spontaneous use of effective strategies. Following that, we will briefly
describe some of our more recent work in this area that provides further insights into what can be done
to cultivate this essential SRL capability and predisposition.
The Necessary Student, Teacher, and Environmental Conditions

The 19 chapters of our 2018 book (written by various expert researchers in the field) indicate that
spontaneous strategy use is supported when a set of conditions is in place during educational
experiences. Those conditions pertain to characteristics of the students themselves, their teachers, and
the learning environments in which they operate.
Student Conditions
Two basic conditions are essential if students are to manifest spontaneity in their strategy use. First,
they must possess sufficient knowledge about the strategies they should be using, and second, they
must be aware of and committed to the value of using those strategies (cf. the other chapters in Part 4
of this volume). For example, if a student is preparing for a test that will require her to explain various
concepts, she has learned in her science class, she has a choice of strategies she can use. She can just re-
read her notes, which is what the majority of students tend to do. However, this method is not likely to
be very effective, as it does not prepare her for what she will need to do during the test, namely,
grasping the meaning of those concepts and writing explanations that make sense of them (not simple
reading!). It would be better if she engaged, for example, in practice testing (e.g., Dunlosky, 2013).
However, practice testing—although more likely to be effective—would also require more effort from
her, so she is unlikely to use it as a strategy unless the two conditions noted above are present.
To begin with, she needs to know about this strategy and exactly how to use it: from looking for and
generating a list of possible concepts she could get asked, then trying to write explanations for those
under self-imposed test conditions (i.e., not looking up the answers), through checking the correctness
of her explanations, and then re-learning—and possibly re-testing—the concepts she forgot or had not
been able to explain well. If she lacks confidence in being able to use this strategy, she will most
probably not use it.
Knowing how to use a strategy, however, is not sufficient on its own for promoting spontaneous use of
this strategy. Students also need to appreciate and be committed to the value of using that strategy.
Continuing with the same example, the student will need to believe that she will likely perform better if
she used the practice testing strategy (cf. other chapters in Part 4 of this volume). If the student is
uncertain about the merits of that strategy, she may not consider it worth the extra effort that will be
involved (i.e., compared to just re-reading her notes, which is much easier). In addition to this, she must
appreciate the value of doing well on the test—enough to invest the effort necessary in using a more
demanding strategy. Appreciating the benefits that will result from actions we take is a fundamental
requirement of becoming motivated to take those actions, as research studies on the expectancy-value
theory have shown (e.g., Muenks et al., 2018; Wigfield & Eccles, 2000).
396
These student conditions have some important and practical implications for teachers. First, if a teacher
considers that students could benefit from using a certain learning strategy, it would be helpful if the
teacher could explicitly introduce the strategy to the students and guide them through its use (e.g.,
Pressley & Harris, 2009). This could be through direct instruction or through learning tasks (and resulting
experiences) that the teacher provides. It is not enough to simply tell students about the strategy; they
need to experience adequate practice in actually using it (e.g., Manalo & Uesaka, 2016).
Second, it would be helpful if the teacher provided opportunities for the students to experience the
benefits that arise from using the strategy. For example, the teacher could provide a practice
assessment (such as a practice test) that would allow the students to experience better performance
following use of the strategy. A teacher could also share research findings supporting the value of the
strategy with students. Furthermore, it is important for the teacher to ensure that students appreciate
the value of what they are learning. Students, for example, need to understand why it might be
important for them to learn those science concepts—such as when, in real life (outside the classroom),
it might come in handy to know those concepts. They need to know what knowing those concepts
would enable them to do in the present or in the foreseeable future.
Teacher Conditions
If students are to manifest spontaneity in their use of effective learning and reasoning strategies, there
are two important conditions for their teachers to meet. First, the teacher needs to be able to teach or
guide the students in the use of strategies that are appropriate for the various learning tasks students
will encounter. Second, the teacher needs to be able to apply pedagogical content knowledge (PCK) in
the design and implementation of the students’ learning environments, which includes not only the
classroom and other teaching spaces but also the materials that are used for teaching and learning
purposes. PCK refers to teachers’ knowledge about how to teach the subject that they teach (Shulman,
1986). It includes teachers’ knowledge about how to make the contents of that subject understandable
and the aspects of that subject that might be easy or difficult for students to learn depending on their
age and background. With respect to strategy instruction, it includes knowledge of how particular
strategies should be tailored to particular subject matter as well as how to teach these subject-specific
strategies.
The first of these conditions, that of the teacher being able to teach/guide students in the use of
effective strategies, is important because even though some students will acquire the use of task-
appropriate strategies on their own, many more students will not. As Garner (1990) concluded, many
students rely on the use of poor or inappropriate strategies for learning, and the use of those strategies
is maintained not because they are somehow effective, but because they do produce some results,
albeit sub-optimal or even poor results. For example, the strategy of simply rereading notes to prepare
for tests would enable students to remember and, hence, write some answers in a test but of course not
as well as they would have managed had they used practice testing. Thus, apart from developing their
students’ content knowledge in respective subject disciplines, teachers also need to be able to cultivate
corresponding learning and reasoning strategies—those useful in that discipline and possibly also
transferrable to other disciplines.
The second condition, that the teacher is able to apply PCK in the design and implementation of the
students’ learning environments, is also essential. If the teacher is able to teach appropriate learning
and reasoning strategies to his or her students, spontaneity in the use of those strategies would likely
not be manifested if the learning tasks, assessments, and materials that the teacher uses do not support
the use of those strategies. For example, teachers could teach or guide their students through the use of
various critical thinking strategies (e.g., consideration of multiple perspectives and alternative
397
views/explanations, provision of supporting evidence and/or reasons for claims being made, etc.),
focusing on those strategies that are particularly useful for the subject matter at hand. However, even if
the students can use those strategies, they may not spontaneously use them if the teacher’s approach
to teaching is purely didactic, differing views are rarely called for in classroom interactions, and
assessments are based solely on the production of ‘correct answers’ (as the teacher has specified). For
the students to spontaneously use those critical thinking strategies, they must perceive a clear need for
them, including in the specific learning tasks, materials, and anticipated assessments that the teacher
will administer.
The teacher conditions noted here have obvious implications for teacher professional development:
teachers need to be better prepared to teach and guide students in strategy use (Dunlosky, 2013;
Manalo et al., 2018). They should be provided training not only in teaching content to their students, but
also in the use of effective methods that promote student learning of that content. Furthermore, it is
crucial that teachers’ capabilities in aligning course objectives, instructional methods, and assessments
are effectively developed through pre-service training and in-service professional development
programs (e.g., Biggs, 1996; DeLuca & Bellara, 2013). Misalignments can lead to many problems in
student learning outcomes, including the lack of spontaneity in use of effective strategies.
Environmental Conditions
Apart from student and teacher conditions, we also explained in our book the need for environmental
conditions to be met if students are to manifest greater spontaneity in their use of effective learning and
reasoning strategies. “Learning environments” here refer to the curriculum, structures of teaching and
learning activities, tasks and problems, materials, and technologies in which students engage during the
process of learning. There are again two conditions: first, that the environment provides opportunities
for student-driven knowledge and skills building, and second, that it clearly supports the value of
strategy use through the instructional methods, activities, and assessments that students are provided.
The first environmental condition of providing opportunities for student-driven knowledge and skills
building is necessary because spontaneity, by its very essence, requires a self-driven or self-initiated
response from students to the learning demand they face. It would be difficult for students to develop
spontaneity in learning and reasoning strategy use if their teachers always impose considerable control
over the knowledge and skills building actions they take. Spontaneous strategy use requires the
application of metacognition (e.g., Borkowski et al., 1987; Garner, 1990; Manalo et al., 2018; Schraw,
2001), the development of which requires experience and practice (e.g., Delclos & Harrington, 1991;
Joseph, 2010). In other words, students need to be given the agency to evaluate and make decisions on
how they will acquire the necessary knowledge and skills—which includes selecting the appropriate
strategies to use, implementing them, monitoring progress, making adjustments, and so on. If their
learning environments do not adequately allow those actions to occur, many students may fail to
develop not only crucial knowledge and skills but also confidence in spontaneous strategy use.
The second condition of the learning environment supporting the value of strategy use is likewise
crucial. Even if students develop the necessary knowledge and skills in the use of effective learning and
reasoning strategies, their spontaneity in using them would likely be constrained if they do not perceive
their environment as genuinely supporting the value of using those strategies. For example, students do
not engage in evaluating sources because they know that teachers do not ordinarily expect that they do
so, except during lessons directed specifically at evaluating sources (Paul et al., 2017). The environment
here includes the methods by which they are provided instruction, the tasks they are given, the activities
in which they are required to engage, and the assessments used to determine their learning success.
Questioning skills, for example, are essential for critical thinking, and students may learn how to ask
398
appropriate questions to facilitate their learning development. However, they are unlikely to use
questioning skills spontaneously (despite their ability to do so) if in the classroom the teacher allows
few, if any, opportunities for questioning (e.g., if the teacher’s or the textbook’s authority is never
questioned). For students to spontaneously use an effective strategy for which they possess knowledge
and skills, they need to perceive that that strategy has genuine value in their learning environment. In
other words, they perceive that their use of an appropriate, effective strategy will lead to desirable or
beneficial outcomes.
What More is Important for Spontaneous Strategy Use?

Since the publication of our 2018 book, we have conducted a number of new studies that have provided
further clarification about how it may be possible to successfully promote students’ spontaneous use of
learning and reasoning strategies. We outline some of those findings here, especially those that may
have a direct bearing on teaching practices and are therefore beneficial for teachers to know. Some of
these studies have, at the time of writing this chapter, only recently been completed or are still in
progress (with only preliminary results), so for those we only refer to presentations we have made at
conferences or other research meetings. The first three subsections summarize recent work by Manalo
and Uesaka, and the third notes work by Chinn and his colleagues.
The Mental Cost of Strategy Use
One important point we have obtained further clarification and evidence for is the explanation that
students’ perceptions of the amount of mental effort required in using strategies affect their
spontaneity in using them. In two earlier papers, we proposed this explanation: when students perceive
that the cost of using a strategy is too high, they tend not to use it (Manalo & Uesaka, 2012; Uesaka &
Manalo, 2012). The mental/cognitive resources demanded by the strategy use may, in fact, exceed what
the student has available. An example of this is when students have to explain complex information in a
foreign language in which they lack fluency. These are the very students who are unlikely to construct a
diagram to supplement their explanation, even though an appropriate diagram would probably help
toward clarification (Manalo & Uesaka, 2012). In some cases, the reluctance to use a strategy may
simply be due to the strategy appearing to require considerable effort. Students might, for example,
choose to draw simple illustrations during their attempts at solving mathematical word problems (i.e.,
they just represent information as they are described in the problem, such as where objects might be
and the distances between them) because producing such representations of the problem is less
demanding of mental effort than to construct more abstract diagrams (e.g., tables and graphs), requiring
more transformations of the information provided in the problem. Thus, fewer students spontaneously
use more abstract diagrams in mathematical word problem solving, even in cases where they may be
the most appropriate and helpful representations to use (Uesaka & Manalo, 2012).
In more recent research, we have been able to demonstrate that providing adequate instruction and
practice on diagram use reduces students’ perceptions of the mental effort associated with such use
(Ayabe et al., 2018). This is congruent with research on cognitive load (e.g., Sweller et al., 1998; Ashman
& Sweller, this volume): that appropriate instruction enables the construction of schemas that cluster
together the pertinent elements of information being learned (e.g., the steps involved in the use of a
strategy), reducing and making more manageable the cognitive load required for the use of that
information. However, what we have also been able to empirically demonstrate is that instruction not
only leads to a reduction in perceived mental effort, but also to an increase in the spontaneous use of
the strategy and more successful task performance (Ayabe et al., 2018). We conducted much of this
research in the use of self-constructed diagrams as a strategy in problem solving and in communication,
so the findings to which we refer here are in this area. For example, providing students with instruction
399
and practice in the construction of diagrams for solving mathematical word problems lead to reports of
perceptual reductions of the mental effort involved in the use of that strategy, leading to greater
spontaneity in using that strategy during problem solving, and increased correct answer rates. We
obtained similar results where the use of diagrams to supplement written explanations in a foreign
language is concerned (Manalo et al., 2020). Fundamentally, these findings support the need for
students to be adequately provided the necessary instruction or guidance in the target strategy use to
alleviate any misgivings they may have about the strategy being too difficult, time consuming, and/or
troublesome to use.
Specific Strategies for Specific Task Requirements
One important point we have recently verified is the need to ensure that students are fully cognizant of
the task features that provide indications of what particular strategies are appropriate for them to use.
In other words, the strategies to use must match the task requirements, and being able to tell when a
strategy is appropriate to use is crucial (i.e., having conditional knowledge; cf. Paris & Paris, 2001;
Koedinger et al., this volume). For example, we can provide general instruction or guidance on the use of
diagrams when attempting to solve mathematical word problems. However, variations exist in such
problems, and students need to be able to construct and use the kinds of diagrams to match the types
of problems presented. When presented with a problem involving quantity changes that follow a certain
rule, for instance, the construction of tables or arrays is helpful in working out the missing or need-to-
predict quantity. On the other hand, when the problem involves patterns of change occurring between
two indices of quantities and a prediction of what will happen (or a reconstruction of what had
happened) needs to be made, constructing a graph would more likely be helpful. Unless students
possess such conditional knowledge—and the corresponding procedural knowledge (in this case, how to
construct and use the appropriate diagrams) —spontaneity in strategy use will not likely occur.
In our research, we found that providing instruction only on the construction and use of one kind of
diagram (e.g., the use of tables) leads to spontaneous and appropriate use of that kind of diagram for
the corresponding type of problem. We found no significant increase in the use of other kinds of
diagrams (e.g., graphs) in attempts at solving other types of problems (Ayabe et al., 2019). The
important point, therefore, is that providing only generic instruction in strategy use would likely present
limitations in improving students' spontaneous strategy use. Students need to discover the content-
specific use of the strategies (i.e., how to use them in real learning tasks they have to do in the
classroom and other contexts), which is the procedural and conditional knowledge in using the strategy,
for spontaneous use to more reliably be promoted.
Rethinking Delivery Methods and Components of Strategy Instruction
Closures of schools and tertiary institutions due to the coronavirus pandemic have made it even more
urgent that educators recognize the importance of developing SRL capabilities in students. Without
teachers directly supervising their learning while they were at home, it became more obvious that
students really need to possess the motivation, knowledge, and skills for spontaneously using
appropriate strategies to satisfactorily complete their learning tasks. Toward this end, we developed
online training programs for promoting students’ ability to effectively use learning strategies, both while
they were learning at home as well as later when they returned to school. These training programs
covered a range of topics, from employing strategies for more meaningful learning (that aids in
retention) to developing effective learning cycles (e.g., effective preparation strategies before class,
learning methods during class, and revision strategies after class, including at home). Feedback from
teachers and students suggests that these programs have been effective in facilitating the development
400
of the intended SRL skills in students, including the spontaneous use of learning strategies that enabled
students to manage their learning during the pandemic period (Uesaka et al., 2021).
There are several other aspects of learning strategy instruction that we have been investigating that
have, to date, delivered positive outcomes. One of those involves the provision of encouragement for
the use of learning strategies. We collaborated with schoolteachers so that after students had received
instruction on the use of various learning strategies, the teachers provided encouragement for students
employing those strategies in daily classroom learning in the context of the subjects they were teaching.
One indication of the success of this approach is that in interviews with students in an English class who
participated in the program, there is clear evidence indicating that students valued those strategies and
incorporated them into their own repertoire of potential strategies (Uchida, 2021).
We have also been investigating the use of scaffolding to promote spontaneous strategy use among
younger learners (Uesaka, 2021). More specifically, we have developed a way of instructing first-grade
elementary school students how to draw and use diagrams (visual representations) in solving
mathematical problems. This instruction focused on scaffolding the children’s use of representations,
from concrete to semi-concrete, and eventually to more abstract forms (diagrams), so that they not only
understand what the drawings stand for but also how their construction helps them in mastering
problem solving. Previous research has shown the value of scaffolding in instructional practices with
young children (e.g., Anghileri, 2006), but not so much in strategy use development. The results in our
study show that spontaneous appropriate diagram use was promoted in the students who received
scaffolded instruction (Uesaka, 2021).
Promoting Spontaneous Transfer to Thinking in a Post-Truth World
The current work by Chinn and his colleagues has focused on how to promote better thinking in a post-
truth world (Barzilai & Chinn, 2018; Chinn et al., 2020a, 2020b). Construed through the lens of this
chapter, this work addresses how to promote spontaneous use of reasoning strategies in the complex
informational world that students encounter out of school. Barzilai and Chinn (2020) and Chinn et al.
(2020b) discussed a variety of challenges posed to good judgment and decision making by the
contemporary information world, which is rife with conflicting claims and misinformation. These
challenges include difficulties with cognitive and metacognitive engagement with such information, as
well as difficulties in adapting reasoning from one situation to another, engaging effectively with social
groups to reason well, and in caring about and enjoying apt thinking in a complex informational world
(Barzilai & Chinn, 2018).
Chinn et al. (2020b) discussed five broad instructional approaches to encourage greater use of effective
thinking in this world. First, classrooms need to present much more of the informational complexity of
the real world. If students have little experience reasoning about very low-quality information, they will
not be able to engage effectively with such information in the real world. Second, students need to learn
to adapt their thinking effectively to different contexts. For instance, they should learn when they have
the specific technical knowledge to evaluate evidence themselves, and when it is better to recognize
their bounded knowledge and defer to experts who have greater competence to evaluate the quality
and quantity of evidence. Third, students should engage in explorations into knowing (see also Chinn et
al., 2020a)— that is, reflecting on good reasoning practices and even investigating which are most
reliable. Fourth, teachers should develop learning environments that promote caring about good
thinking, as well as enjoyment of the effortful processes needed to do it well. Specific instructional
approaches can involve fostering epistemic virtues (Lapsley & Chaloner, 2020) and granting students
epistemic agency to develop their own norms and reasoning practices. Finally, learning environments
need to focus more on how social institutions produce knowledge—including understanding the
401
conditions under which they succeed and fail at doing so. Together, these instructional approaches are
intended to foster the dispositions and competence for students to engage more effectively with the
informational chaos of the modern informational world.
Conclusion and Some Questions on Which to Reflect

Spontaneity in the use of learning and reasoning strategies is clearly crucial to both SRL and lifelong
learning. Thus, its absence in many learners’ methods of engaging with learning tasks is a serious
problem, particularly when considering the demands of the modern, fast-changing, and unpredictable
environments in which we now operate. However, as we have explained in this chapter, research
findings clearly show that it is possible to promote greater spontaneity in students’ use of effective
learning and reasoning strategies—albeit with the proviso that certain student, teacher, and
environmental conditions are met. We outlined those conditions here, based on the main findings
reported in our book dealing with this topic (Manalo et al., 2018), with additional clarifications and
requirements drawn from findings emerging from our more recent research work.
We want to wrap up this this chapter by listing some ‘take home messages’ for readers in the form of
questions to ask and on which to reflect. First, for those who teach, whether at school or at the
college/university level, we think the following questions are pertinent to ask:
• In the courses I teach, what do I consider to be the effective learning and reasoning strategies
that students should be using?
• Do my students know about those strategies and how and when to use them?
• For those students who do not know about effective strategies, how can I provide the necessary
instruction or guidance (including metacognitive instruction)? Or, alternatively, how can I
redesign my course teaching to facilitate students’ acquisition and use of those strategies?
• How can I encourage students’ agency in assuming ownership of these strategies?
• How do I encourage the application of those strategies to the content that students have to
learn in my courses?
• When students make use of those strategies to match the task demands in my courses, does it
lead to beneficial outcomes—such as improved understanding, achievement of learning goals,
and better assessments results? How do I actually know, or how can I find out?
We believe that asking these questions and considering the answers to them can lead teachers to
redesign some of the ways they approach teaching and use assessments—so that those become more
conducive to facilitating appropriate and self-regulated strategy use. Furthermore, it can reveal specific
gaps in teachers’ current knowledge and skills in teaching and promoting strategy use as they apply to
the courses or subject areas they teach. This should not cause alarm or embarrassment as such
knowledge and skills are often inadequately dealt with in pre-service teacher training (e.g., Dunlosky,
2013; Manalo et al., 2018). In the case of college/university teachers, many do not even receive or
partake in any kind of teaching-related training at all before they are required to teach courses (e.g.,
Postareff et al., 2007)!
Second, for readers who work with teachers and/or students in providing various forms of training,
professional development, and/or support, we believe the following questions are important to ask:
• What are the generic and content-specific learning and reasoning strategies that would be
useful for me to cultivate in the teachers or students with whom I work?
• How do I find out their needs as far as training, professional development, or support in strategy
use development is concerned?
402
• What research evidence is there that the strategies I am teaching them are effective when
appropriately used?
• Have I recently updated the strategies I teach based on more recent research evidence that has
emerged? Which kinds of strategies need updating?
• Do teachers I train understand the importance of aligning the classroom values they promote
and assessments they use in order to cultivate their students’ ability and willingness to use
appropriate strategies?
• Do teachers appreciate the role of metacognition and student agency in mastering strategies in
ways that support use in new contexts?
• Do students I train and/or support understand not only what strategies are effective and how to
use them, but also the task characteristics that inform them of when those strategies are
appropriate to use?
• What school or institutional support mechanisms and/or resources might be helpful to provide
to enable teachers (or students) to incorporate use of the strategies in their daily teaching (or
learning) contexts?
Asking these questions would likely be helpful for professional developers/trainers to identify any gaps
in their own knowledge and skills that, if addressed, can make them more effective.
Finally, for researchers in the area of learning and reasoning strategy use, we believe the following
questions would be beneficial to ask:
• What are the most pressing gaps to address in matching learning/reasoning tasks and
appropriate strategies to use?
• How can we promote the transfer—especially ‘far transfer’ (i.e., appropriate use of strategies in
dissimilar tasks or contexts; cf. Salomon & Perkins, 1989 )—of learning and reasoning strategies?
• How can we overcome or circumvent inflexible school/institutional practices that are not
conducive to promoting spontaneity in strategy use?
• How can we incorporate adequate ‘space’ in the curriculum to allow for the cultivation of
learning and reasoning strategies?
• How do learning and reasoning strategies link to each other, and how can we facilitate such
linking? For example, use of deeper learning strategies often leads to application of critical
thinking to tasks being undertaken—and vice versa (Manalo, 2020), so understanding and
promoting such linking could be beneficial.
• What kinds of innovative instruction could enable students to make much more progress in
learning strategies in ways that support use outside of school?
There are many more important questions to address in research and the list above is by no means
comprehensive. As the more recent work we outlined earlier in this chapter suggests, there is a lot more
we need to find out to effectively promote the strategy use component of SRL and lifelong learning. For
now, however, what we know already is that promoting learner spontaneity in effective strategy use is
achievable if the necessary conditions are met.
Author Note
Emmanuel Manalo’s and Yuri Uesaka’s work on this chapter was supported by grants-in-aid (19H00616,
and 20K20516) received from the Japan Society for the Promotion of Science. The section on promoting
403
thinking in a post-truth world is based on ongoing collaborative work among Sarit Barzilai, Ravit Golan
Duncan, and Clark Chinn.
References
Anghileri, J. (2006). Scaffolding practices that enhance mathematics learning. Journal of Mathematics
Teacher Education, 9, 33–52. https://doi.org/10.1007/s10857-006-9005-9
Ayabe, H., & Manalo, E. (2018). Can spontaneous diagram use be promoted in math word problem
solving? In P. Chapman, G. Stapleton, A. Moktefi, S. Perez-Kriz, & F. Bellucci (Eds.),
Diagrammatic representation and inference. Diagrams 2018. Lecture Notes in Computer Science,
vol. 10871 (pp. 817–820). Springer. https://doi.org/10.1007/978-3-319-91376-6
Ayabe, H., Manalo, E., & Hanaki, N. (2019). Teaching diagram knowledge that is useful for math word
problem solving. Proceedings of the EAPRIL (European Association for Practitioner Research on
Improving Learning) 2019 Conference (pp. 388–399). European Association for Practitioner
Research on Improving Learning. https://eapril.org/sites/default/files/2020-
04/Proceedings2019_3.pdf
Barzilai, S., & Chinn, C. A. (2020). A review of educational responses to the “post-truth” condition: Four
lenses on “post-truth” problems. Educational Psychologist, 55(3), 107-119.
Biggs, J. (1996). Enhancing teaching through constructive alignment. Higher Education, 32, 1-18.
Borkowski, J., Carr, M., & Pressley, M. (1987). “Spontaneous” strategy use: Perspectives from
metacognitive theory. Intelligence, 11(1), 61–75. https://doi.org/10.1016/0160-2896(87)90027-
4
Chinn, C. A., Barzilai, S., & Duncan, R. G. (2020a). Disagreeing about how to know: The instructional
value of explorations into knowing. Educational Psychologist, 55(3), 167-180.
Chinn, C. A., Barzilai, S., & Duncan, R. G. (2020b). Education for a “post-truth” world: New directions for
research and practice. Educational Researcher, 52(1), 51-60.
Delclos, V. R., & Harrington, C. (1991). Effects of strategy monitoring and proactive instruction on
children's problem-solving performance. Journal of Educational Psychology, 83(1), 35–42.
DeLuca, C., & Bellara, A. (2013). The current state of assessment education: Aligning policy, standards,
and teacher education curriculum. Journal of Teacher Education, 64(4), 356–372.
Educator, 37(3), 12–21.
Garner, R. (1990). When children and adults do not use learning strategies: Toward a theory of settings.
Review of Educational Research, 60(4), 517–529. https://doi.org/10.2307/1170504
404
Joseph, N. (2010). Metacognition needed: Teaching middle and high school students to develop
strategic learning skills. Preventing School Failure, 54(2), 99–103.
Lapsley, D., & Chaloner, D. (2020). Post-truth and science identity: A virtue-based approach to science
Manalo, E., Chen, O., & Wang, J. (2020). Using diagrams when writing explanations enables EFL students
to convey more important points. Proceedings of the 37th JSET (Japan Society for Educational
Technology) Conference (pp. 563–564). Japan Society for Educational Technology.
Manalo, E., & Uesaka, Y. (2012). Elucidating the mechanism of spontaneous diagram use in explanations:
How cognitive processing of text and diagrammatic representations is influenced by individual
and task-related factors. In P. Cox, B. Plimmer, & P. Rodgers (Eds,), Diagrammatic representation
and inference. Diagrams 2012. Lecture Notes in Computer Science, 7352 (pp. 36–50). Springer.
https://doi.org/10.1007/978-3-642-31223-6_9
Manalo, E., & Uesaka, Y. (2016). Hint, instruction, and practice: The necessary components for
promoting spontaneous diagram use in students’ written work? In M. Jamnik, Y. Uesaka, & S.
Elzer Schwartz (Eds.), Diagrammatic representation and inference. Diagrams 2016. Lecture
Notes in Computer Science, 9781 (pp. 157–171). Springer. https://doi.org/10.1007/978-3-319-
42333-3_12
Manalo, E., Uesaka, Y., & Chinn, C. A. (Eds.) (2018). Promoting spontaneous use of learning and
reasoning strategies: Theory, research, and practice for effective transfer. Routledge.
Marton, F., & Säljö, R. (1976). On qualitative differences in learning: I. Outcome and process. British
Journal of Educational Psychology, 46(1), 4–11. https://doi.org/10.1111/j.2044-
8279.1976.tb02980.x
Muenks, K., Wigfield, A., & Eccles, J. S. (2018). I can do this! The development and calibration of
children’s expectations for success and competence beliefs. Developmental Review, 48, 24–39.
https://doi.org/10.1016/j.dr.2018.04.001
Paris, S. G., & Paris, A. H. (2001). Classroom applications of research on self-regulated learning.
Educational Psychologist, 36(2), 89–101. https://doi.org/10.1207/S15326985EP3602_4
Paul, J., Macedo-Rouet, M., Rouet, J.-F., & Stadtler, M. (2017). Why attend to source information when
reading online? The perspective of ninth grade students from two different countries.
Computers & Education, 113, 339-354. https://doi.org/10.1016/j.compedu.2017.05.020
Postareff, L., Lindblom-Ylänne, S., & Nevgi, A. (2007). The effect of pedagogical training on teaching in
higher education. Teaching and Teacher Education, 23(5), 557–571.
https://doi.org/10.1016/j.tate.2006.11.013
Pressley, M., & Harris, K. R. (2009). Cognitive strategies instruction: From basic research to classroom
instruction. Journal of Education, 189(1–2), 77–94. https://doi.org/10.1177/0022057409189001-
206
Salomon, G., & Perkins, D. N. (1989). Rocky roads to transfer: Rethinking mechanisms of a neglected
phenomenon. Educational Psychologist, 24(2), 113–142.
405
Schraw G. (2001) Promoting general metacognitive awareness. In H. J. Hartman (Ed.), Metacognition in
learning and instruction. Neuropsychology and Cognition, 19 (pp. 3–16). Springer.
https://doi.org/10.1007/978-94-017-2243-8_1
Shulman, L. S. (1986). Those who understand: Knowledge growth in teaching. Educational Researcher,
15(2), 4–14. https://doi.org/10.3102/0013189X015002004
Sweller, J., van Merrienboer, J. J. G., & Paas, F. G. W. C. (1998). Cognitive architecture and instructional
design. Educational Psychology Review, 10(3), 251–296.
https://doi.org/10.1023/A:1022193728205
Uchida, N. (2021). Reconsidering English vocabulary instruction from a psychological perspective:
Investigation of English teachers’ instruction and suggestion of a practice (Working Paper Series
in Young Scholar Training Program). Center for Advanced School Education and Evidence-Based
Research, The University of Tokyo. https://www.schoolexcellence.p.u-tokyo.ac.jp/wp/wp-
content/uploads/2021/06/5fbd7f4d387092ede2c926d413b1b608.pdf
Uesaka, Y. (2021, August 23–28). Developing abstraction and diagram use skills for solving math
problems in early elementary school [Paper presentation]. The 19th Biennial EARLI (European
Association for Research on Learning and Instruction) Conference.
https://earli.org/assets/files/BOA-2021.pdf
Uesaka, Y., & Manalo, E. (2012). Task-related factors that influence the spontaneous use of diagrams in
math word problems. Applied Cognitive Psychology, 26(2), 251–260.
Uesaka, Y., Uchida, N., Saso, S., Shiba, S., Ota, E., Liu, M., Mizuno, Y., and Tomita, M. (2021, Oct 16–17).
Development of an "online-based learning skills enhancement program" to promote self-
regulated learning: Instructional devices to suit on-line learning, and effects of the program
[Poster presentation]. Annual Autumn Conference of JSET (Japan Society for Educational
Technology), Kita-Kyusyu, Japan.
Wigfield, A., & Eccles, J. S. (2000). Expectancy–value theory of achievement motivation. Contemporary
Educational Psychology, 25(1), 68–81. https://doi.org/10.1006/ceps.1999.1015
Zimmerman, B. J. (1990). Self-regulated learning and academic achievement: An overview. Educational
Psychologist, 25(1), 3–17. https://doi.org/10.1207/s15326985ep2501_2
406
In Their Own Words The Science of Learning Initiative at Colorado
State University
to Apply the Science of Learning in Anne M. Cleary and Matthew G. Rhodes
Colorado State University
Editors
Lauren L. Kordonowy
Victor A. Benassi
Unique Challenges to Applying the Science of Learning

The first author of this chapter (AC) was once asked by a professor in a school of engineering: “Isn’t
learning just intuitive? Wouldn’t research on how to help people learn just confirm what we already
know from intuition?” The answer to this question is a firm “no.” Learning is not intuitive, and the best
strategies for effective and efficient learning are often those that feel the least to the learner like they
are working (e.g., Kornell & Bjork, 2007; Hartwig & Dunlosky, 2012; Holm et al., 2020; McCabe, 2011;
Morehead et al., 2016; Rhodes et al., 2020).
The fact that learning can seem to a layperson like it should be intuitive while not actually being so
presents a set of unique challenges toward applying the science of learning in an effort to enhance and
optimize learning in educational and training settings, both in and out of the classroom. First, students
and instructors alike may not immediately think that they need to change anything about how they are
going about learning—they may believe that they already have a good understanding of how people
learn and what is and is not beneficial for learning (cf. Hunter & Lloyd, 2018; Morehead et al., 2016).
Second, if efficient and effective strategies seem to students like they are not so in terms of the feeling
one gets while engaged in them (and worse, if ineffective, inefficient strategies feel like they must be
working), students may be reluctant to engage in the effective strategies for long enough to observe
their benefits. In such cases, even if they are told which learning strategies are effective and which are
not, they may have the impression that the strategies apply to other people but not themselves—that
somehow, they learn differently.
A possible reason why students might be quick to believe that they learn differently is that many
students coming into college believe that there are particular learning styles (e.g., a visual vs. verbal
style), and that adherence to one’s particular learning style is a factor that is important to how well one
learns (see Rhodes et al., 2020, for a review). Despite the pervasiveness of the belief in the importance
of learning styles to learning (e.g., Howard-Jones, 2014; Morehead et al., 2016), the notion of learning
styles is not supported by scientific evidence (e.g., Massa & Mayer, 2006; see Pashler et al., 2009, or
Willingham et al., 2015, for reviews). As an example of the prevalence of the belief in learning styles
among college students, a first day survey of students taking a Science of Learning course at Colorado
State University (see more on the course below) revealed that 81.5% of the students endorsed (with
either “agree” or “strongly agree”) the statement “Adherence to a student’s learning style (such as
visual learner, auditory learner, etc.) is very important to effective learning.” This means that the vast
majority of students entering this class endorsed the importance of learning styles when it comes to
learning. Note that a belief in learning styles can carry with it an impression that different people learn
differently and that not all study and learning strategies apply to all people. This belief can potentially
407
stand in the way of consistently and persistently engaging in effective study strategies, particularly if
those strategies do not feel in the moment to the student like they are working.
Strategies Underway at Colorado State University

There are currently several strategies underway at Colorado State University (CSU) attempting to
address these unique challenges and apply the science of learning to enhance student learning. These
efforts fall into two general categories: 1. courses aimed at teaching students learning strategies that
are grounded in science, and 2. devising ways of increasing student engagement in effective, science-
based strategies outside of the classroom.
Courses Aimed at Teaching Students Effective Learning Strategies

A 3-Credit Course on Science of Learning
Both authors of the present chapter were instrumental in developing a 3-credit, 100-level all-university
course at CSU (Science of Learning, PSY 152) aimed at teaching students to look toward science for
guidance on how to learn, and to begin applying specific, science-based strategies and techniques in
their everyday learning habits. To support the course, the authors co-wrote a textbook, A Guide to
Effective Studying and Learning: Practical Strategies from the Science of Learning (Rhodes et al., 2020).
The course has four key parts to it: 1. address the disconnect between preexisting student beliefs about
learning and what the science suggests, 2. address the disconnect between student impressions of their
learning while engaged in a learning technique and their actual learning outcomes, 3. cover what the
science suggests helps learning, and 4. equip students with strategies and techniques that are grounded
in that science and that they begin applying and using immediately.
To address the disconnect between students’ impressions of their learning and what the science
suggests, the course starts with a survey that helps to highlight prominent myths about learning and the
fact that many students bring beliefs in these myths with them to the class. An example is the
aforementioned survey statement about learning styles that 81.5% of students in a particular course
endorsed on the first day of class. Discussion of these survey points is followed by a presentation of
research on this issue. For example, in the case of learning styles, we cover the research on learning
styles (e.g., Massa & Mayer, 2006; see Pashler et al., 2008, or Willingham et al., 2015, for reviews) and
why the science does not support their importance in learning. Students are often surprised to learn
this, as many students grew up learning about the importance of learning styles, which is evidenced in
part by the aforementioned Day 1 survey results whereby 81.5% of students entering the course
endorsed a statement about the importance of adherence to one’s learnings styles to actual learning—
the students enter the class believing in the importance of learning styles because they have learned
this elsewhere prior to taking the class; also, Morehead et al. (2016) reported that 77% of their student
respondents believed that people had different learning styles. Highlighting this type of disconnect
between the students’ preexisting beliefs about learning and the science serves as a useful springboard
to begin to discuss the science itself, and why it is important to look toward science for guidance on
what helps learning.
Discussing the science itself begins by addressing the disconnect between students’ impressions of their
own learning while they are in the midst of learning something versus their actual learning outcomes as
measured afterward, which helps to further the goal of teaching students to rely on the science rather
than on intuition or impressions. Students engage in active demonstrations that illustrate a disconnect
between their impressions of their learning versus their actual learning outcomes, and then are shown
the results of research studies. One example is the disconnect between student impressions of spacing
versus massing (i.e., cramming) and the actual learning outcomes of these two strategies. Research has
408
consistently shown that people perceive themselves to be learning more from massing than spacing
information, when the reverse pattern is found for actual learning outcomes (King et al., 1980; Kornell,
2009; Kornell & Bjork, 2008). Research has also shown that people tend to regard restudying (e.g.,
rereading) information to lead to more learning than being tested on the same information, even
though testing is a more effective method of learning than restudying alone (e.g., Kornell & Son, 2009;
Roediger & Karpicke, 2006; Roediger & Karpicke, 2018; Rowland, 2014; Yang et al., this volume). Other
examples include erroneous impressions of enhanced learning based on perceived font size (Rhodes &
Castel, 2008) or loudness of words (Rhodes & Castel, 2009) or in-the-moment answer retrievability
(Benjamin et al., 1998). A major goal in carrying out in-class demonstrations on these types of
disconnects and reviewing the research on them is to help students learn to place their confidence in
more effective learning strategies than they would likely otherwise come to on their own (e.g.,
Anthenien et al., 2018; Blasiman et al., 2017; Hartwig & Dunlosky, 2012; McCabe, 2011; Morehead et al.,
2016; Yan et al., 2016).
One objective of the first two key parts of the course is to gain buy-in from the students that the science
of learning matters—that they should look toward the science for information on what helps learning.
The next major component of the course involves delving into what the science actually suggests helps
learning, topic by topic. The science itself and its applications are covered according to different strategy
topics (e.g., spacing, testing, metacognition, elaboration, imagery, organization, etc.), and many
domains of learning are covered. For example, a sample topics list from a course syllabus is:
Week 1: Introduction, Intuitions about Learning
Week 2: Critical Thinking & The Importance of Science
Week 3: Spacing: Space it Out!
Week 4: Testing: Do it & Do it Often!
Week 5: Metacognition: How can we Better Monitor our Own Learning?
Week 6: Elaboration: Make Connections!
Week 7: Imagery: Picture it!
Week 8: Organization: Give it Hierarchical Structure!
Week 9: Forgetting
Week 10: Cues: The Key to Getting Information Out of Memory
Week 11: Collaboration: The Good & The Bad
Week 12: Sleep: The Silent Learning Aid
Week 13: Understanding: Making Giant Mental Leaps
Week 14: Creativity & Innovation
The final key objective of the course is to teach students strategies that they can immediately start using
in their coursework. Thus, as each strategy topic is covered, hands-on ways of applying that strategy are
also taught. As part of the effort to encourage students to begin applying each strategy as it is learned,
students are expected to develop and continue to expand a Study Strategies Portfolio throughout the
course. As new study strategies are learned, students continue to add to the ways that they are applying
that strategy in their own coursework to their Study Strategies Portfolio. The portfolio can include
images, such as a picture or screenshot of a study schedule for showing how spacing is being
implemented, or an image of a concept map that was drawn for elaboration in a course, or of a
hierarchical diagram that was drawn for implementing organization in a course. A first draft of the Study
Strategies Portfolio is due at the midway point of the semester; feedback is given on the draft, and a
409
final draft is due at the end of the semester that is responsive to the feedback on it while also including
the new strategies covered since that first draft was written.
Throughout the course, class time is also used for engaging in specific study strategies such as drawing
concept maps (Blunt & Karpicke, 2014; Grimaldi et al., 2015; Karpicke, 2018), drawing a hierarchical
structure (Bower et al., 1969), engaging in various forms of practice testing (McDaniel et al., 2007;
Rawson et al., 2015; Roediger & Karpicke, 2006; Rowland, 2014) or testing oneself on information, such
as through writing a Letter to Grandparents on a topic from a class (Cleary & Myers, 2019), and
explaining topics in one’s own words (Bisra et al., 2018) or by taking on the role of the “instructor” for
another student (Nestojko et al., 2014; Fiorella, this volume). Also “stop and jot” is used and
encouraged, whereby the lecture is paused, and students are asked to jot down everything they can
remember from the past 10 minutes or so (Szpunar et al., 2013). This is useful and encouraged in online
versions of the course as well (Szpunar et al., 2013), whereby students are encouraged to pause a video
lecture and engage in “stop and jot.”
Whether the course leads to better subsequent academic performance in other classes or a higher
likelihood of retention is currently being investigated. However, it is apparent that the course is
successful in changing students’ basic beliefs about learning. This has been shown, for example, by
comparing survey responses solicited on the first day of the course with those on the last day of the
course. Whereas 81.5% of students endorsed the aforementioned statement, “Adherence to a student’s
learning style (such as visual learner, auditory learner, etc.) is very important to effective learning,” on
the first day of class, that number was reduced to 8.82% on the last day of class. In short, despite
entering the class with that basic belief, most students no longer held that belief by the end of the class.
For another survey item, “Brain training games are a good way to sharpen your mind to become a better
learner,” 47% of students endorsed it on Day 1 whereas only 4.4% endorsed it on the final day of the
course. (See Redick et al., 2013, Redick et al., 2015, and Shipstead et al., 2012, for examples of the lack
of evidence that brain training games help people to generally become better thinkers or learners.) As
one more example, the survey item “Repeating information over and over again is an effective way to
commit it to memory,” received 55% of students’ endorsements on the first day but only 4.4% of
students’ endorsements on the final day. (See Rhodes et al., 2020, for a review of how and why this
strategy is ineffective and examples of strategies that are more effective, such as elaboration, imagery,
testing, and retrieval practice.) Of course, these pre-post survey results might be subject to demand
characteristics, whereby students know what the survey designer is hoping the responses will be and
respond accordingly. Therefore, it would be ideal to follow-up after a longer time period, such as six
months or a year following the course. It would also be ideal to examine learning outcomes themselves,
such as grades in classes for semesters that follow the course, and we hope to pursue this.
That said, changing students’ beliefs about learning is only part of the effort. Another important part of
helping students to learn more efficiently and more effectively is getting them to actually engage in the
types of strategies that work. One of the components of the Study Strategies Portfolio is to evaluate the
obstacles to implementing effective study strategies. A common obstacle identified by students is
difficulty implementing spacing. Students often express having difficulty developing and adhering to a
spaced study schedule, and often state that procrastination or simply being too busy prevents them
from spacing out their studying (see also Blasiman et al., 2017). This has prompted us to test some of
the strategies discussed below in the section Increasing Student Engagement in Effective Strategies
Outside of the Classroom.
410
A 1-Credit Intervention
Through a college-level funding initiative, the first author of this chapter (AC) spearheaded the
development of a 1-credit intervention intended to help students improve their grade in a particular
course. The specific course is Principles of Human Physiology (BMS 300)—an upper level biomedical
sciences course required of all Psychology majors at CSU. The course has a DFW (D-F-Withdrawl) rate
well above most courses in the major and has a reputation among students as one of the most difficult
courses in the Psychology curriculum. This 1-credit intervention for BMS 300 differs from the above-
mentioned Science of Learning course in being much briefer, more condensed, and even more hands-
on. Students register for the intervention concurrently with BMS 300. Therefore, there is an immediate
need, starting on the first day, to begin learning strategies to apply in BMS 300 as the course is
unfolding. To try to meet this very immediate need, the intervention is designed to present students
with more of an emergency kit of tools than a thorough background on the science of learning and its
applications. The background science itself is limited to a roughly 5-10 minute overview of the basis of
the strategy at hand each week. Then, the rest of the 50 minute meeting period is spent engaging in the
strategy at hand. For example, if the day’s strategy is learning to write a Letter to Grandparents (as a
means of retrieving what one knows and understands about a topic, elaborating on what one knows,
and assessing gaps in one’s knowledge, described below), the time period will be spent having students
work on writing this letter. The letter activity is then followed by completing a Reflections worksheet in
which students reflect on the experience of the writing of the letter and the gaps they were able to
identify in their knowledge from the process of writing the letter (described below). Students then write
a plan for filling those identified gaps that becomes part of their study plan for their current BMS 300
coursework.
In addition to engaging in this type of activity in each 1-hour meeting per week with the intervention
leader, students also meet for an hour a week with a peer mentor. The peer mentor has the students
spend their meeting time engaging in the techniques from the growing repertoire of strategies that the
students are accumulating. Because spacing is one of the most effective learning strategies (e.g.,
Carpenter et al., 2012; Cepeda et al., 2008), not only is it taught on Day 1 of the intervention by
developing a spaced study schedule, but spacing is actively practiced as part of the intervention
strategy. Actual spaced repetition of engaging with course material outside of the BMS 300 class time
occurs two times per week—once during the 1-hour meeting with the intervention leader, and once
during each one-hour meeting with the peer mentor. Thus, students receive two different spaced hours
of study with the BMS 300 course material each week by engaging with the study strategies that serve
as the foundation of the intervention course. These spacing sessions are intended to help offset the
common student problem of not being able to independently space out their study efforts.
Rather than a textbook, this intervention uses the manual Course Grade Rescue Kit: A Rapid Science-
based Learning Intervention (Cleary & Myers, 2019), which contains brief background descriptions of the
logic behind each technique, a one-page sheet summarizing the what-why-how of the technique at
hand, and worksheets for engaging in the technique. For example, one set of worksheets contains the
Letter to Grandparents instructions followed by a Reflections worksheet. The Letter to Grandparents
instructions in the worksheet are as follows:
411
In the Reflections worksheet that follows the Letter to Grandparents, the student is asked to answer the
following questions: What gaps in your knowledge were you able to identify through this exercise?
Based on these gaps, where do you think your efforts should be placed in a targeted review of the
material? What is your plan for making sure that you fill these gaps in your knowledge? If you cannot
find an answer in your targeted review, where or to whom can you go to find the answer?
Other exercises are similarly followed by these Reflections questions. For example, an exercise called
Draw it Out asks students to draw a diagram depicting a physiological process that is currently being
covered in the Principles of Human Physiology course, followed by the Reflections questions. Both the
Letter to Grandparents and the Draw it Out exercises are methods of implementing testing, retrieval
practice, elaboration, and the Delay & Test method of monitoring one’s own learning (Rhodes et al.,
2020). Although the 1-credit intervention was designed to help Psychology majors succeed in Principles
of Human Physiology, it can be applied to almost any course as an emergency intervention strategy.
Moreover, this particular intervention is designed so that the person leading it does not need to have
content expertise in the subject area for which the intervention is needed. With the structure and tools
provided in the manual (Cleary & Myers, 2019), anyone can implement the intervention and for any type
of course in which a student may be struggling. Data collection on the effectiveness of this intervention
for improving course grades is ongoing.
Increasing Student Engagement in Effective Strategies Outside of the

Classroom
As mentioned earlier in this chapter, students in the 3-credit Science of Learning course commonly
indicate having the most difficulty with spacing as a study strategy-specifically, between-session spacing,
whereby spacing is spread out across sessions and days. The reason is that this type of spacing requires
some degree of behavior change on the part of the student: The student has to shed old habits and
attempt to develop and follow a spaced study schedule, beginning studying well in advance of a test,
and continuing a spaced structure over time even once the information feels like it is known or even
when the person’s schedule fills up and it feels as if there is no time to stick to this schedule. The
requirement for such behavior change presents a novel challenge relative to the other study strategies
taught in both the 3-credit Science of Learning course and the 1-credit rapid intervention, as it is about
more than simply knowing what to do; spacing requires substantial behavior change on the part of many
students.
Simply teaching people about how to change their behavior or about potentially beneficial changes that
they can make to their routines or habits is typically ineffective at eliciting actual behavior change (e.g.,
Johnson & Goldstein, 2003; Thaler & Sunstein, 2009). That is, just as knowledge that physical activity is
part of a healthy lifestyle does not necessarily lead individuals to adopt a consistent regimen of exercise,
simply knowing about effective study strategies does not inevitably lead students to adopt those
412
strategies. More is typically required in order to influence behavior change—something over and above
simply having the right information about what to do. Below, we detail two different approaches that
we have been investigating and testing at CSU as possible ways of increasing student engagement in
spaced study strategies outside of the classroom.
Smartwatches for Automatizing Spaced Testing and Exposure Outside of the
Classroom
One way to increase optimal behaviors is to create circumstances that make it easy to engage in them,
such as through automatization or default settings, so as not to create barriers for the person in
question (e.g., Johnson & Goldstein, 2003; Thaler & Sunstein, 2009). The idea that smartwatches might
be useful for this purpose outside of the classroom—for automatizing spaced studying—came to the
first author of this chapter (AC) while taking a course in conversational Chinese that was meant to help
prepare a delegation of CSU faculty for an upcoming trip to China. The course was not going well insofar
as the typical faculty demands of teaching courses, engaging in service, running a research lab, and
carrying out research and publishing articles were preventing any serious attention to studying the
material and practicing each week. It suddenly became apparent that this predicament was similar to
that expressed by many students in the 3-credit Science of Learning class when listing their obstacles to
applying effective learning strategies in their everyday habits: There simply was not enough time to
space out studying. Many students have full time jobs, heavy course loads, family responsibilities, etc.,
and have expressed that there is not enough time to devote to studying or to fit spacing in to an already
packed schedule.
In brainstorming what might work for achieving some degree of spaced testing (e.g., Rawson et al.,
2013) and retrieval practice (e.g., Karpicke & Blunt, 2011; Kornell et al., 2011), the first author started to
integrate her smartwatch into her learning. Toward this end, she typed specific test questions into the
calendar app on her computer (which interfaced with the watch to send alarms for each calendar item),
and set these questions to appear at specific times—along with a haptic buzz to the wrist as an alert to
the question’s presence on the watch face—during the walk to one campus building from another in
between classes. The prompts (e.g., “How do you say ‘to run’ in Mandarin?”) would appear every couple
of minutes during the walk, prompting consideration of what the answer might be, then followed a
minute later with a prompt containing the answer (e.g., “pao bu”). This technique succeeded in
improving personal vocabulary performance in the conversational Chinese class despite devoting no
dedicated study time to that course other than the watch face prompts that occurred during walks to
classes. This insight provided the impetus for an empirical study carried out by Cleary et al. (2021) on the
potential effectiveness of smartwatches for the automatization of spaced testing outside of the
classroom.
Cleary et al. (2021) reported five experiments showing that receiving prompts delivered via a
smartwatch during the interval between initially reading scientific passages and later being tested on
them increased test performance, even across a delay of two days. In their study, Cleary et al. compared
being periodically prompted on the watch face with mere re-statements of facts that had appeared in
the original passage (i.e., restudying) with instead being prompted on the watch face with a test
question urging an attempt at retrieval about that fact, followed a minute later with the answer (i.e.,
testing with feedback). Results generally showed that prompting with test questions followed by
feedback led to even better final test performance than prompting that involved mere restudying
(although both forms of prompting led to better final test performance than no prompting at all). Cleary
et al. also examined whether these benefits could be realized when smartwatch prompts occurred in the
midst of a compelling primary task, such as reading a magazine, engaging with one’s own phone, or even
watching a favorite TV show. Indeed, the benefits of the smartwatch prompt occurred regardless of the
413
type of primary task. This finding suggests that, like the earlier anecdotal example of being prompted
while walking across campus, automatized watch prompts can occur during any number of types of
tasks and still benefit learning. Thus, students could decide to be prompted with watch prompts to
reinforce their learning outside of the classroom during any number of non-study-time activities. For
example, they could deliberately set the watch prompts to occur while watching an evening TV episode,
and this should confer a learning and retention benefit over simply spending the evening watching TV
without being prompted. As another example, they could set prompts to occur during walks to and/or
across campus. Future research should examine the potential benefits of smartwatch prompting in
actual classroom settings and should explore and test the possibility of apps that can automatize the
process even further for students. As not all students have access to smartwatches, an alternative could
be to use a cellphone, if one has such a device. So long as the person keeps the phone nearby and
engages with it each time a prompt appears, this should confer similar benefits to what were shown in
the smartwatch study (though this has yet to be directly investigated).
A Learning Management System Tool for Incentivizing Optimal Spacing Behavior
Another approach being investigated at CSU is the potential use of a Learning Management System tool
for incentivizing useful spacing behaviors on the part of students outside of the classroom. The tool,
called U-Behavior, was developed through CSU’s Center for the Analytics of Learning and Teaching (C-
ALT) and has shown promise in initial investigations of its use in classes (McKenna et al., 2020). First,
students are encouraged to space out their attempts at taking online quizzes through the Learning
Management System. They are further encouraged not to simply stop taking a quiz once they get the
high score, as research suggests that it is beneficial to learning to keep testing over time, even after
getting an answer right (e.g., Karpicke & Roediger, 2008; Rawson et al., 2013; see also Dunlosky et al.,
this volume, on successive relearning). Thus, they should not only space out their quiz-taking efforts, but
they should retake quizzes in a spaced fashion even after they have reached the highest possible score
on them. The tool is used to provide students with a visualization of what their online quiz-taking
behavior looks like. Specifically, the students are periodically presented with a plot of when and how
often they took each quiz, and what the scores were each time (with a unique colored symbol
representing each particular quiz, the quiz score on the Y axis, and the date that it was taken on the X
axis). Thus, if a student is massing their quiz-taking such that all quizzes are taken in quick succession
right before an upcoming exam, this will be apparent in the visualization. Along the same lines, if a
student is spacing, this will be apparent in the visualization. Furthermore, if a student is engaging in the
practice of not retaking a quiz again once the high score is attained, this will be apparent in the
visualization as well. Students are shown this visualization and provided with feedback on how to attain
an optimal visualization of behavior through their actual quiz-taking behaviors. As McKenna et al. (2020)
report, initial findings on this method are very promising. Students given the visualizations showed
behavior more consistent with the optimal behavior pattern than when not given the visualizations. This
tool is in the process of being incorporated into the aforementioned Science of Learning class at CSU.
Summary and Future Directions

Reports from students and survey data from our courses paint a consistent picture: Students frequently
come to the university with little knowledge of effective study strategies, although they may be largely
unaware that they lack this knowledge. Although most institutions require students to have
demonstrated at least rudimentary mathematical and writing skills, we offer that skills for effective
lifelong learning are just as important; these skills, however, receive far less attention. We have
attempted to remedy this imbalance at CSU by implementing multiple courses that provide students
background and training on applying effective study strategies. We continue to pursue this as a research
414
endeavor and to consider methods of creating less ‘friction’ for students in applying these strategies
(e.g., Cleary et al., 2021). Indeed, these activities have led to a host of questions that we are only
beginning to address or seek to address in the future:
• Does training the science of learning lead to sustained transfer of skills and knowledge across
courses, time, settings, majors, etc.? (See the chapter by Manalo, et al., this volume for an in-
depth consideration of this question.)
• To what degree do faculty practices in course settings influence whether students implement
effective strategies?
• Does training help students from across all levels of student achievement, motivation, and
background?
• Can training in effective study strategies be doled out (spaced) over time or must it occur in a
single course? Similarly, would students exposed to study strategy training benefit from
receiving periodic ‘boosters’?
• To what extent can learning be reinforced outside of the classroom through the regular use of
smartwatch or smartphone prompting, such as through a specialized app for that purpose?
• Can students be trained to habitually engage in optimal study behavior outside of the
classroom? (see chapters in Part 4 of this volume)
In our own work, and the in the work represented in this volume, we believe that the answers have the
potential to positively influence the future of undergraduate education.
Author Note
The authors are grateful to the Colorado State University administration, including the Office of the
Provost and Vice President, the College of Natural Sciences, and the Department of Psychology, for
supporting the development of the course and interventions described in this chapter. We are also
grateful to the Center for the Analytics of Learning and Teaching (C-ALT) at Colorado State for
supporting the smartwatch study described in this chapter. Correspondence concerning this article
should be addressed to Anne M. Cleary, [email protected], Department of Psychology,
Colorado State University, Fort Collins, CO 80523
References
Anthenien, A. M., DeLozier, S. J., Neighbors, C., & Rhodes, M. G. (2018). College student normative
misperceptions of peer study habit use. Social Psychology of Education, 21(2), 303-322.
https://doi-org.unh.idm.oclc.org/10.1007/s11218-017-9412-z
Benjamin, A.S., Bjork, R.A., & Schwartz, B.L. (1998). The mismeasure of memory: When retrieval fluency
is misleading as a metamnemonic index. Journal of Experimental Psychology: General, 127(1),
55-68. https://doi-org.unh.idm.oclc.org/10.1037/0096-3445.127.1.55
Bisra, K., Liu, Q. Nesbit, J.C. et al. (2018). Inducting self-explanation: A meta-analysis. Educational
Psychology Review, 30, 703-725. https://doi.org/10.1007/s10648-018-9434-x
Comparing intended versus actual study behaviour. Memory, 25(6), 784-792. https://doi-
org.unh.idm.oclc.org/10.1080/09658211.2016.1221974
415
Blunt, J. R., & Karpicke, J. D. (2014). Learning with retrieval-based concept mapping. Journal of
Educational Psychology, 106(3), 849-858. https://doi-org.unh.idm.oclc.org/10.1037/a0035934
Bower, G.H. , Clark, M.C., Lesgold, A.M., & Winzez, D. (1969). Hierarchical retrieval schemes in recall of
categorized word lists. Journal of Verbal Learning and Verbal Behavior, 8(3), 323-343.
Carpenter, S. K., Cepeda, N. J., Rohrer, D., Kang, S. H., & Pashler, H. (2012). Using spacing to enhance
diverse forms of learning: Review of recent research and implications for instruction.
Educational Psychology Review, 24(3), 369-378. https://doi.org/10.1007/s10648-012-9205-z
Cepeda, N.J., Vul, E., Rohrer, D., Wixted, J.T., & Pashler, H. (2008). Spacing effects in learning: A temporal
ridgeline of optimal retention. Psychological Science, 19(11), 1095-1102.
Cleary, A.M., McNeely-White, K.L., Hausman, H., Dawson, J., Kuhn, S., Osborn, R.M., Huebert, A.M., &
Rhodes, M.G. (2021). Wearable technology for automatizing science-based study strategies:
Reinforcing learning through intermittent smartwatch prompting. Journal of Applied Research in
Memory and Cognition, 10(3), 444-457. https://doi.org/10.1016/j.jarmac.2021.01.001
Cleary, A.M. & Myers, S.J. (2019). Course Grade Rescue Kit: A Rapid Science-based Learning Intervention.
Anne M. Cleary.
Grimaldi, P. J., Poston, L., & Karpicke, J. D. (2015). How does creating a concept map affect item-specific
encoding? Journal of Experimental Psychology: Learning, Memory, and Cognition, 41(4), 1049-
1061. https://doi.org/10.1037/xlm0000076
scheduling related to achievement? Psychonomic Bulletin & Review, 19(1), 126-134.
Holm, A. J., Hausman, H., & Rhodes, M. G. (2020). Study strategies and “study drugs”: investigating the
relationship between college students’ study behaviors and prescription stimulant misuse.
Journal of American College Health, 70(4), 1-10.
Howard-Jones, P.A. (2014). Neuroscience and education: Myths and messages. Nature Reviews
Neuroscience, 15(12), 817-824. https://doi.org/10.1038/nrn3817
Hunter, A. S., & Lloyd, M. E. (2018). Faculty discuss study strategies, but not the best ones: A survey of
suggested exam preparation techniques for difficult courses across disciplines. Scholarship of
Teaching and Learning in Psychology, 4(2), 105-114. https://doi.org/10.1037/stl0000107
Johnson, E. J., & Goldstein, D. (2003). Do defaults save lives? Science, 302(5649), 1338-1339.
Karpicke, J. D. (2018). Concept mapping. In B. Frey (Ed.), The SAGE encyclopedia of educational research,
measurement, and evaluation (pp. 351-354). Publishing, Inc.
Karpicke, J. D., & Roediger, H. L. (2008). The critical importance of retrieval for learning. Science,
319(5865), 966-968. https://doi.org/10.1126/science.1152408
416
King, J. F., Zechmeister, E. B., & Shaughnessy, J. J. (1980). Judgments of knowing: The influence of
retrieval practice. The American Journal of Psychology, 93(2), 329-343.
Kornell, N. (2009). Optimizing learning using flashcards: Spacing is more effective than cramming.
Applied Cognitive Psychology, 23(9), 1297-1317. https://doi.org/10.1002/acp.1537
Kornell, N., & Bjork, R. A. (2007). The promise and perils of self-regulated study. Psychonomic Bulletin &
Review, 14(2), 219-224. https://doi.org/10.3758/BF03194055
Kornell, N. & Bjork, R. A. (2008). Learning concepts and categories: Is spacing the “enemy of induction”?
Kornell, N., Bjork, R. A., & Garcia, M. A. (2011). Why tests appear to prevent forgetting: A distribution-
based bifurcation model. Journal of Memory and Language, 65(2), 85-97.
https://doi.org/10.1016/j.jml.2011.04.002
Kornell, N. & Son, L.K. (2009). Learners’ choices and beliefs about self-testing. Memory, 17(5), 493-501.
Massa, L. & Mayer, R. (2006). Testing the ATI hypothesis: Should multimedia instruction accommodate
verbalizer-visualizer cognitive style? Learning and Individual Differences, 16(4), 321-335.
https://doi.org/10.1016/j.lindif.2006.10.001
Cognition, 39(3), 462-476. https://doi.org/10.3758/s13421-010-0035-2
classroom. European Journal of Cognitive Psychology, 19(4-5), 494–513.
McKenna, K., Folkestad, J., & Moraes, M. (2020). U-behavior: Visual-form learning analytics to enhance
teaching and learning. 6th International Conference on Higher Education Advances (HEAd’20), 30-
05-2020, 697-705.
Morehead, K., Rhodes, M.G. & DeLozier, S. (2016). Instructor and student knowledge of study strategies.
Memory, 24(2), 257-271. https://doi.org/10.1080/09658211.2014.1001992
Nestojko, J. F., Bui, D. C., Kornell, N., & Bjork, E. L. (2014). Expecting to teach enhances learning and
organization of knowledge in free recall of text passages. Memory & Cognition, 42(7), 1038-
Psychological Science in the Public Interest, 9(3), 106-119. https://doi.org/10.1111/j.1539-
6053.2009.01038.x
Rawson, K.A., Dunlosky, J., & Sciartelli, S.M. (2013). The power of successive relearning: Improving
performance on course exams and long-term retention. Educational Psychology Review, 25(4),
Rawson, K. A., Vaughn, K. E., & Carpenter, S. K. (2015). Does the benefit of testing depend on lag, and if
so, why? Evaluating the elaborative retrieval hypothesis. Memory & Cognition, 43(4), 619-633.
Redick, T. S., Shipstead, Z., Harrison, T. L., Hicks, K. L., Fried, D. E., Hambrick, D. Z., Kane, M. J., & Engle, R.
W. (2013). No evidence of intelligence improvement after working memory training: a
417
randomized, placebo-controlled study. Journal of Experimental Psychology: General, 142(2),
359–379. https://doi.org/10.1037/a0029082
Redick, T. S., Shipstead, Z., Wiemers, E. A., Melby-Lervåg, M., & Hulme, C. (2015). What's working in
working memory training? An educational perspective. Educational Psychology Review, 27(4),
Rhodes, M. G., & Castel, A. D. (2008). Memory predictions are influenced by perceptual information:
Evidence for metacognitive illusions. Journal of Experimental Psychology: General, 137(4), 615-
Rhodes, M. G., & Castel, A. D. (2009). Metacognitive illusions for auditory information: Effects on
monitoring and control. Psychonomic Bulletin & Review, 16(3), 550-554.
https://doi.org/10.3758/PBR.16.3.550
Rhodes, M.G., Cleary, A.M., & DeLosh, E.L. (2020). A Guide to Effective Studying and Learning: Practical
Strategies from the Science of Learning. Oxford University Press.
Roediger, H.L. III, & Karpicke, J.D. (2006). Test-enhanced learning: Taking memory tests improves long-
9280.2006.01693.x
Roediger, H.L. III, & Karpicke, J.D. (2018). Reflections on the resurgence of interest in the testing effect.
Perspectives on Psychological Science, 13(2), 236-241.
Rowland, C.A. (2014). The effect of testing versus restudy on retention: A meta-analytic review of the
Shipstead, Z., Redick, T. S., & Engle, R. W. (2012). Is working memory training effective? Psychological
Bulletin, 138(4), 628–654. https://doi.org/10.1037/a0027473
Szpunar, K. K., Khan, N.Y., & Schacter, D. L. (2013). Interpolated memory tests reduce mind wandering
and improve learning of online lectures. Proceedings of the National Academy of Sciences,
110(16), 6313-6317. https://doi.org/10.1073/pnas.1221764110
Thaler, R.H. & Sunstein, C.R. (2009). Nudge: Improving decisions about health, wealth, and happiness.
Penguin Books.
Willingham, D.T., Hughes, E.M., & Dobolyi, D.G. (2015). The scientific status of learning styles theories.
Teaching of Psychology, 42(3), 266-271. https://doi.org/10.1177/0098628315589505
418
In Their Own Words Teaching Students to ‘Study Smart’ – A Training
Program Based on the Science of Learning
to Apply the Science of Learning in Felicitas Biwer and Anique B. H. de Bruin
Maastricht University, The Netherlands
Editors
Lauren L. Kordonowy
Victor A. Benassi
When students enter a college or university, they need to prepare for tutorials and exams, learn from
lectures and seminars, and deal with a high amount of information in a self-regulated way. The
strategies students use to study their learning materials and monitor and control their learning are
crucial for their academic performance (Richardson et al., 2012). But how accurate is students’
knowledge of effective learning strategies, and how can we improve their learning strategies? And how
can we support students to study effectively and use more effective learning strategies during self-
study? In this chapter, we discuss factors that influence students’ learning strategies and outline six
principles which we developed when teaching students how to study more effectively. We then
explicate a design model that outlines how these principles were addressed in the implementation of
our learning strategy training program ‘Study Smart’ (Biwer, de Bruin, Schreurs, & oude Egbrink, 2020;
Biwer, oude Egbrink, Aalten, & de Bruin, 2020).
Over the last decade or so, research in cognitive and educational psychology has led to a fundamental
understanding of the characteristics and underlying processes of effective learning strategies for long-
term learning (Agarwal & Roediger, 2018; Bjork et al., 2013; Dunlosky et al., 2013). The learning
strategies that are most effective in fostering long-term learning often create desirable difficulties: initial
learning becomes more difficult and effortful, but long-term learning and transfer are enhanced (Bjork &
Bjork, 2014). Desirable difficulties are created by, for example, stimulating active retrieval of information
from memory (retrieval practice or practice testing; Adesope et al., 2017; Rowland, 2014) or by spacing
out study sessions over time (distributed practice or spaced practice; Cepeda et al., 2006). Some
learning strategies that create desirable difficulties involve learning activities (e.g., retrieval practice;
elaboration); other learning strategies require planning and structuring of the learning activities (e.g.,
interleaved practice; distributed practice).
The fact that effective learning strategies can create difficulties during initial learning, and ineffective
strategies often feel easy and therefore appear effective during initial learning, makes learners
susceptible to use rather ineffective learning strategies. Moreover, because the effects of desirably
difficult learning strategies are long-term rather than immediate, it is even more difficult for students to
correctly judge the effectiveness of these strategies (Nunes & Karpicke, 2015). As shown in a study by
Kirk-Johnson et al. (2019), students rated strategies that felt more effortful during study as less
effective, and they were less likely to choose these strategies for future study. Survey studies show that
approximately 60-80% of students rely on passive and ineffective learning strategies, such as rereading
or cramming the night before the exam (Blasiman et al., 2017; Karpicke et al. 2009). Even when students
know the benefits of effective learning strategies, they often fail to implement these strategies
sustainably in their self-study (Foerst et al., 2017). This emphasizes the importance of learning strategy
training. However, only 20-40% of students reported studying the way they do because they were
taught to study that way (Morehead et al., 2016). Curricula tend to emphasize the acquisition of
419
content-based knowledge or critical thinking skills but not general learning strategies. Without
awareness of desirable difficulties, students are likely to continue using less effective learning strategies,
because the perceived costs of desirably difficult learning strategies are too high compared to their
perceived utility (Eccles & Wigfield, 2002; Finn, 2020; Wigfield & Eccles, 2000).
Teaching students how to study more effectively does not only entail making them aware of what works
and what does not (Dunlosky et al., 2013), but also involves motivating students to embrace the effort
of desirably difficult learning strategies (Zepeda et al., 2020) and supporting them in implementing these
strategies in their daily study life (Fiorella, 2020). The latter aspect, the role of practicing support and
building habits is, however, often underestimated in learning strategy training (Fiorella, 2020). Existing
interventions to teach students self-regulated learning or learning strategies usually targeted students’
knowledge and motivation. These programs focused mainly on teaching self-regulated learning in
different contexts or developing context-specific strategies, e.g., writing skills, reading ability, or
mathematical problem-solving (de Boer et al., 2018; Donker et al., 2014). A recent meta-analysis
identified 49 intervention programs aimed at improving self-regulated learning skills for college and
university students. These intervention programs included direct instruction on cognitive,
metacognitive, and resource management strategies. They had positive effects on students’ academic
performance, self-regulated learning strategies, and motivation (Theobald, 2021). Only a few of the
included training programs, however, specifically addressed desirably difficult learning strategies and
the aspect of transferring the acquired strategies to self-study. Given the peculiarity of students’ beliefs
and misleading experiences when using learning strategies, teaching students the self-regulated use of
effective learning strategies should include the most effective cognitive, metacognitive, and
motivational strategies (McDaniel & Einstein, 2020; see also McDaniel & Einstein, this volume) but also
focus on the aspect of guided practice. From the theoretical perspective of behavior change, students’
attitudes, subjective norms, and perceived behavioral control will predict their intention, and
consequently, initiate their resulting learning strategy behavior (Fishbein & Ajzen, 2011). It is essential to
take these factors into account when attempting to improve students’ learning strategy behavior.
The Study Smart Model

Based on the outlined challenges of desirably difficult learning strategies and structured around the
theory of planned behavior, we developed a set of principles important to address while teaching
students how to study more effectively. Based on these principles, we designed our learning strategy
training ‘Study Smart’ (Biwer, de Bruin, et al., 2020; Biwer, oude Egbrink et al., 2020). Study Smart is one
way of addressing these principles, but they can be applied in various ways with similar effects. The
principles can serve as a design framework when designing a similar intervention focusing on teaching
students how to study more effectively. Below, we outline our principles and how we addressed them in
the Study Smart training program. Figure 1 depicts the Study Smart Model relating the principles to the
three program sessions--awareness, practice, and reflection—and to the underlining teaching methods
of collaborative learning, reflective learning, and contextual learning. Before outlining the principles in
detail, we will introduce the model and briefly describe where in the model the principles apply.
First, students need to acquire accurate scientific knowledge about effective learning strategies
(principle 1). They need to learn about what works and what does not (declarative knowledge) and how,
when, and why to apply different learning strategies (conditional knowledge). Learning myths and
idiosyncratic ideas about learning need to be debunked. Second, students need to gain insight into their
study strategies and habits—to become more aware of what they are already doing (principle 2).
Students might then become aware of potential discrepancies between empirically effective strategies
and their own strategy use. Furthermore, gaining insight into (potentially) misleading subjective learning
experiences (principle 3) might increase this awareness and, consequently, increase their intention to
420
use more effective learning strategies. The intention to use more effective learning strategies is also
determined by three other factors: a positive attitude towards desirable difficulties and effort, perceived
behavioral control (are students convinced that they have the skills and resources to use these effective
strategies?), and the subjective norm (what are other students doing, or what is necessary to achieve
high grades?). Therefore, it is important to address students’ uncertainty and resistance to change
(principle 4) to positively influence and increase students’ intention to use more effective learning
strategies. Once the intention is formed, students need to set specific goals and formulate action plans
(principle 5) to bridge a potential intention-behavior gap. Last, students’ behavioral control moderates
whether they will actually use effective learning strategies during self-study. To increase behavioral
control, it is important to provide guided practice and context-embedded support in using effective
learning strategies (principle 6). As depicted in Figure 1, the three program sessions (awareness,
practice, and reflection) overlap and address several principles. Several principles are also addressed in
more than one session; the colors indicate the focus of a session. The symbols indicate where the
underlying teaching principles of collaborative learning, reflective learning, and contextual learning are
applied.
Figure 1
The Study Smart Model
Principles
1. Students need to acquire scientific knowledge about effective learning strategies
2. Address students’ existing study habits
3. Students need to gain insight into (potentially) misleading subjective learning experiences
4. Address uncertainty and resistance to change
5. Students need to set specific goals
6. Students need guided practice and context-embedded support in using effective learning
strategies
Teaching principles
collaborative learning
reflective learning
contextual learning
421
Program sessions
Awareness
Practice
Reflection
Note. The principles are numbered in the order in which they need to be addressed in the training. This
does not always lead to a logical numbering in the model, since the model integrates the six principles.
Principle 1: Students Need to Acquire Scientific Knowledge About

Effective Learning Strategies
Many students lack accurate knowledge about how learning works and which learning strategies are
most effective for long-term learning. For example, studies showed that students fail to correctly predict
the benefit of spaced practice compared to massed practice, practice testing compared to restudy, and
interleaved compared to blocked practice (Birnbaum et al., 2013; Kornell & Bjork, 2008, 2009; McCabe,
2011; Roediger & Karpicke, 2006a). Furthermore, common myths about learning, such as the learning
style myth (Kirschner, 2017; Kirschner & van Merriënboer, 2013) impede knowledge acquisition and
mislead students from finding the most effective learning strategies. Many students still have strong
idiosyncratic beliefs about their own learning; they believe that they are unique in the way they learn
best. It is important to refute these idiosyncratic ideas effectively, address misbeliefs, and correct
information to concurrently fill the mental gap created by the correction (de Bruin, 2020). Therefore, to
support students in their use of the most effective and evidence-informed strategies, students need to
acquire knowledge about learning and gain insight into how, why, and when these strategies work. This
knowledge includes not only declarative knowledge about which learning strategies are effective or not,
but also conditional knowledge about when (i.e., in which situations and for which learning materials),
why (i.e., the working mechanisms of effective learning strategies), and how (i.e., how to apply these
strategies).
This Principle in Study Smart:
In the first session of the training program, called ‘awareness’, the teacher presents students with
empirical evidence on the effectiveness of the most commonly used learning strategies (based on
Dunlosky et al., 2013). Students are first asked to rank the strategies based on their effectiveness to
enhance long-term learning. The results are discussed afterward, and the teacher explains the varied
functions of different cognitive learning strategies, such as organization, elaboration, and rehearsal
(Weinstein et al., 2010), as well as their metacognitive effects, including which learning strategies
provide students with feedback that would help them to monitor their learning progress. Students
receive additional information on the basic principles and the importance of active learning and effort
investment. In the second session, called ‘practice’, students are equipped with more specific,
conditional knowledge about how to apply different learning strategies in different situations by using
specific practice exercises.
Principle 2: Address Students’ Existing Study Habits

When entering higher education, students arrive with strong study strategy routines from high school
(Ekuni et al., this volume). The most commonly used strategies are rereading notes or textbooks (83%,
Karpicke et al., 2009; 66%, Hartwig & Dunlosky, 2012), underlining or highlighting (72%, Kuhbandner and
Emmerdinger, 2019), and cramming before the test (66%, Hartwick & Dunlosky, 2012). Some students,
422
however, reported using effective strategies, such as free recall (65%) or connecting new content to
prior knowledge (52%) (Kuhbandner & Emmerdinger, 2019). Both rereading and testing are relatively
common learning strategies; however, many students use testing insufficiently (Bartoszewski & Gurung,
2015; Morehead et al., 2016; Susser & McCabe, 2012). Because prior habits are the main predictors of
future behavior (Danner et al., 2008) and breaking existing habits is more effortful than making new
habits (Lally & Gardner, 2013), addressing and connecting to students’ prior learning strategies early is
vital in facilitating change.
The awareness session starts with a guided brainstorm about commonly used learning strategies.
Students are asked to share their favorite strategies and how they use these strategies to prepare for
practicals, lectures, and exams. To stimulate their reflection, students prepare a photo log of their
academic experiences. They are asked to take a picture that reflects an academic experience, challenge,
or a typical study situation, and to write a reflection on external and internal factors that affected their
learning in that situation. The photologs are meant to help students become aware of their existing
study habits and strategies and to set goals for using more effective learning strategies. In general, we
strive to incorporate Study Smart as early as possible in the first year of the academic curriculum,
because learners will likely be more open to adapt new habits in the transition to higher education.
Principle 3: Students Need to Gain Insight Into (Potentially)

Misleading Subjective Learning Experiences
Experiences during learning can often be misleading, which can potentially lead to ineffective strategy
decisions. Students very likely base their strategy decisions on perceived learning, which can greatly
differ from actual learning (Kirk-Johnson et al., 2019). For example, while rereading a text, learning feels
easy and, consequently, students often overestimate their memory for the text. This experienced-
learning-versus-actual-learning paradox makes students susceptible to mistakenly interpret a sense of
familiarity or fluency during rereading as indicative of effective learning (Kornell et al., 2011;
Oppenheimer, 2008; Yan et al., 2016). When engaged in desirably difficult learning activities, such as
retrieval practice, students are unlikely to experience overt feelings of fluency or familiarity.
Consequently, they tend to underestimate the long-term memory benefits for these strategies (Bjork et
al., 2013; Finn & Tauber, 2015). Gaining insight into these subjective learning experiences is, however,
not easy and requires a multimodal approach. First, theory-based methods include educating students
about these experiences (e.g., Ariel & Karpicke, 2017). Second, experienced-based methods include
demonstrating and letting students experience the difference between two strategies (e.g.,
DeWinstanley & Bjork, 2004; Einstein et al., 2012). Third, explicit feedback about perceived effort,
perceived learning, and actual learning while using desirably difficult learning strategies is important to
aid students in developing that insight (e.g., Hui et al., 2021). Related to principles 1 and 2, gaining
insight into potentially misleading subjective learning experiences is thought to support students in
recognizing and becoming aware of a potential discrepancy between their strategy use and empirically
effective strategies. This awareness, in turn, can enhance students’ intention to change their study
behavior and help them to use more effective learning strategies, principle 5 (Biwer, oude Egbrink, et al.,
2020).
A second element of the awareness session is the explication of the experienced-learning-versus-actual-
learning paradox (Biwer, oude Egbrink, et al., 2020). Using a theory-based approach, research studies on
the testing effect and judgments of learning (Nunes & Karpicke, 2015; Roediger & Karpicke, 2006b) are
423
discussed. Students then reflect with their peers about their own experiences with this paradox.
Applying an experience-based method in one session is challenging, due to the often-delayed effect of
desirably difficult strategies on learning results. The learning benefit of desirably difficult learning
strategies is often not observed immediately but only on tests delayed for at least two days. We
therefore focus on specific examples from research, comparing one effective strategy (e.g., interleaving)
with one ineffective strategy (e.g., blocking), and ask students to predict the results. We then ask
students to reflect on their experiences with these strategies in light of empirical evidence just
reviewed. Having students directly experience the benefits of effective strategies during the training is
no easy feat. In earlier versions of the training, we included a classroom demonstration similar to
McDaniel and Einstein (2020): students were divided into two groups and asked to study two texts with
either reading and highlighting or reading and testing themselves. After a short delay, they took a test
on both readings, provided judgments of learning for both texts, and were asked to reflect on their
experiences during learning. Due to the complexity of desirable difficulties and the often-observed
delayed effect of desirably difficult strategies (in this case, practice testing), students had difficulty
appreciating the benefits of practice testing, as the effect at the time was rather small or not visible in
that session. Nevertheless, students gaining insight into misleading subjective learning experiences is
crucial. This insight could be further stimulated by experience-based modules online, outside the face-
to-face training sessions, for example, by studying vocabulary pairs with either testing or restudy in an
online learning environment and having a final test one week later demonstrating the effects of both
strategies.
Principle 4: Address Uncertainty and Resistance to Change

According to the theory of planned behavior, behavioral intentions are influenced by the attitude
towards the behavior, subjective norm, and perceived behavioral control (Ajzen, 2020). How does this
translate to the intention of using more effective, but also more effortful, learning strategies? First,
students need to develop a positive attitude both towards these strategies and towards effort (Inzlicht
et al., 2018). Ideally, students become motivated through recognition of a discrepancy between their
learning strategies and evidence-informed strategies. However, changing from passively summarizing to
actively practice testing requires time and effort, and thus, increased cost. Although students often view
effort as costly and tend to avoid it, effort is necessary for learning. In domains other than learning, such
as sports or music, effort is valued and rewarded (Inzlicht et al., 2018). Thus, our approach to champion
the value and importance of effortful learning—that is, the attitude towards effort—is one important
factor to increase the intention to use desirably difficult learning strategies.
Second, subjective norms regarding what strategies other students use and regarding requirements
from the assessment system may further influence students’ intention. Is it necessary to use more active
but effortful learning strategies to get a good grade? Is it normal to use practice testing as a learning
strategy? How do other students use practice quizzing? The expected utility of a strategy for students’
goals in relation to the costs of that strategy is at stake (Zepeda et al., 2020). When one is not certain
that the invested effort will result in higher grades, motivation to use these more effortful strategies will
probably be low. Furthermore, when students’ peers are mainly using summarizing or rereading, the
hurdle to use these different strategies might be higher.
Third, perceived behavioral control (e.g., whether students perceive that they are able to apply these
learning strategies given their available time, resources, and skills) can influence students’ intentions.
However, students will likely experience many challenges and drawbacks when engaging in desirably
difficult learning strategies; these strategies tend to be unfamiliar and generally require more effort
through active processing. In our training program, students reported being uncertain about how and
when to use which strategies and wondered how to effectively apply the strategies to their subject
424
domain. They were also uncertain about the effects these changes would have on their academic
performance (Biwer, oude Egbrink, et al., 2020). Consequently, it is essential to address students’
uncertainty and motivation to foster their intention to use more effective learning strategies.
First, to prepare students for the importance of effort and to improve their attitude towards effort, we
include a reflective writing exercise in the first session. In this exercise, students are asked to think about
an experience where they either acquired a new skill (e.g., learned a musical instrument) or changed a
past behavior (e.g., started to eat more healthily). Guided by reflective prompts, students then write
about what they have learned or changed, how they approached it, what challenges and difficulties they
encountered, and how they eventually overcame these difficulties. In our experience, this reflection
helps students to recognize the benefit and value of effort and difficulty in learning something new or
when changing past behavior. Together with peers, students then translate their take-home message
from this reflection to the new challenge of using desirably difficult learning strategies during self-study.
Second, to address the subjective norm, collaborative learning is an essential part of the Study Smart
program. In small groups of 10-15, students discuss and share their favorite learning strategies,
challenges they experience, and how to solve them. Furthermore, we emphasize the use of desirably
difficult learning strategies as a group effort by, for example, inviting students to share self-generated
practice questions with each other. It is important to address in what way the assessment system may
influence students’ study motivations. Are students mainly motivated to achieve high grades or to avoid
failure? Desirably difficult learning strategies do not immediately result in high grades. Depending on the
assessment system, short-term learning strategies might be sufficient to pass exams. However, desirably
difficult learning strategies support long-term learning and support transfer to other contexts (e.g., a
profession). Addressing the subjective norm and students’ motivation is crucial to increase their attitude
and intention to invest in more effortful, but effective learning strategies.
Third, several exercises aim to increase perceived behavioral control. In the practice session, students
practice different strategies using their own learning materials, with peers and guided by the teacher, to
alleviate uncertainty about how to apply the strategies. In the reflection session, students reflect with
their peers about encountered challenges and how to overcome them.
Principle 5: Students Need to Set Specific Goals

When students have acquired the knowledge about effective learning strategies (principle 1), became
aware of their own study habits (principle 2) and of a potential discrepancy between the two (principle
3), and when they have developed a positive attitude towards effort, a supportive subjective norm, and
high perceived behavioral control (principle 4), students are more likely to develop the intention to use
more effective learning strategies. Without a specific action plan on how to set the intention to action,
however, the intention is likely to remain just an intention (Gollwitzer & Sheeran, 2006). To bridge the
intention-behavior gap, past research has stressed the importance of formulating specific goals and
action plans, such as implementation intentions (Gollwitzer & Sheeran, 2006). Implementation
intentions specify when, where, and how to obtain a goal in the form of an if-then plan. For example, to
increase the use of practice testing, one might specify a situation that is usually associated with
studying, specify an action designed to start using practice testing in that situation, and link the
situational cue with the goal-directed response using the if-then format (e.g., ‘If the tutorial meeting is
over, I will go to the library and write down three practice questions.’). Formulating implementation
intentions has been shown to improve the intended behavior, even when participants had to break
weak or moderately unwanted habits (Webb et al., 2009). In line with the cyclical model of self-
425
regulated learning (Zimmerman, 2002), it is important to not only include goal-setting, but also
continuous self-reflection and evaluation of these goals to promote their attainment.
Study Smart sessions include a goal-setting activity at the end of each session and a follow-up activity on
these goals at the beginning of the next session. Thus, students engage in goal-setting and reflection
throughout the entire Study Smart program. At the end of the first session, students think about how to
make their existing and favorite learning strategies more effective. With peer partners, students
critically check their strategies and make a specific plan on how to make this strategy more active and
effective, for example, by creating an opportunity for retrieval practice. Peers and the teacher check
whether these goals are ‘SMART’ (specific, measurable, achievable, relevant, and timebound) (Bjerke &
Renger, 2017). The second session starts with a follow-up on these goals and what potential difficulties
or challenges students might have encountered. After students engage in guided practice exercises
about these strategies, students formulate and refine their goals. The third session includes a reflection
exercise using a critical incident method (Branch, 2005). Together with their peers, students analyze
which challenges they encountered while studying (e.g., how to manage a high amount of information;
how to set a distributed study schedule) and collect ideas on how to master these challenges (e.g., focus
on main parts; use an app that facilitates distributed practice). Afterwards, students set specific
implementation intentions (Gollwitzer & Sheeran, 2006), including back-up plans to initiate and facilitate
the development of the habit of effective strategy use.
Principle 6: Students Need Guided Practice and Context-Embedded

Support in Using Effective Learning Strategies
Simply informing students about what works and what does not work is not enough to change their
learning strategy behavior. The last principle might be the most important one to support students in
sustainably implementing effective learning strategies during self-study. Research has shown that even
though students have the intention to use more practice testing or other effective learning strategies,
they often fail to put these into practice in the long term (Biwer, oude Egbrink, et al., 2020).; Blasiman et
al., 2017; Foerst et al., 2017). Having the intention to engage in more testing or spacing typically
requires a substantial change from students’ existing study habits. How and when can this be
supported?
First, transitioning to college or university is a life transition. This can be the ideal time and context to
disrupt old habits and facilitate the development of new, beneficial habits. To achieve this, it is
important that the new context enables effective strategy use and creates a supportive environment for
beneficial learning strategies (Fiorella, 2020; Lally & Gardner, 2013). For example, teachers could start or
end a lecture with a low-stakes practice test or build in reflection moments (with feedback) for students
on their study behavior. Developing a new habit does not happen overnight. As shown in a study by Lally
et al. (2010), building a consistent habit took participants between 18 to 254 days, depending on the
complexity of the behavior. More complex behaviors, such as exercising after dinner, took longer to
reach automaticity than less complex behaviors, such as drinking a glass of water after breakfast.
Because using desirably difficult learning strategies can be considered a complex behavior, it is
necessary to provide continuous support over a long time.
Second, guided practice can take different forms and shapes, whether provided by teachers, peers, or
tools (e.g., an app). For example, prompting students to regularly use practice testing by providing
automatized reminders by wearable technology such as smartwatches (Cleary et al., 2021) is a promising
tool to support students in actually using these strategies. Other applications, such as ‘Anki’, include
426
smart algorithms that prompt spaced repetition and remind students regularly to practice retrieval. The
role of teachers in guiding the practice of effective learning strategies should not be underestimated.
Teachers can remind students of their learning goals or implement practice quizzes in their lectures to
scaffold the use of effective learning strategies (Carpenter et al., 2017). Contextual learning and the use
of authentic tasks can support students in acquiring the complex skill of using effective learning
strategies (Kirschner & Van Merriënboer, 2008).
The Study Smart program is embedded as much as possible in students’ learning contexts. Working with
a train-the-trainer principle (e.g., Pearce et al., 2012), researchers are training mentors or tutors to
provide the Study Smart program to their students. In small groups with their peers, first-year students
attend the three training sessions, which are divided across the first half of their first academic year. The
advantage of such a learning context-embedded training program is that mentors also meet students
individually and can provide additional support or feedback on struggles students face when changing to
more effective learning strategies. Furthermore, teachers are also trained on how to implement
effective learning strategies in their lectures or tutorials, for example, by starting a tutorial with practice
questions. One often-mentioned problem is the lack of availability of practice questions. In order to
support students in sustainably using practice testing, providing more practice tests and old exam
questions is crucial. Addressing guided and context-embedded practice, students receive step-by-step
support in using effective learning strategies. Worked out examples (Renkl, this volume), exercises on
how to write a summary using the read-recite-review method (McDaniel et al., 2009), how to generate
good flashcards (Lin et al., 2018), how to use self-explanations (Dunlosky et al., 2013), and how to use
the dual-coding method (Mayer & Anderson, 1991) are provided. Together with peers, students choose
a strategy they want to practice on their learning materials. Planning strategies, such as distributed
practice and interleaved practice, are first demonstrated in class, and then students create their
schedule on which they receive feedback. By distributing the three sessions (awareness, practice, and
reflection) over several weeks, students have the opportunity to try out the strategies in their own
context and then return to the session to receive more support and guidance.
Conclusion
With the aim to support students in becoming effective, self-regulated learners and to decrease
inequality between students in learning skills, we developed the Study Smart program in 2017 (Biwer, de
Bruin, et al., 2020). After several rounds of design, implementation, evaluation, and redesign, we
developed a framework for teachers who aim to address the same issues (Biwer, de Bruin, et al., 2020).
Overall, students’ evaluations of the training program have been positive. That is, students indicated
they wanted to be taught how to study more effectively, especially in their transition from high school
to college or university. After the training, however, some students, who judged their grades to be
sufficient, remained resistant or unmotivated to change their existing strategies. Nevertheless, we
advise implementation for all first-year students. More than half of the students in higher education
have difficulty self-monitoring their learning, and they fail to recognize when their learning strategies
are poor (e.g., Dunlosky & Rawson, 2021; Dunlosky & Thiede, 2013). The self-regulated use of effective
learning strategies can only advance when teachers dedicate direct and explicit attention to training
these skills (Dignath & Veenman, 2020).
One major challenge for many students, however, is the actual behavioral control. How can teachers
support students to continue using these effective, but effortful learning strategies? We emphasize the
importance of a context-embedded learning strategy training to support students in their self-regulated
use of these effortful, but more effective learning strategies. The Study Smart program is now provided
427
as three program sessions, and it is usually taught by the students’ personal mentors in the various
study programs. Further embedding it in a regular program with teachers that reflect regularly with their
students about their strategies or use distributed practice or retrieval practice in their classroom is
desirable. Naturally, the assessment structure plays a big role, as assessment is a main motivation for
many students. Innovation in higher education towards implementation of more formative assessment
opportunities, as in ‘assessment for learning’ instead of ‘assessment of learning’, are stimulating
students and teachers to put more effort in scientifically supported learning strategies (such as retrieval
practice) and to build a stronger base for life-long learning. In order to enable students to engage in
desirably difficult learning strategies, a learning culture focused on improvement rather than on
performance is needed (Watling & Ginsburg, 2019). A direct learning strategy training, such as Study
Smart, which is context-embedded as much as possible and supports students in their behavior change,
is a first and important step. In order to stimulate sustainable change, the learning culture must change
towards a culture of improvement and long-term learning, and curricula must embrace the
development of self-regulation skills.
Author Note
The authors thank the Dutch Ministry of Education for funding the design of the Study Smart program
through a Comenius fellowship grant. We also thank EDLAB, the Maastricht center for educational
innovation and the Educational Institute of the Faculty of Health, Medicine, and Life Sciences for
collaboration in designing the Study Smart program and further funding. Corresponding author is
Felicitas Biwer, [email protected]
References
Adesope, O. O., Trevisan, D. A., & Sundararajan, N. (2017). Rethinking the Use of Tests: A Meta-Analysis
of Practice Testing. Review of Educational Research, 87(3), 659-701.
Agarwal, P. K., & Roediger, H. L. (2018). Lessons for learning: How cognitive psychology informs
classroom practice. Phi Delta Kappan, 100(4), 8-12. https://doi.org/10.1177/0031721718815666
Ajzen, I. (2020). The theory of planned behavior: Frequently asked questions. Human Behavior and
Emerging Technologies, 2(4), 314-324. https://doi.org/10.1002/hbe2.195
exam score. Scholarship of Teaching and Learning in Psychology, 1(3), 219-228.
Birnbaum, M. S., Kornell, N., Bjork, E. L., & Bjork, R. A. (2013). Why interleaving enhances inductive
learning: The roles of discrimination and retrieval. Memory & Cognition, 41(3), 392-402.
Biwer, F., de Bruin, A. B. H., Schreurs, S., & oude Egbrink, M. G. A. (2020). Future Steps in Teaching
Desirably Difficult Learning Strategies: Reflections from the Study Smart Program. Journal of
Applied Research in Memory and Cognition, 9(4), 439-446.
428
Biwer, F., oude Egbrink, M. G. A., Aalten, P., & de Bruin, A. B. H. (2020). Fostering Effective Learning
Strategies in Higher Education—A Mixed-Methods Study. Journal of Applied Research in Memory
Bjerke, M. B., & Renger, R. (2017). Being smart about writing smart objectives. Evaluation and program
planning, 61, 125-127. https://doi.org/10.1016/j.evalprogplan.2016.12.009
difficulties to enhance learning. In M. A. Gernsbacher & J. Pomerantz (Eds.), Psychology and the
real world: Essays illustrating fundamental contributions to society (2nd ed.) (pp. 59-68). Worth.
Bjork, R. A., Dunlosky, J., & Kornell, N. (2013). Self-regulated learning: beliefs, techniques, and illusions.
Annual Review of Psychology, 64, 417-444. https://doi.org/10.1146/annurev-psych-113011-
143823
Comparing intended versus actual study behaviour. Memory, 25(6), 784-792.
Branch, W. T., Jr. (2005). Use of critical incident reports in medical education. A perspective. Journal of
general internal medicine, 20(11), 1063-1067. https://doi.org/10.1111/j.1525-
1497.2005.00231.x
Carpenter, S. K., Rahman, S., Lund, T. J., Armstrong, P. I., Lamm, M. H., Reason, R. D., & Coffman, C. R.
(2017). Students' Use of Optional Online Reviews and Its Relationship to Summative Assessment
Outcomes in Introductory Biology. CBE Life Sciences Education, 16(2).
tasks: A review and quantitative synthesis. Psychological Bulletin, 132(3), 354-380.
Cleary, A. M., McNeely-White, K. L., Hausman, H., Dawson, J., Kuhn, S., Osborn, R. M., Rhodes, M. G.
(2021). Wearable technology for automatizing science-based study strategies: Reinforcing
learning through intermittent smartwatch prompting. Journal of Applied Research in Memory
Danner, U. N., Aarts, H., & de Vries, N. K. (2008). Habit vs. intention in the prediction of future
behaviour: the role of frequency, context stability and mental accessibility of past behaviour.
British Journal of Social Psychology, 47(2), 245-265. https://doi.org/10.1348/014466607x230876
de Boer, H., Donker, A. S., Kostons, D. D. N. M., & van der Werf, G. P. C. (2018). Long-term effects of
metacognitive strategy instruction on student academic performance: A meta-analysis.
Educational Research Review, 24, 98-115. https://doi.org/10.1016/j.edurev.2018.03.002
de Bruin, A. B. H. (2020). Debunking myths in medical education: The science of refutation. Medical
Education, 54(1), 6-8. https://doi.org/10.1111/medu.14028
DeWinstanley, P. A., & Bjork, E. L. (2004). Processing strategies and the generation effect: Implications
for making a better reader. Memory & Cognition, 32(6), 945-955.
https://doi.org/10.3758/bf03196872
Dignath, C., & Veenman, M. V. J. (2020). The Role of Direct Strategy Instruction and Indirect Activation of
Self-Regulated Learning—Evidence from Classroom Observation Studies. Educational Psychology
Review. https://doi.org/10.1007/s10648-020-09534-0
429
Donker, A. S., de Boer, H., Kostons, D., Dignath van Ewijk, C. C., & van der Werf, M. P. C. (2014).
Effectiveness of learning strategy instruction on academic performance: A meta-analysis.
Educational Research Review, 11, 1-26. https://doi.org/10.1016/j.edurev.2013.11.002
Dunlosky, J., & Rawson, K. A. (2012). Overconfidence produces underachievement: Inaccurate self
evaluations undermine students’ learning and retention. Learning and Instruction, 22(4), 271-
Dunlosky, J., & Thiede, K. W. (2013). Metamemory. In D. Reisberg (Ed.), The Oxford handbook of
cognitive psychology (pp. 283-298). Oxford University Press.
Dunning, D., Johnson, K., Ehrlinger, J., & Kruger, J. (2003). Why People Fail to Recognize Their Own
Incompetence. Current Directions in Psychological Science, 12(3), 83-87.
https://doi.org/10.1111/1467-8721.01235
Eccles, J. S., & Wigfield, A. (2002). Motivational beliefs, values, and goals. Annual Review of Psychology,
53, 109-132. https://doi.org/10.1146/annurev.psych.53.100901.135153
Einstein, G. O., Mullet, H. G., & Harrison, T. L. (2012). The testing effect: Illustrating a fundamental
concept and changing study strategies. Teaching of psychology, 39(3), 190-193.
Finn, B. (2020). Exploring Interactions Between Motivation and Cognition to Better Shape Self-Regulated
Learning. Journal of Applied Research in Memory and Cognition, 9(4), 461-467.
Finn, B., & Tauber, S. K. (2015). When Confidence Is Not a Signal of Knowing: How Students’ Experiences
and Beliefs About Processing Fluency Can Lead to Miscalibrated Confidence. Educational
Fiorella, L. (2020). The Science of Habit and Its Implications for Student Learning and Well-being.
Educational Psychology Review, 32(3). https://doi.org/10.1007/s10648-020-09525-1
Fishbein, M., & Ajzen, I. (2011). Predicting and changing behavior: The reasoned action approach.
Psychology press.
Foerst, N. M., Klug, J., Jostl, G., Spiel, C., & Schober, B. (2017). Knowledge vs. Action: Discrepancies in
university students' knowledge about and self-reported use of self-regulated learning strategies.
Frontiers in Psychology, 8, 1288. https://doi.org/10.3389/fpsyg.2017.01288
Gollwitzer, P. M., & Sheeran, P. (2006). Implementation intentions and goal achievement: A meta‐
analysis of effects and processes. Advances in experimental social psychology, 38, 69-119.
Hui, L., de Bruin, A. B. H., Donkers, J., & van Merriënboer, J. J. G. (2021). Does Individual Performance
Feedback Increase the Use of Retrieval Practice? Educational Psychology Review, 33(4)
Inzlicht, M., Shenhav, A., & Olivola, C. Y. (2018). The effort paradox: Effort is both costly and valued.
Trends in Cognitive Sciences, 22(4), 337-349. https://doi.org/10.1016/j.tics.2018.01.007
430
Karpicke, J. D., Butler, A. C., & Roediger, H. L., 3rd. (2009). Metacognitive strategies in student learning:
Do students practise retrieval when they study on their own? Memory, 17(4), 471-479.
study strategy choice. Cognitive Psychology, 115, 101237.
Kirschner, P., & Van Merriënboer, J. J. G. (2008). Ten steps to complex learning: A new approach to
instruction and instructional design. In T. L. Good (Ed.), 21st century education: A reference
handbook (Ch. 26). Sage.
Kirschner, P. A. (2017). Stop propagating the learning styles myth. Computers & Education, 106, 166-171.
https://doi.org/10.1016/j.compedu.2016.12.006
Kirschner, P. A., & van Merriënboer, J. J. G. (2013). Do learners really know best? Urban legends in
Kornell, N., & Bjork, R. A. (2009). A stability bias in human memory: Overestimating remembering and
underestimating learning. Journal of Experimental Psychology: General, 138(4), 449-468.
Kornell, N., Rhodes, M. G., Castel, A. D., & Tauber, S. K. (2011). The ease-of-processing heuristic and the
stability bias: dissociating memory, memory beliefs, and memory judgments. Psychological
Science, 22(6), 787-794. https://doi.org/10.1177/0956797611407929
Kuhbandner, C., & Emmerdinger, K. J. (2019). Do students really prefer repeated rereading over testing
when studying textbooks? A reexamination. Memory, 27(7), 952-961.
Lally, P., & Gardner, B. (2013). Promoting habit formation. Health Psychology Review, 7(sup1), S137-
S158. https://doi.org/10.1080/17437199.2011.603640
Lally, P., van Jaarsveld, C. H. M., Potts, H. W. W., & Wardle, J. (2010). How are habits formed: Modelling
habit formation in the real world. European Journal of Social Psychology, 40(6), 998-1009.
https://doi.org/10.1002/ejsp.674
Lin, C., McDaniel, M. A., & Miyatsu, T. (2018). Effects of Flashcards on Learning Authentic Materials: The
Role of Detailed Versus Conceptual Flashcards and Individual Differences in Structure-Building
Ability. Journal of Applied Research in Memory and Cognition, 7(4), 529–539.
Mayer, R. E., & Anderson, R. B. (1991). Animations need narrations: An experimental test of a dual-
coding hypothesis. Journal of Educational Psychology, 83(4), 484-490.
431
McDaniel, M. A., & Einstein, G. O. (2020). Training Learning Strategies to Promote Self-Regulation and
Transfer: The Knowledge, Belief, Commitment, and Planning Framework. Perspectives on
Psychological Science, 15(6), 1363-1381. https://doi.org/10.1177/1745691620920723
McDaniel, M. A., Howard, D. C., & Einstein, G. O. (2009). The read-recite-review study strategy: Effective
and portable. Psychological Science, 20(4), 516-522. https://doi.org/10.1111/j.1467-
9280.2009.02325.x
strategies. Memory, 24(2), 257-271. https://doi.org/10.1080/09658211.2014.1001992
Nunes, L. D., & Karpicke, J. D. (2015). Retrieval‐Based Learning: Research at the Interface between
Cognitive Science and Education. Emerging Trends in the Social and Behavioral Sciences: An
Interdisciplinary, Searchable, and Linkable Resource, 1-16.
https://doi.org/10.1002/9781118900772.etrds0289
Oppenheimer, D. M. (2008). The secret life of fluency. Trends in Cognitive Sciences, 12(6), 237-241.
https://doi.org/10.1016/j.tics.2008.02.014
Pearce, J., Mann, M. K., Jones, C., van Buschbach, S., Olff, M., & Bisson, J. I. (2012). The Most Effective
Way of Delivering a Train-the-Trainers Program: A Systematic Review. Journal of Continuing
Education in the Health Professions, 32(3), 215-226. https://doi.org/10.1002/chp.21148
Richardson, M., Abraham, C., & Bond, R. (2012). Psychological correlates of university students'
academic performance: A systematic review and meta-analysis. Psychological Bulletin, 138(2),
353-387. https://doi.org/10.1037/a0026838
Roediger, H. L., & Karpicke, J. D. (2006a). The Power of Testing Memory: Basic Research and Implications
for Educational Practice. Perspectives on Psychological Science, 1(3), 181-210.
Roediger, H. L., & Karpicke, J. D. (2006b). Test-Enhanced Learning:Taking Memory Tests Improves Long-
Term Retention. Psychological Science, 17(3), 249-255. https://doi.org/10.1111/j.1467-
9280.2006.01693.x
Rowland, C. A. (2014). The effect of testing versus restudy on retention: a meta-analytic review of the
spaced study. Instructional Science, 41(2), 345-363. https://doi.org/10.1007/s11251-012-9231-8
Theobald, M. (2021). Self-regulated learning training programs enhance university students’ academic
performance, self-regulated learning strategies, and motivation: A meta-analysis. Contemporary
Educational Psychology, 66, 101976. https://doi.org/10.1016/j.cedpsych.2021.101976
Watling, C. J., & Ginsburg, S. (2019). Assessment, feedback and the alchemy of learning. Medical
Education, 53(1), 76-85. https://doi.org/10.1111/medu.13645
Webb, T. L., Sheeran, P., & Luszczynska, A. (2009). Planning to break unwanted habits: habit strength
moderates implementation intention effects on behaviour change. British Journal of Social
Psychology, 48(3), 507-523. https://doi.org/10.1348/014466608x370591
Weinstein, C. E., Jung, J., & Acee, T. W. (2010). Learning strategies. In E. Baker & B. McGaw (Eds.),
International encyclopedia of education (Third Edition) (pp. 323-329). Elsevier.
432
Wigfield, A., & Eccles, J. S. (2000). Expectancy-value theory of achievement motivation. Contemporary
Educational Psychology, 25(1), 68-81. https://doi.org/10.1006/ceps.1999.1015
Zepeda, C. D., Martin, R. S., & Butler, A. C. (2020). Motivational Strategies to Engage Learners in
Desirable Difficulties. Journal of Applied Research in Memory and Cognition, 9(4), 468-474.
Zimmerman, B. J. (2002). Becoming a self-regulated learner: An overview. Theory into Practice, 41(2),
64-70. https://doi.org/10.1207/s15430421tip4102_2
433
In Their Own Words
The Student Cognition Toolbox: Empowering
Students to Become Better Learners
Catherine E. Overson, Lauren L. Kordonowy,
Jennifer F. Calawa, Elizabeth M. Tappin, and
Editors
Victor A. Benassi
Lauren L. Kordonowy
Victor A. Benassi University of New Hampshire
For over a decade and a half, University of New Hampshire’s (UNH) Center for Excellence and Innovation
in Teaching and Learning (CEITL) staff has worked with course instructors to apply instructional
principles designed to improve student learning outcomes (see Benassi et al., 2014; Benassi et al., this
volume). We work with instructors to design, implement, and assess the impact of instructional
interventions to meet their learning goals. These interventions require students to complete out-of-class
and/or in-class activities that incorporate cognitively supported principles that positively affect student
learning. This approach has been successful; we have documented consistent and often substantial
gains in student learning, retention, and transfer of learning. However, regardless of our dissemination
efforts, we cannot reach all instructors to assist them in planning and deploying effective instructional
interventions. With the Student Cognition Toolbox (SCT), we continue to work with instructors for SCT
implementation, but our focus in this project is directly on students.
Background
Students enter college with a variety of past experiences and beliefs about how to prepare for
assessments of their academic performance (e.g., exams; problem solving and procedural skills). Many
of them use relatively ineffective study strategies as they prepare for these assessments. Considerable
research supports that most students routinely prefer and use study strategies that are ineffective
relative to strategies supported by applied research on cognition and learning. In a study of nearly 6,000
high school seniors’ approaches to their study, students reported they preferred study strategies mainly
comprising reading and rereading material (Butler, 2018). More noteworthy, the majority of students
reported using only one approach to study. Research shows that different study strategies are needed
for different kinds of learning—one size does not fit all. Perhaps most concerning is that less prepared
students may be especially prone to using study strategies that are relatively ineffective and inefficient.
Our overarching goal in developing a tool to support students’ academic success was to instruct them on
the appropriate use of effective study strategies—techniques that address the kinds of course-related
learning goals set by course instructors. We began our first explorations into addressing this issue when
we developed a ‘low tech’ instructional package (PowerPoint© slides) that included several strategies
designed to improve college students’ learning of academic material (e.g., retrieval practice, distributed
study). The instructional package concluded with students developing a study plan for an upcoming
course exam. Students who completed this instruction performed better on a subsequent exam
compared to students who completed an instructional package that offered advice on strategies such as
studying in a quiet place, getting a good night’s sleep before an exam, and so forth.
434
The Student Cognition Toolbox
As with our previous tool, the SCT addresses what can be done to improve students’ understanding and
application of cognitively supported study strategies. The SCT is an integrated set of modules, each of
which instructs students on the procedure and use of a particular study strategy or schedule of study
(informed by science of learning research). Situated within Carnegie Mellon University’s (CMU) Open
Learning Initiative platform (OLI), this openly available new tool enables us to offer instruction to a wide
audience of instructors and learners.
In addition to instructing students on how to select appropriate study strategies for different learning
goals, the SCT provides students with practice in using the strategies with academic material related to
courses they are taking. It also includes assessments that measure how well students have mastered the
strategies. At the end of each instructional module, students respond to a series of self-reflection
questions designed to help them develop their metacognitive skills (knowing which strategies to apply in
varying contexts) with the ultimate goal of helping them become self-regulated learners.
Student Cognition Toolbox Modules

SCT modules comprise a robust set of evidence-based learning strategies that facilitate student learning,
retention, and transfer of academic knowledge. The SCT has the following components: an introductory
module; four modules on specific study strategies (Table 1); a module in which students apply what they
have learned in preceding modules; a summary module (Figure 1).
Table 1
Learning Strategies and Study Schedules Covered Within the SCT
Study Strategy Study Strategy Described

Practice Quizzing Actively retrieving from memory material you previously encountered
Using prompts to help you describe your understanding of new information
during the learning process
Self-Explanation
Thinking about prior knowledge you may have in the context of the new-to-you
information that you have just acquired
Making connections with the information you already know (through lecture,
Elaborative
class notes, readings, videos, labs, etc.) as you respond to how and why
Interrogation
concepts interact with one another and/or might be applied to a new situation
A step-by-step demonstration of how to perform a task or solve a problem, for
Worked Examples
which you explain each step
Schedule of Study Study Schedule Described
A type of practice in which the learner studies one subtopic over several
Blocked
repetitions before moving on to the next subtopic
A schedule of study in which you space your studying of course material across
Spaced
multiple learning opportunities
A type of spaced practice in which the learner moves between studying related
Interleaved
subtopics in a single learning session and across multiple learning sessions
435
Note. Excerpted from the Student Cognition Toolbox, licensed under a Creative Commons Attribution:
Noncommercial-Share Alike 4.0 License. © 2022 Open Learning Initiative. (Overson et al., 2022, Syllabus:
Student Cognition Toolbox [Open & Free]) https://oli.cmu.edu/courses/student-cognition-toolbox-open-
free/
Figure 1
Screenshot: SCT Syllabus
Noncommercial-Share Alike 4.0 License. © 2022 Open Learning Initiative. (Overson et al., 2022, Syllabus:
Student Cognition Toolbox [Open & Free]) https://oli.cmu.edu/courses/student-cognition-toolbox-open-
free/
436
Each of the study strategy modules includes two sections (Figure 1). The first section provides learners
with academic material (e.g., from an introductory psychology text) to provide a context for—and
practice engaging in—the study strategy that is the topic of a particular module. We used open-source
content from academic disciplines to provide opportunities for students to use the study strategy
covered in each module. For example, in Section 1 of the Reflective Questioning module, students read
a brief passage from an open-source book and then respond to four self-explanation writing prompts
(Figure 2). The activities in Section 1 provide students with examples of—and experience in using—the
study strategy about which they will learn in detail in Section 2.
Figure 2
Screenshot: Self-Explanation Writing Prompts in Section 1 of the Reflective Questioning Module
Noncommercial-Share Alike 4.0 License. © 2022 Open Learning Initiative. Student Cognition Toolbox
[Open & Free]) https://oli.cmu.edu/courses/student-cognition-toolbox-open-free/
437
The second section of modules 2 through 5 provides information on the module’s focused study strategy
(e.g., practice quizzing, worked examples). Students first learn information about the study strategy,
after which they are instructed on how and when to use it as they work through the module. When
appropriate, discipline-specific content is used in section 2 to illustrate the strategy’s application;
otherwise, content for each strategy is identical. Embedded within the modules are several Learn by
Doing practice activities (with corrective feedback) designed to promote and assess learning of that
strategy.
Each module offers a downloadable Study Strategy Selection Table to which students can refer when
identifying the most appropriate study strategies for their learning goals (Table 2). Additional resources
for further reading along with related videos appear at each module’s end, and a cumulative reference
list appears at the end of the SCT course. These recourses might be particularly useful for instructors
who are interested in learning more about how they can incorporate these strategies into their course
design. The best way to discover the SCT’s structure and content is to take a tour:
https://oli.cmu.edu/product-category/student-success-indep/
Table 2
Study Strategy Selection Table
Note: Excerpted from the Student Cognition Toolbox, licensed under a Creative Commons Attribution:
Noncommercial-Share Alike 4.0 License. © 2022 Open Learning Initiative.
[Open & Free]) https://oli.cmu.edu/courses/student-cognition-toolbox-open-free/
438
We currently have seven SCT discipline-specific versions, all of which are accessible at the OLI site:
• Anatomy and Physiology
• Anthropology
• Biology
• Chemistry
• Physics
• Psychology (Default content)
• Statistics
Student Cognition Toolbox Deployment at UNH

We launched the SCT at UNH in Fall 2019. Since then, we have administered the SCT to over 10,000 UNH
students from a wide range of academic disciplines (Table 3). Although the vast majority of these
learners were undergraduate students, several instructors of graduate level courses have also elected to
assign the SCT to their students. Many of the students were enrolled in introductory STEM courses (e.g.,
anatomy and physiology, biology, chemistry, physics). Additionally, students enrolled in the liberal arts
college (e.g., anthropology, political science, psychology, sociology), the health and human services
college (e.g., health care systems, nursing), and the business college (business statistics) have completed
the SCT. Students enrolled in courses spanning over 40 different academic areas have completed the
SCT to date, many of whom were enrolled in undergraduate gateway courses.
Table 3
Fields and Disciplines in Which the SCT has Been Administered in UNH Courses
Animal Health Mythology
Anthropology Nursing
Biology Phlebotomy
Body Fluids PLTL Leaders
Chemistry Political Science
Cybersecurity Psychology
Earth Sciences Spanish
First-year Writing Statistics
Genetics Sociology
Geography US Health Care Systems
Music Water Quality
Math Weather
Collaborating With Course Instructors at UNH

CEITL staff work with UNH instructors to deploy the SCT in their courses. We begin by meeting with
interested instructors to convey the importance of students’ use of appropriate study strategies to meet
their course learning goals. We follow with a tour of the SCT on the OLI platform. We ask instructors to
include the SCT assignment in their course syllabi and to offer a small amount of course credit for
completion. Our experience is that if an instructor merely offers the SCT as an optional assignment,
many students do not follow through to completion.
439
For each participating course, we link the CMU OLI platform hosting the SCT through UNH’s learning
management system (Canvas©). We advise instructors to ask students to complete the SCT within the
first two weeks of the academic term so they can begin using the strategies early in the course.
Throughout the implementation, CEITL staff respond to students’ questions and requests for assistance
and assign students completion credit in Canvas©. We encourage students to revisit the SCT modules
and Study Strategy Selection Table as they work through the remainder of the academic term, in
particular, prior to major assessments.
Faculty Development Workshops

We offer professional development opportunities to instructors that focus on how they can incorporate
activities and assignments into their courses using the strategies about which students learn in the SCT.
For example, we first describe the strategies of worked examples, practice quizzing, and reflective
questioning, and then present options for in- and out-of-class activities to engage students in the
effective use of these strategies. Some of the participating instructors go on, with support from CEITL
staff, to create and implement the activities covered in our workshops. A recorded version of this
workshop (Interactive Activities to Promote Study Strategies: Companion Activities for the Student
Cognition Toolbox) is accessible.
Making Iterative Improvements

Beginning with the 2019 SCT launch, and continuing with all subsequent deployments, we have
collected data analytics on students’ activities within the tool. This information has been instrumental in
enabling us to make iterative improvements for subsequent SCT deployments. For example, we examine
students' incorrect answers to Learn by Doing activities and end of module quizzes to determine if there
are questions with high error rates. In those cases, we refer back to the SCT content on which the
question is based and consider how we might improve the presentation of the material (for example, by
including an illustration of a concept). We also review students’ responses to open-ended questions and
look for instances of widespread misunderstanding of a particular point we wanted to convey.
As students begin the first module (Introduction to Study Strategies), they rate the extent to which a
series of statements on a Study Behavior Inventory (SBI) are true of them. Half of these statements
reflect ‘deep’ approaches to study (e.g., “I explain concepts to myself and/or a classmate/friend”) and
half of them reflect ‘shallow’ approaches (e.g., “I highlight and/or underline the most important
information in my reading.”). In module 7 (Student Cognition Toolbox Summary), following completion
of the instructional content covered in modules 1-6, students again complete the SBI; this time they
reflect on the extent to which they intend to use the exam preparation strategies prior to a future
assessment. We noted from our first SCT deployment that pre-post SBI comparisons documented that,
on average, students showed a greater degree of agreement with ‘deep’ strategies on the post
assessment. However, on the post SBI, there was little and inconsistent evidence that students endorsed
‘shallow’ strategies to a lesser extent. Upon review of the SCT’s first edition, we noticed that we focused
nearly exclusively on the benefits of using deep study strategies and we gave very little attention to the
rationale for decreasing the use of shallow study strategies. In the next edition of the SCT, we addressed
shallow strategies more intentionally, discussing how using such strategies might seem to be useful even
though they are generally not ideal.
440
Validating the SCT
We will provide a full report of our efforts to validate the SCT in another publication (Overson, et al., in
preparation). We have measures of how well students have learned SCT content, their pre- and post-SCT
SBI scores, their academic self-efficacy, their commitment to use the SCT strategies, and their scores on
course exams. We provide the results below of one analysis that documents students’ pre- and post-SCT
SBI scores.
Figure 3 displays results from two administrations of the SBI to students in all courses from Spring 2020.
For the first administration, students were asked the extent to which they used particular exam
preparation strategies prior to a major assessment. After students completed the SCT they were asked
to rate the extent to which they intended to use the exam preparation strategies prior to a major
assessment. The results, which we have replicated during each subsequent SCT administration, provide
an indication of changes in students’ commitment (intention) to use the strategies they learned in the
SCT. The mean rating for each of the deep processing strategies was higher on the post SCT assessment.
Figure 3
Mean Pre- and Post- Study Behavior Inventory Responses for Students who Completed the SCT
Study Behavior Inventory Deep Processing Responses

Pre- and Post- SCT
6
Student Ratings
1
1 2 3 4 5 6 7 8
SBI Prompt
Note. N = 695. Error bars represent 95% CI.

Scale from 1 (“Not at all true of me”) to 6 (“Very true of me”).
Legend:
1. I space out my study sessions in the time leading up to the exam.
2. I relate what I am currently learning to what I already know.
3. I test myself on course materials without referring to my course materials or notes.
4. I plan effectively for study time between classes.
5. I summarize in my own words information I learn from my study.
6. I explain concepts to myself and/or a classmate/friend.
7. I create outlines, charts, diagrams, or tables, etc., to organize and help me see patterns in course
information.
8. When I am learning to solve new problems, I review examples of solved problems that explain
each step along the way.
441
Conclusion
Our work offers one approach to providing instruction to students on study strategies supported by
science of learning research. The SCT can be used by students on their own. It can also be used by
instructors who administer it in their courses and perhaps create learning opportunities for students to
apply the strategies as they prepare for academic assessments.
There are other approaches to instructing students in the use of effective study strategies. Some of
those approaches are described by colleagues in this volume (Biwer & de Bruin; Brown-Kramer; Cleary &
Rhodes; Endres; Manalo et al.; McDaniel & Einstein; Sumeracki et al.) and elsewhere (e.g., Bernacki et
al., 2020; Bernacki et al., 2021; McCabe et al., 2020; Oakley & Sejnowski, 2022). An advantage of the SCT
is that it is situated on the CMU OLI platform where it can be edited, updated, and otherwise modified.
The study strategies described in the current SCT are not, of course, the only ones that could be used by
students (e.g., see Fiorella’s chapter on ‘teaching as learning’, this volume). Additional modules can be
easily added to the SCT.
Our work on the SCT to date has focused on its development as a valid instructional tool. As discussed
by Manalo et al. (this volume) and McDaniel & Einstein (this volume), teaching students to use study
strategies that benefit their learning is one thing; getting them to persist in using them on their own is
the challenge. Our next line of work focuses on preparing instructors to use techniques that they can
implement in their courses that will make it more likely that students will actually use, and continue to
use, the effective study strategies they learn in the SCT.
Author Note
The Student Cognition Toolbox is made possible by a grant from the Davis Educational Foundation
(Yarmouth, Maine). The Foundation was established by Stanton and Elisabeth Davis after Mr. Davis's
retirement as chairman of Shaw’s Supermarkets, Inc.
The content and structure of the SCT has been developed by UNH’s CEITL staff: Catherine Overson,
Victor Benassi, Lauren Kordonowy, Jennifer Calawa, Meghan Stark, and Christopher Williams. Elizabeth
Tappin is the Assessment Coordinator.
We gratefully acknowledge the support and assistance of colleagues at CMU OLI. CEITL developed and is
responsible for the content of the SCT and its instructional design. OLI has deployed instances of the
SCT, addressed technical requests and issues with the course (e.g., with developing a new Survey Tool),
and provided CEITL with periodic learning analytics that are used for course improvement.
We also acknowledge the ongoing support of the University of New Hampshire’s Office of Academic
Affairs/Provost’s Office. We thank the dozens of course instructors and thousands of students who have
participated in our SCT project. Without their contributions, there would be no SCT.
References
Benassi, V.A., Overson, C. E., & Hakala, C. (Eds.) (2014). Applying science of learning in education:
Bernacki, M. L., Vosicka, L., & Utz, J. C. (2020). Can a brief, digital skill training intervention help
undergraduates “learn to learn” and improve their STEM achievement? Journal of Educational
Psychology, 112(4), 765–781. https://doi.org/10.1037/edu0000405
442
Bernacki, M. L., Vosicka, L., Utz, J. C., & Warren, C. B. (2021). Effects of digital learning skill training on
the academic performance of undergraduates in science and mathematics. Journal of
Educational Psychology, 113(6), 1107–1125. https://doi.org/10.1037/edu0000485
Butler, A. C. (2018). Understanding Normative Educational Practices Can Inform Innovation in STEM
Learning. Presentation at CIRCLE Conference (Washington University at St. Louis, MO,
September 2018).
McCabe, J. A., Friedman-Wheeler, D. G., Davis, S. R., & Pearce, J. (2020). “Set” for success: Targeted
instruction in learning strategies and behavior change in introductory psychology. Teaching of
Oakely, B., & Sejnowski, T. (2022). Learning how to learn: Powerful mental tools to help you master
tough subjects. Coursera, offered by Deep Teaching Solutions.
https://www.coursera.org/learn/learning-how-to-learn#instructors
Overson, C. E., Kordonowy, L. L., Benassi, V. A., Calawa, J., Ange-Stark, M. A., Williams, C. R., & Tappin, E.
M. (2020, v1.4, Open Educational Resource). The Student Cognition Toolbox, Carnegie Mellon
University, Open Learning Initiative, https://oli.cmu.edu/courses/student-cognition-toolbox-
open-free/
Overson, C. E., Kordonowy, L. L., Benassi, V. A., Tappin, E. M., & Calawa, J. (in preparation). Validating
the student cognition toolbox.
443
In Their Own Words
Part 5
Putting the Science of Learning
into Practice
Editors
Lauren L. Kordonowy
Victor A. Benassi

444
In Their Own Words How Students' Decisions to Space Their Practice
are Related to Better Learning
to Apply the Science of Learning in Veronica X. Yan
The University of Texas at Austin
Editors
Lauren L. Kordonowy
Paulo F. Carvalho
Victor A. Benassi
Faria Sana
Athabasca University
In traditional classroom settings, teachers set the pace, guiding when students move on to new
concepts and when they return to old ones. Online courses, remote learning, and MOOCs have
increased the importance of making the correct decisions on how to organize one’s study. Without
increasing class time, these decisions of “what, when, and how often” can have a significant impact on
student learning outcomes. In a recent article (Carvalho et al., 2020), we explored how the study
decisions that students make for themselves can have a large impact on their learning.
What kinds of decisions might students be making? Imagine a student in a traditional class: after a class
session, they might use their notes or textbook to review the materials that were covered during class.
They might choose to do so that very evening, a few days later, or only immediately before the next
exam. Now imagine a student enrolled in an asynchronous online course: they not only have control
over when they review materials and complete assignments, but they have control over when they
engage with lessons in the first place. Consider the lead up to an important exam. A student in a
traditional class will have been exposed to the material in class days or weeks earlier, but a student in an
asynchronous course has the option of studying the content for the first time and taking the exam on
the same day. Although these two students might reach the same benchmark of “passing the exam”,
our research shows that how a student organizes their study across time—something that may appear
inconsequential—can have a great impact on learning outcomes.
Why Does Spacing Matter?

While students and instructors typically understand the importance of what is studied and how it is
studied, it turns out that an equally important question is when it is studied. By when, we do not mean
time of day or how long students spend engaged in study; rather, it is critically important how study is
spaced out over time. Given the same amount of study time (e.g., 10 hours), it can make a huge
difference whether those ten hours are concentrated together in a single session (massed study) or
spread out across multiple different days (spaced study). Although intuition might suggest that one
should sit down and intensively focus in a single long session, empirical evidence from cognitive science
research reveals that the opposite is true. That is, sustained, long-term learning and retention depends
on learners repeatedly returning to previously studied material over time, a phenomenon known as the
spacing effect.
The spacing effect has been extensively studied—across age groups, domains, and various settings from
the lab, to classrooms, to adaptive online learning environments—and is considered one of the most
robust findings in cognitive psychology (Carpenter, 2017). There are many theories as to why spacing
promotes learning and these theories are likely not mutually exclusive. That is, spacing offers many
445
different advantages. For example, when learning is massed, attention often wanes through the study
session; spacing helps prevent the diminished processing from happening (diminished processing
theory, Hintzman, 1976). When study is spread out across different sessions, learners have to more
effortfully retrieve or remind themselves of what they had learned in a previous session, and this
effortful retrieval helps deepen learning (study-phase retrieval theory, Appleton-Knapp et al., 2005;
Thios & D’Agnostino, 1975; reminding theory, Benjamin & Ross, 2011). When learning is spaced out,
information is more likely to be encoded differently across sessions—whether it is because a learner is
more likely to be experiencing a different context (physical or mental), or because they are returning to
the content with new eyes and a new perspective (encoding variability theory, Estes, 1955; Glenberg,
1979; Mozer et al., 2009). Finally, spacing across multiple days also allows for learning to become better
consolidated through sleep (consolidation theory, Bell et al., 2014).
The Study: Who Spaces, Who Benefits?

The general suggestion is that if a student were to devote 10 hours of study to a particular topic, it is
better to spread those hours out across multiple shorter learning sessions than to attempt all study in
only one or two longer learning sessions. Do students use this powerful approach when free to organize
their own study and do they benefit from it? Understanding if students space their learning, and if doing
so improves their learning outcomes, requires allowing students to freely make decisions and then
relating these decisions to their learning outcomes. The increasing availability of educational datasets
and robust analytic approaches make this possible. Our main research questions were: 1. whether
students would use spacing or not and who would use it more, 2. whether students would benefit from
spacing, and 3. whether other study behaviors could compensate for a lack of spacing and yield similar
learning benefits (Carvalho et al., 2020).
We used data from a psychology Massive Online Open Course (MOOC), which was offered to students
via Coursera by an instructor from Georgia Institute of Technology in 2013. The course consisted of
weekly multiple-choice quizzes (worth 30%), written assignments (worth 30%) and a final cumulative
exam (worth 40%). The course was 12 weeks in length (offered from March 25 until June 15), and
covered 12 units of content. Students watched weekly video lectures created by the instructor, and
completed online textbook readings as well as related online textbook activities. Students had one week
to complete the quizzes, which were released every Friday morning. The final exam was released on the
last Friday of the course and remained open for five days. The course was open to the public. A total of
5,615 students enrolled in the course and consented to having their data used for research purposes.
We analyzed data of the 747 students who completed the final exam; most of these students (n = 639)
also completed every unit quiz.
In our study, we analyzed gradebook and clickstream data (i.e., click-by-click information of a
student’s actions) from students interacting with the online textbook. We used statistical methods to
estimate students’ study behaviours in the course and calculated how much they spaced their practice
(i.e., the number of sessions across which a student spread their study). We then related the number of
study sessions (spacing) students spent on each unit and their grade in that unit’s quiz. In the analyses,
we accounted for students’ initial knowledge (as measured by a 20-item true/false pre-test on general
psychology, administered before the course began), controlled for total time spent studying, and
analyzed high and low-performing students (as defined by final exam score) separately.
Table 1 summarizes our key research questions and findings. Our analyses found that high performing
students (those with higher final exam scores) were more likely to space their study than low
performing students. However, spacing out study was especially helpful for lower-performing students.
To illustrate this effect in Figure 1, we plotted changes in the unit quiz results (mean = 87%, SD = 0.14%)
446
as a function of the number of sessions completed (i.e., spacing; mean = 4.08, SD = 3.17) and final exam
scores. Keeping total study time constant, when lower-performing students spaced their study of a unit
out across more sessions, they received larger improvements in that unit’s quiz score. In fact, the
learning outcomes for low performing students who spaced their practice across many sessions was
nearly equivalent to that of high performing students. In other words, the students who would benefit
from spacing the most were also the ones who were least likely to space their study.
Table 1
Key Research Questions and Findings
Key Research Questions Key Findings
Which students are more likely to space? Higher-performing students are more likely to
choose to space
Does greater spacing relate to greater quiz Yes, and especially so for low-performing
performance? students
Can other study behaviors compensate for a lack Yes, when students engage with more embedded
of spacing? activities, there is a smaller influence of spacing
Figure 1
Spacing Benefit By Learner Level
Note. The axes represent standardized deviation (SD) from the mean. Controlling for total study time,
spacing is positively related to better quiz performance for students at three different levels of final
447
exam performance (-1SD below average, average, and +1SD above average). Spacing was most strongly
related to better quiz performance for the lower-performing students.
Finally, we investigated whether other study behaviors could compensate for a lack of spacing. We
found that spacing was particularly useful for students who did not engage with the course practice
activities and instead focused only on the text.
Implications for Teachers

Students are often tempted to be “efficient” and complete everything all at once. Why study one hour a
day across ten days, when you could instead study for five hours a day across two days? Our findings
show that this kind of strategy can backfire. Rather than students massing their study over one or two
sessions, our findings emphasize the importance of spacing out study time for promoting learning,
especially for students who are under-performing. Although our study examined data from an online
asynchronous course, our findings have implications for all instructional formats.
Both our study and prior studies on spacing provide suggestions for what teachers can do to support
their students’ use of spacing and their students’ learning. Teachers’ own choices to space—by returning
to previously taught concepts in both their lessons and assignments—can have a large impact on how
well their students are able retain and build on their knowledge, not just in one class, but for future
classes too. They can create cumulative assignments so that students have to return to previously
taught concepts in order to complete current assignments, or create interim assignments (e.g.,
complete a reading, create an outline) that have to be submitted several days before the full assignment
is due. Teachers can also structure their courses so that there are built-in incentives to nudge students
to space out their own learning. Teachers using learning management systems can adjust the timing of
when resources are locked or unlocked to encourage students to space—for example, locking relevant
readings a day before the assignment is due, so that students are discouraged from completing the
reading and assignments all on the same day.
Importantly, when teachers create these spacing structures, it can be helpful to be explicit to students
about why spacing is effective for learning. That is, teachers can both model how to space and at the
same time teach their students about why they should be trying to use it in their own studying too. Our
results suggest that this explicit instruction may be most effective when targeted at students who are
struggling—they are the least likely to use spacing in their own studies, but the most likely to benefit
from its use.
Conclusion
Plenty of evidence has suggested that spacing students’ study of the same material across time
improves learning outcomes. Our research built on this evidence to show that students can intrinsically
benefit from spacing by organizing their study appropriately. However, not all students do so, and
lower-performing students are the least likely to space their study. Our findings suggest that in addition
to incorporating spacing schedules as part of course development, it might be critical to familiarize
students with the benefits of spaced practice, create assignments that require them to return to
previously taught concepts, and structure the course in ways that nudge students to space their study.
Author Note
Veronica X. Yan: https://orcid.org/0000-0002-3988-3184
Paulo F. Carvalho: https://orcid.org/0000-0002-0449-3733
Faria Sana: https://orcid.org/0000-0002-2202-7592
448
All authors contributed equally to this work.
Preparation of this manuscript was supported by National Science Foundation grant # BCS-1824257 to
PFC and Social Science and Humanities Research Council Insight grant #435-2021-0426 to FS and VXY.
Correspondence concerning this chapter should be addressed to Veronica X. Yan, Department of
Educational Psychology, The University of Texas at Austin, College of Education, 1912 Speedway, STE
5.708, Mail Code: D5800, Austin, Texas 78712. E-mail: [email protected]
References
Appleton-Knapp, S. L., Bjork, R. A., & Wickens, T. D. (2005). Examining the spacing effect in advertising:
Encoding variability, retrieval processes, and their interaction. Journal of Consumer Research,
Bell, M. C., Kawadri, N., Simone, P. M., & Wiseheart, M. (2014). Long-term memory, sleep, and the
spacing effect. Memory, 22(3), 276-283. https://doi.org/10.1080/09658211.2013.778294
Benjamin, A. S., & Ross, B. H. (2011). The causes and consequences of reminding. In A.S. Benjamin (Ed.),
Successful remembering and successful forgetting (pp. 89-106). Psychology Press.
Carpenter, S. K. (2017). Spacing effects in learning and memory. In J. T. Wixted, & J. H. Byrne (Eds.),
Learning and memory: A comprehensive reference (pp. 465–485). Academic Press.
Carvalho, P. F., Sana, F., & Yan, V. X. (2020). Self-regulated spacing in a massive open online course is
related to better learning. npj Science of Learning, 5(1), 1-7. https://doi.org/10.1038/s41539-
020-0061-1
Estes, W. K. (1955). Statistical theory of distributional phenomena in learning. Psychological Review,
62(5), 369-377. https://doi.org/10.1037/h0046888
Glenberg, A. M. (1979). Component-levels theory of the effects of spacing of repetitions on recall and
recognition. Memory & Cognition, 7(2), 95-112. https://doi.org/10.3758/BF03197590
Hintzman, D. L. (1976). Repetition and memory. In G. Bower (Ed.). The psychology of learning and
memory (Vol. 11, pp. 47–91). Academic Press. https://doi.org/10.1016/S0079-7421(08)60464-8
Mozer, M. C., Pashler, H., Cepeda, N., Lindsey, R., & Vul, E. (2009). Predicting the optimal spacing of
study: A multiscale context model of memory. In Y. Bengio, D. Schuurmans, J. Lafferty, C.K.I.
Williams, & A. Culotta (Eds.), Advances in neural information processing systems 22 (pp. 1321-
1329). NIPS Foundation. https://dl.acm.org/doi/10.5555/2984093.2984242
Thios, S. J., & D'Agostino, P. R. (1976). Effects of repetition as a function of study-phase retrieval. Journal
of Verbal Learning and Verbal Behavior, 15(5), 529-536. https://doi.org/10.1016/0022-
5371(76)90047-5
449
In Their Own Words Sweat So You Don’t Forget

Michelle Ogrodnik
You to Know About Why and How McMaster University
Barbara Fenesi
Editors
Western University
Lauren L. Kordonowy
Victor A. Benassi

Jennifer J. Heisz
McMaster University
The Learning Issue: A Distracted Mind

A focused mind facilitates learning. Unfortunately, the mind is easily distracted. Modern classrooms are
partly to blame, fraught with electronic devices (and their unrelenting notifications) that can draw
students’ attention away from learning (Ma et al., 2020; May & Elder, 2018; Shelton et al., 2009).
However, the mind’s inability to stay focused is also hardwired. The term “mind wandering” is used to
describe the brain’s default mode. When learners become tired or bored, their brains revert to its
default mode, which allows the mind to wander away with internal thoughts, carrying us back into the
past or forward into the future; anywhere but here and now (Smallwood & Schooler, 2006). It is
estimated that students spend about 30% of their day mind wandering (Kane et al., 2007; Lindquist &
McLean, 2011; Szpunar et al., 2013), and most of their mind wandering is done while studying or in class
(Unsworth et al., 2012). When students’ minds are wandering, it is unlikely they are learning the task at
hand.
Consider this scenario:

Marcy is in her classroom seat and the class period half over. She learns that there is a test next
week and the content of today’s lesson will be on that test. Her teacher is explaining exactly
what students need to know. “It’s important that you learn this for the test,” the teacher
emphasizes.
Marcy nods. But wait . . . did she actually hear her? She didn’t. Oh, no! “Excuse me, professor.
Can you repeat that last bit?” Marcy asks.
Her teacher repeats: “It’s very important that you remember this.”
“Hmm . . .I’m hungry! What’s for lunch?” Marcy asks herself. “How can you be daydreaming at a
time like this,” Marcy scolds herself, as her teacher continues, and she realizes she has missed it
again.
In this chapter, we describe why the brain makes it hard for students to stay focused and how you can
help your students stay focused and, as a result, learn better. The approach? Move the body, sharpen
the mind. Active students focus longer and learn better.
Why it is Difficult to Stay Focused

The ability to stay focused and ignore distractions depends on the brain’s executive functions that are
governed by the prefrontal cortex (PFC) (Moriguchi & Hiraki, 2013). Staying focused requires brain
450
power and the PFC works best when it has enough fuel. The fuel? Oxygenated blood, which is delivered
to the PFC via the blood stream.
When we sit and learn for extended periods of time, blood flow to the brain drops (Carter et al., 2018).
This means that the PFC does not have the fuel it needs to stay focused. To make matters worse, many
elementary and secondary schools have reduced the amount of time that students get to be physically
active. Among students at schools in Ontario, Canada, nearly 60% of elementary students and over 70%
of secondary students are not meeting the guidelines of 60 minutes of daily physical activity (Ogrodnik
et al., 2020). Unfortunately, this is a global trend (Tremblay et al., 2014). What do students do with that
extra time? They reallocate it to core subjects such as math and language (Wilkins et al., 2003).
However, our research suggests that this approach may be counterproductive because students who are
the least physically active also perform the least well academically (Ogrodnik et al., 2020), suggesting
that students who forgo physical activity may be doing more harm than good.
Fortunately, there is a solution. Move the body, fuel the mind. Short frequent movement breaks prevent
the loss of blood flow to the PFC from prolonged sitting (Carter et al., 2018). Interrupting prolonged
sitting by standing up and moving around gives the brain the power it needs to stay focused. Exercising
for longer periods—and more vigorously—increases oxygenated PFC blood flow even more (Giles et al.,
2014).
Can We Apply This Science to the Classroom?

Yes! However, most studies have focused on younger students. For example, in elementary classrooms,
short exercise breaks of 10 minutes or less has been shown to improve on-task behavior, focused
attention, and even academic performance (Bedard et al., 2019; Howie, Beets & Pate, 2014; Ma, Le
Mare & Gurd, 2015). Although this research is promising, it begs the question:
Can Exercise Breaks Help Older Students Stay Focused and Learn Too?
To address this question, our lab group recruited 77 undergraduate students (between 18-22 years old)
who were enrolled in a blended-learning Introductory Psychology course that used video lectures to
deliver some of its content (Fenesi et al., 2018). Each student came to the lab to complete the
experiment over two days.
On the first day, students were invited to the lab to watch a 50-minute online lecture from their
Introductory Psychology course, one that they had not yet watched. Prior to watching the lecture, each
student was randomly assigned to one of three groups:
• Control group, who watched the lecture continuously with no breaks.
• Exercise group, whose lecture-watching was interspersed with short exercise breaks during
which students performed five exercises: 1. jumping jacks, 2. heeltaps, 3. high knees, 4. split
jumps, and 5. hamstring kickers. Each exercise was done for 50 seconds followed by a 10 second
rest. All exercises were done without equipment.
• Non-exercise group, whose lecture-watching was interspersed with short non-exercise breaks
during which students played a computer-game called Bejeweled that requires players to match
tiles to solve a puzzle.
Both break types lasted five minutes and were interspersed approximately every 17 minutes during the
lecture. Just prior to the first break (or after a similar passage of time for the control group), a question
appeared on the computer screen asking students if they were mind wandering. Students who reported
that they were “on task” indicated that they were focused on the lecture and students who reported
that they were “off task” indicated that they were not focused on the lecture but thinking about
451
something else, that is, they were mind wandering. Students were asked this question again before the
second break and again at the end of the lecture.
Our Findings
Most students were focused on the lecture at the start but became more distracted as the lecture
progressed. That is, except for students in the exercising group, who remained focused throughout the
entire lecture (Refer to Fenesi, et al., 2018, Figure 2A for details).
Did the Sharper Focus From the Exercise Breaks Help Students Learn Better?
Yes! Student’s comprehension for the lecture material was tested using two 30-question multiple choice
quizzes: one quiz was administered at the end of the lecture and the other quiz was administered two
days later. The quizzes used unique questions but were from the same course test bank. In addition,
they assessed students’ understanding of the same context, and were of similar difficulty.
Most important, students in the exercise breaks group performed better on both quizzes than the other
two groups. See Figure 1 below.
Figure 1
Mean Percentage Correct for Each Condition on Both the Immediate and Delayed Quiz
Note. The Y axis begins at 38% not 0%. Error bars represent standard error.
Therefore, interspersing short exercise breaks during a lecture helped these university students to stay
focused and learn. Sweat so you don’t forget!
452
Educational Implications: Ready to Incorporate Exercise in Your
Classroom?
Not sure where to start? Get your students to move their bodies in any way! Remember, prolonged
stationary activity is bad for brain blood flow. Any break from sitting is better than no break.
The following frequently asked questions may help you decide on the exercise breaks that will meet
your class needs.
How Long Should the Exercise Break Last?
Some is better than none! Even short 5-minute exercise breaks may promote learning. In our study, a
few 5-minute exercise breaks were enough to improve students focus and memory. However, keep in
mind that these short breaks were done at a high intensity.
Are Low Intensity Exercise Breaks Effective?
Based on the current science, low, moderate, and vigorous intensity exercises can increase blood flow to
the PFC, though longer vigorous exercise increases it more than less vigorous activity (Giles et al., 2014).
So, yes, low intensity exercise breaks may help keep your students focused too.
How Should I Get My Students to Move During the Exercise Break?
Most research testing exercise breaks have used aerobic exercises such as running-on-the-spot or
jumping. But participation is key! Get your students moving in any way that you can comfortably
facilitate, and they can comfortably execute. It is important that all students feel comfortable and can
participate in the experience. Be aware that prescribed exercise suggestions may not work for every
student in your class. Provide options and autonomy for students to move in ways that work best for
their bodies. Accessibility, inclusivity, and fun should be top of mind as you encourage students to move.
When Should I Give My Students the Opportunity to Take an Exercise Break?
Break up a long lesson with a short exercise break. Try incorporating exercise into a lesson by having
students jump out the solution to a math problem, walk to different active learning stations, or go for a
hike to learn about biodiversity. If that’s not feasible, consider giving your students the opportunity to
move before or after lessons (Bedard et al., 2019; Naylor et al., 2015). Exercising before a lesson
supplies the brain with neural chemicals that prime it for learning and exercising after a lesson helps the
brain remember newly learned information (McNerney & Radvansky, 2014). Research shows that a
short bout of exercise can benefit the PFC and its executive functions for up to two hours afterwards
(Basso et al., 2015; Basso & Suzuki, 2017).
Do Exercise Breaks Help Students With Attention Deficit Hyperactivity Disorder
(ADHD)?
Yes! ADHD is a neurodevelopmental disorder that is characterized by symptoms of inattention and
impulsivity (DSM-5; American Psychiatric Association, 2013). Although ADHD is recognized as one of the
most prevalent neurodevelopmental disorders in school-aged children (Vysniauske et al., 2020), what
many people do not realize is that it impacts older students too, and it is one of the leading disabilities
among post-secondary students (Gapin et al., 2015). In addition to fueling the PFC, exercise boosts the
neural chemicals that students with ADHD may lack, including dopamine (Wu et al., 2012). In fact,
exercise increases blood flow and dopamine in a similar way as drugs used to treat ADHD, such as Ritalin
and Adderall (Wigal et al., 2012). Research shows that 20 minutes of exercise helps children with and
453
without ADHD stay focused (Pontifex et al., 2013), and exercising for longer may be even better
(Vysniauske et al., 2020).
Can Exercise Breaks Help Anxious Students Learn?
Yes! Students with anxiety or depression have difficulty concentrating; in fact, it is part of the diagnosis
(DSM-5; American Psychiatric Association, 2013; Diamond, 2013). Exercise can reduce symptoms of
anxiety and depression (Hu et al., 2020; Jayakody, Gunadasa & Hosker, 2013; Stubbs et al., 2017),
promote resiliency, and help to prevent the onset of mental illness (Arida & Teixeira-Machado, 2021;
Childs & de Wit, 2014).
Is Extracurricular Physical Activity Important for Learning?
Physical activity guidelines for physical health recommend that children aged 5 to 17 years old get 60
minutes of moderate-to-vigorous physical activity every day (Tremblay et al., 2011). However, our
research suggests that the brain may function well with less. Our research found that elementary
students who are physically active for 1 to 2 days a week and secondary students who are physically
active for 3 to 4 days a week have better academic achievement than students who are not physically
active (Ogrodnik et al., 2020). These results suggest that it is no sweat to skip a day or two of
extracurricular activity during the week.
What Is Next for This Research?

More research is needed to uncover the optimal frequency, intensity, duration, and type of exercise
breaks that benefit learners and to examine whether exercise breaks enhance other instructional study
strategies such as practice quizzing.
In conclusion, active students focus longer and learn better. Teachers should consider providing
students with more opportunities to move throughout the class periods. Try incorporating exercise
breaks into a lesson or between lessons. Remember, some is better than none and all students should
feel comfortable and be able to participate in the experience. Movement fuels the PFC with the
resources it needs to keep the mind focused on learning. The bottom-line: Students who move more,
learn better.
Author Note
The authors would like to acknowledge the critical work of Kristen Lucibello and Dr. Joseph Kim on the
original research paper referenced in this article. The authors also acknowledge Paulina Rzeczkowska for
making the figures. Funding for the original article was provided by the Social Sciences and Humanities
Research Council of Canada to Dr. Jennifer Heisz.
References
American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.).
American Psychiatric Association Publishing.
https://doi.org/10.1176/appi.books.9780890425596
Arida, R. M., & Teixeira-Machado, L. (2020). The contribution of physical exercise to brain
resilience. Frontiers in Behavioral Neuroscience, 14, 626769.
https://doi.org/10.3389/fnbeh.2020.626769
454
Basso, J. C., Shang, A., Elman, M., Karmouta, R., & Suzuki, W. A. (2015). Acute exercise improves
prefrontal cortex but not hippocampal function in healthy adults. Journal of the International
Neuropsychological Society, 21(10), 791-801. https://doi.org/10.1017/s135561771500106x
Basso, J. C., & Suzuki, W. A. (2017). The effects of acute exercise on mood, cognition, neurophysiology,
and neurochemical pathways: a review. Brain Plasticity, 2(2), 127–152.
https://doi.org/10.3233%2FBPL-160040
Bedard, C., St John, L., Bremer, E., Graham, J. D., & Cairney, J. (2019). A systematic review and meta-
analysis on the effects of physically active classrooms on educational and enjoyment outcomes
in school age children. PloS one, 14(6), e0218633.
https://doi.org/10.1371/journal.pone.0218633
Carter, S. E., Draijer, R., Holder, S. M., Brown, L., Thijssen, D. H., & Hopkins, N. D. (2018). Regular walking
breaks prevent the decline in cerebral blood flow associated with prolonged sitting. Journal of
Applied Physiology, 125(3), 790–798. https://doi.org/10.1152/japplphysiol.00310.2018
Childs, E., & de Wit, H. (2014). Regular exercise is associated with emotional resilience to acute stress in
healthy adults. Frontiers in Physiology, 5, 161. https://doi.org/10.3389/fphys.2014.00161
Diamond, A. (2013). Executive functions. Annual Review of Psychology, 64, 135–168.
Fenesi, B., Lucibello, K., Kim, J. A., & Heisz, J. J. (2018). Sweat so you don’t forget: Exercise breaks during
a university lecture increase on-task attention and learning. Journal of Applied Research in
Memory and Cognition, 7(2), 261–269. https://doi.org/10.1016/j.jarmac.2018.01.012
Gapin, J. I., Labban, J. D., Bohall, S. C., Wooten, J. S., & Chang, Y.-K. (2015). Acute exercise is associated
with specific executive functions in college students with ADHD: A preliminary study. Journal of
Sport and Health Science, 4(1), 89–96. https://doi.org/10.1016/j.jshs.2014.11.003
Giles, G. E., Brunyé, T. T., Eddy, M. D., Mahoney, C. R., Gagnon, S. A., Taylor, H. A., & Kanarek, R. B.
(2014). Acute exercise increases oxygenated and deoxygenated hemoglobin in the prefrontal
cortex. NeuroReport, 25(16), 1320–1325. https://doi.org/10.1097/wnr.0000000000000266
Howie, E. K., Beets, M. W., & Pate, R. R. (2014). Acute classroom exercise breaks improve on-task
behavior in 4th and 5th grade students: a dose–response. Mental Health and Physical
Activity, 7(2), 65–71. https://doi.org/10.1016/j.mhpa.2014.05.002
Hu, M. X., Turner, D., Generaal, E., Bos, D., Ikram, M. K., Ikram, M. A., Cuijpers, P., & Penninx, B. W. J. H.
(2020). Exercise interventions for the prevention of depression: a systematic review of meta-
analyses. BMC Public Health, 20(1, article 1255). https://doi.org/10.1186/s12889-020-09323-y
Jayakody, K., Gunadasa, S., & Hosker, C. (2014). Exercise for anxiety disorders: systematic review. British
Journal of Sports Medicine, 48(3), 187–196. https://doi.org/10.1136/bjsports-2012-091287
Kane, M. J., Brown, L. H., McVay, J. C., Silvia, P. J., Myin-Germeys, I., & Kwapil, T. R. (2007). For whom the
mind wanders, and when: An experience-sampling study of working memory and executive
control in daily life. Psychological Science, 18(7), 614–621. https://doi.org/10.1111/j.1467-
9280.2007.01948.x
Lindquist, S. I., & McLean, J. P. (2011). Daydreaming and its correlates in an educational
environment. Learning and Individual Differences, 21(2), 158–167.
https://doi.org/10.1016/j.lindif.2010.12.006
455
Ma, J. K., Mare, L. L., & Gurd, B. J. (2014). Classroom-based high-intensity interval activity improves off-
task behaviour in primary school students. Applied Physiology, Nutrition, and
Metabolism, 39(12), 1332–1337. https://doi.org/10.1139/apnm-2014-0125
May, K. E., & Elder, A. D. (2018). Efficient, helpful, or distracting? A literature review of media
multitasking in relation to academic performance. International Journal of Educational
Technology in Higher Education, 15(1), 1–17. https://doi.org/10.1186/s41239-018-0096-z
McNerney, M. W., & Radvansky, G. A. (2015). Mind racing: The influence of exercise on long-term
memory consolidation. Memory, 23(8), 1140–1151.
Moriguchi, Y., & Hiraki, K. (2013). Prefrontal cortex and executive function in young children: A review of
NIRS studies. Frontiers in Human Neuroscience, 7, 867.
https://doi.org/10.3389%2Ffnhum.2013.00867
Naylor, P. J., Nettlefold, L., Race, D., Hoy, C., Ashe, M. C., Higgins, J. W., & McKay, H. A. (2015).
Implementation of school based physical activity interventions: A systematic review. Preventive
Medicine, 72, 95–115. https://doi.org/10.1016/j.ypmed.2014.12.034
Ogrodnik, M., Halladay, J., Fenesi, B., Heisz, J., & Georgiades, K. (2020). Examining associations between
physical activity and academic performance in a large sample of Ontario students: The role of
inattention and hyperactivity. Journal of Physical Activity and Health, 17(12), 1231–1239.
https://doi.org/10.1123/jpah.2020-0174
Pontifex, M. B., Saliba, B. J., Raine, L. B., Picchietti, D. L., & Hillman, C. H. (2013). Exercise improves
behavioral, neurocognitive, and scholastic performance in children with attention-
deficit/hyperactivity disorder. The Journal of Pediatrics, 162(3), 543–551.
https://doi.org/10.1016/j.jpeds.2012.08.036
Shelton, J. T., Elliott, E. M., Eaves, S. D., & Exner, A. L. (2009). The distracting effects of a ringing cell
phone: An investigation of the laboratory and the classroom setting. Journal of Environmental
Psychology, 29(4), 513–521. https://doi.org/10.1016%2Fj.jenvp.2009.03.001
Smallwood, J., & Schooler, J. W. (2006). The restless mind. Psychological Bulletin, 132(6), 946–958.
Stubbs, B., Vancampfort, D., Rosenbaum, S., Firth, J., Cosco, T., Veronese, N., Salum, G. A., & Schuch, F.
B. (2017). An examination of the anxiolytic effects of exercise for people with anxiety and stress-
related disorders: a meta-analysis. Psychiatry Research, 249, 102–108.
https://doi.org/10.1016/j.psychres.2016.12.020
Szpunar, K., Moulton, S., & Schacter, D. (2013). Mind wandering and education: From the classroom to
online learning. Frontiers in Psychology, 4, 495. https://doi.org/10.3389/fpsyg.2013.00495
Tremblay, M. S., Gray, C. E., Akinroye, K., Harrington, D. M., Katzmarzyk, P. T., Lambert, E. V., Liukkonen,
J., Maddison, R., Ocansey, R. T., Onywera, V. O., Prista, A., Reilly, J. J., Rodríguez Martínez, M. del
P., Sarmiento Duenas, O. L., Standage, M., & Tomkinson, G. (2014). Physical Activity of Children:
A Global Matrix of Grades Comparing 15 Countries. Journal of Physical Activity & Health, 11,
S113–S125. https://doi.org/10.1123/jpah.2014-0177
Tremblay, M. S., Warburton, D. E. R., Janssen, I., Paterson, D. H., Latimer, A. E., Rhodes, R. E., Kho, M. E.,
Hicks, A., LeBlanc, A. G., Zehr, L., Murumets, K., & Duggan, M. (2011). New Canadian physical
456
activity guidelines. Applied Physiology, Nutrition & Metabolism, 36(1), 36–
46. https://doi.org/10.1139/h11-009
Unsworth, N., McMillan, B. D., Brewer, G. A., & Spillers, G. J. (2012). Everyday attention failures: An
individual differences investigation. Journal of Experimental Psychology: Learning, Memory, and
Vysniauske, R., Verburgh, L., Oosterlaan, J., & Molendijk, M. L. (2020). The effects of physical exercise on
functional outcomes in the treatment of ADHD: A meta-analysis. Journal of Attention
Disorders, 24(5), 644–654. https://doi.org/10.1177/1087054715627489
Wigal, S. B., Emmerson, N., Gehricke, J. G., & Galassetti, P. (2013). Exercise: Applications to childhood
ADHD. Journal of Attention Disorders, 17(4), 279–290.
Wilkins, J., Graham, G., Parker, S., Westfall, S., Fraser, R., & Tembo, M. (2003). Time in the arts and
physical education and school achievement. Journal of Curriculum Studies, 35(6), 721–734.
Wu, J., Xiao, H., Sun, H., Zou, L., & Zhu, L. Q. (2012). Role of dopamine receptors in ADHD: A systematic
meta-analysis. Molecular Neurobiology, 45(3), 605–620. https://doi.org/10.1007/s12035-012-
8278-5
457
In Their Own Words Learning How to Learn (Better) in Introduction
to Psychology
to Apply the Science of Learning in Carolyn R. Brown-Kramer
University of Nebraska – Lincoln
Editors
Lauren L. Kordonowy
Victor A. Benassi
New college students often have intuitive ideas about how to study. Unfortunately, they are often
wrong. Even more unfortunate, they often figure out they are wrong only after the first exam yields a
disappointing score, or perhaps if they are in denial, after a second or third low exam score.
Many students rely on well-learned habits from high school, such as highlighting everything on the page,
rereading (or worse, skipping the assigned chapter and reading just the summary at the end),
multitasking, and cramming (Bjork et al., 2013; Morehead et al., 2016; see several chapters in this
volume). Even when they understand that some of their studying behaviors are not very effective,
students may persist with such habits due to familiarity or ease, lack of time due to procrastination, or
perceived lack of skill with new habits (Blasiman et al., 2017; Rowell et al., 2021). It appears that simply
telling people what they are doing is wrong and suggesting ways to improve is not enough.
Sound familiar?
I teach about 600 students in Introduction to Psychology each year—over 400 in the fall and another
180 in the spring--so I have a lot of conversations that begin with, “What am I doing wrong? I studied for
the exam, and I still got a bad grade.” Over the years I have tried a lot of different interventions to help
students learn how to study better. This chapter is about an assignment that actually worked to improve
students’ learning strategies and their exam scores.
Which Learning Strategies Are Effective?

Prior work has shown, not surprisingly, that some learning strategies are more effective than others.
Dunlosky and colleagues (2013) characterized 10 strategies as high, moderate, or low in utility, while
also pointing out substantial differences in the effectiveness of the strategies across different types of
learners, materials, tasks, and contexts. The high-utility strategies included practice testing and spaced
practice; the moderate-utility strategies included elaborative interrogation, self-explanation, and
interleaved practice; and the low-utility strategies included summarization, highlighting/underlining, use
of the keyword mnemonic, forming mental images, and rereading.
Broadly speaking, the more active and cognitively challenging the task, the more learning it produces
and the higher its utility in an academic setting. Note that many students’ favorite strategies (such as
highlighting and rereading) fall into the “low utility” category, whereas many less popular strategies
(such as spaced practice—that is, spreading studying out over days or weeks instead of cramming) are
among the most effective.
Research with college students has demonstrated that using high- and moderate-utility learning
strategies such as those shown above is associated with higher grades in both correlational and
experimental or quasi-experimental studies (Bartoszewski & Gurung, 2015; Cathey et al., 2016; Chen et
al., 2017; Dunlosky & Rawson, 2015).
458
There is a clear and important disconnect between what works and how many students study. How can
instructors help students to understand not only which learning strategies are more effective, but also
why some work better than others and how to move toward using more effective strategies in their own
academic lives?
What I Did in My Course

Below I describe the results of a three-year classroom intervention study that are described in more
detail in two recent papers. In the first paper (Brown-Kramer, 2021), I compared student outcomes
when I changed an old term paper assignment to focus on learning strategies, with students randomly
assigned to read and write about one of four different articles about a learning strategy. In the second
paper (Brown-Kramer, 2022), I replicated and extended my previous work by streamlining the term
paper assignment and examining different levels of academic achievement to see if the results differed
by grade point average (GPA).
Beginning in fall 2017, I conducted a classroom intervention study that was designed to improve my
Introduction to Psychology students’ use of learning strategies and, ultimately, their performance in the
class (see Figure 1 for the course timeline). For comparison, I treated the previous year (fall 2016) as a
control semester so I could assess the effectiveness of my intervention, and then I made further
refinements to the intervention the following year (fall 2018). In all three semesters, exams were
administered in the same weeks of the semester, and paper drafts and revisions were due in the same
weeks of the semester.
This is a high-enrollment course, with over 400 students enrolled at the beginning of each fall semester.
Students in the class are largely first-year (~81%), white (~74%), women (~ 69%), and young (~ 72% aged
17 or 18), and most students at my institution are from the Midwest USA (~ 76% in-state). During each
of the three 16-week semesters reported here, students were required to attend twice-weekly 75-
minute lectures held in a large performing arts center led by their professor (me) and a once-weekly 50-
minute recitation section of approximately 25 students led by a graduate teaching assistant. Recitation
is focused less on relearning and more on active learning, often in groups, to illustrate particularly
challenging concepts or to extend lecture material to applied contexts.
Figure 1
Course Timeline for Fall 2017 Semester
Note. In fall 2016, there were no surveys or random assignment to articles; in fall 2018 there was no
random assignment to articles. Otherwise, the timeline was the same across all three semesters.
I have had a term paper assignment in my Introduction to Psychology course for years. This assignment
involves reading an empirical research article from a psychology journal, then writing a 3–4-page paper
that summarizes, critiques, and applies the article. Paper assignments like this are often used in
Introduction to Psychology to expose students to scientific writing and methodology in psychology,
strengthen critical thinking, and give them practice with original sources. To help students prepare for
the term paper assignment each semester, we spent one recitation period working through a sample
459
research article on a topic unrelated to learning strategies. We laid out the purpose and structure of a
psychological research article, and then students worked through the sample article to practice
summarizing and critiquing elements of the article (literature review, methods, results, etc.). Since I
already had the term paper assignment integrated into my course, I decided to keep the essence of the
assignment, retained most of the original grading rubric for consistency, and I just changed the article to
focus on learning strategies.
In fall 2017 (see Brown-Kramer, 2021), I randomly assigned my students to complete their paper on one
of four different articles, each describing an empirical test of the efficacy of a different learning strategy.
The articles were similar in length (9-13 pages), research design (experimental), and reading difficulty,
and included the following:
• Seabrook et al. (2005) on spaced practice (studies 2 and 3 only);
• Rawson and Kintsch (2005) on rereading;
• McDaniel et al. (2011) on practice testing;
• Schmeck et al. (2014) on forming mental images while learning.
I scaled students’ scores on the four conditions of the paper assignment to prevent any grade inequity
on the assignment, so after this scaling there were no significant differences in paper grades across the
conditions.
In addition to the paper assignment, I surveyed students at the beginning and end of the semester about
their use of the 10 learning strategies listed above (survey adapted from Bartoszewski & Gurung, 2015).
I also collected their scores on four 50-question multiple-choice exams, with the first and second exams
taking place before they read their assigned article, the third exam taking place after they submitted a
draft of their paper for feedback, and the fourth exam taking place after they submitted their final
paper. Additionally, I collected their paper grades and overall course grade.
Note that I did not instruct nor recommend students to implement the strategies they read about in
their articles. If they chose to implement a learning strategy they had read (and written) about, that was
up to them. While each of the articles provided evidence of the effectiveness of the strategy described
in the article, including ideas on how they could be implemented, students could choose to study
however they saw fit at any point throughout the semester, with no bonus or penalty.
What I Found
Of the 416 students in the course in fall 2017, 361 (86.8%) consented to have their data used for
research purposes and completed one or both of the surveys. The results showed a few interesting
trends.
First, over the course of the semester, students reported that they decreased their use of low-utility
strategies and increased their use of moderate-to-high utility strategies.
Second, students who completed the term paper assignment focused on practice testing—a high-utility
strategy—performed better on the third and fourth exams compared to students in the other three
conditions, scoring an average of six percentage points higher. Students in the practice testing condition
also performed better in the course overall, finishing the course with an average percentage of 84.37%
versus an average of 77.74% for students in the other three conditions—a grade of B versus C+. Bear in
mind that students were randomly assigned to complete their different term paper articles (rather than
allowing students to self-select which paper they wanted to read), so any differences in outcomes can
be attributed to the manipulation. In other words, we can conclude that the term paper assignment
460
actually caused improvements in students’ exam scores and overall course grades—and not merely that
there is an association between the article read and students’ course performance.
Third, I compared students’ performance to the previous year (fall 2016), which covered the same
course content and used the same exams, assignments, course structure, teaching style, etc., but
involved a term paper assignment on an article unrelated to learning strategies. Student performance
on the first two exams—before the introduction of the paper assignment—was the same across
semesters, which suggested that the groups of students enrolled in the two classes were not
systematically different from one another at the beginning of the semester and that the courses were
equivalent before the paper assignment intervention. On the third and fourth exams, however, students
did better in fall 2017 than in fall 2016—that is, they did better if they had written their papers about a
learning strategy rather than about an unrelated topic. Looking at final course grades, students in the
intervention semester outperformed those in the control semester, ending up with an average grade of
B- overall rather than C+ overall (see Table 1).
Table 1
Course performance across semesters; results from Brown-Kramer (2022).
Fall 2017
Fall 2018
Fall 2016 (paper assignment on 4
(paper assignment on
(control) different learning
practice testing)
strategies)
Exam 1 M = 79.0, SD = 13.4 M = 77.5, SD = 12.5 M = 79.1, SD = 12.3
Exam 2 M = 72.6, SD = 16.6 M = 73.7, SD = 14.3 M = 74.9, SD = 14.5
Exam 3 M = 72.8, SD = 15.9 M = 79.7, SD = 15.8 M = 79.2, SD = 14.5
Exam 4 M = 68.9, SD = 13.9 M = 78.3, SD = 12.4 M = 75.8, SD = 12.3
Final course M = 78.7, SD = 16.0 M = 81.3, SD = 16.5 M = 82.5, SD = 16.0
grade (C+ average) (B- average) (B- average)
Finally, results showed that students who reported using more self-explanation, elaborative
interrogation, and practice testing—all of which are considered “moderate” or “high” utility strategies
(Dunlosky et al., 2013)—performed better on exams and in the course overall. On the other hand,
reported use of the low-utility strategies, such as highlighting/underlining, use of keywords, forming
mental images, and rereading was unrelated to performance, meaning that students who reported
using these strategies heavily did no better than students who reported barely using them at all. Finally,
reported use of summarization was actually negatively related to students’ overall course grades.
These results were eye-opening to me. It was thrilling to find the same result that others have
demonstrated—that reported use of certain learning strategies really did produce better grades than
others. I was also excited to see that students reported improvement in their learning strategies over
time, which (I hope) meant that they were adapting to the rigors of college in beneficial ways. Most
important, my intervention worked! Students who wrote their term paper about learning strategies
performed better than students who wrote about something else, and students who wrote about one
learning strategy in particular—practice testing—did best of all.
461
The Next Step: Replication and Extension
Armed with this knowledge from my fall 2017 course, the following year I refined the term paper
assignment further, while also fixing some of the issues I encountered—things like student concerns
about fairness due to different students having to write about different articles, and the time required
to grade papers about four different articles rather than one. Here is what I did the second time around
(Brown-Kramer, 2022).
In fall 2018, the class was a similar size and had similar student demographics, with 402 students
enrolled and 385 students (95.8%) agreeing to have their data used for research purposes. Once again, I
kept the same exams, assignments, course structure, and teaching style as before. I again surveyed
students about their learning strategies at the beginning and end of the semester, with the exams and
surveys administered in the same weeks of the semester as in fall 2017. I made one change to the term
paper assignment. Rather than assigning students to read and write about one of four different articles,
I assigned all students to read and write about the one article that had produced the best outcomes the
previous year—the article on practice testing (McDaniel et al., 2011). I wanted to see if my previous
findings would be replicated or were just a one-time fluke. I also wanted to see if the term paper
assignment was more effective for some students than for others, so I decided to look at a measure of
academic achievement as well—grade point average (GPA). Because most students were first-years, I
asked them to report their college GPA if they had one, or their high school GPA otherwise.
As with my previous study, students generally reported that they increased their use of high-utility
strategies and reduced their use of lower-utility strategies throughout the semester. But this trend
nearly disappeared when I accounted for GPA. In essence, that means that the students who said that
they switched to using more of the effective strategies and fewer of the ineffective strategies by the end
of the semester were those who came in the door with higher GPAs. In contrast, the students who said
they stuck with the same (often ineffective) strategies all semester tended to be those with lower GPAs.
I also found once again that when students reported use of several of the moderate- and high-utility
strategies at the end of the semester, they performed better on the third and fourth exams as well as in
the course overall. When I took GPA into account, there were three learning strategies that stood above
the rest. Reported use of practice testing, self-explanation, and spaced practice were unique predictors
of exam and course grades. In other words, if you have students at diverse levels of academic
achievement, the more they report using these three strategies, the better they will do across the
board. Put yet another way, these three strategies seemed to “lift all boats,” helping both lower-
achieving and higher-achieving students. In contrast, when students increased their reported use of
ineffective strategies, such as highlighting/underlining and forming mental images over the course of the
semester, they actually did worse in the course, even after controlling for GPA.
Last but not least, I checked to see if the more streamlined version of the term paper assignment in fall
2018 worked as well as the version the previous year involving four different articles. Happily, both
versions worked equally well in terms of similar scores on exams three and four and in the course
overall (see Table 1). Given the choice between a logistically challenging assignment and a more
streamlined assignment that accomplishes the same thing, I would choose the streamlined one.
Take-Home Points and Recommendations

The key messages I learned from these two studies include:
• It matters how students spend their studying time. Those who report using more effective
strategies—like practice testing, spaced practice, self-explanation, and interleaved practice—
462
perform better on exams than those who report using less effective strategies such as
summarizing, highlighting or underlining, and rereading.
• Students generally report improving their learning strategies over time, but improvements can
be uneven. Over the semester, students tended to report using effective strategies more and
ineffective strategies less, but without intervention, these improvements tend to be
concentrated among the high-achieving students. Interventions such as the term paper
described here may help shrink the disparity and help all students—not just the A students—
improve their learning strategies.
• Some reported increases in strategies can help students overcome weaker academic
backgrounds. Reports of practice testing, self-explanation, and spaced practice helped students
across the board—not just those who came in with strong grades and high standardized test
scores.
• Classroom interventions can improve students’ reported use of learning strategies and grades. I
made a pretty simple switch—replacing an article in a term paper assignment with another
article, one focusing on learning strategies. Based on data I collected, I refined the assignment
further to make my life easier without harming student outcomes.
After years of telling students what they should be doing as they study and trying a variety of class
demonstrations and brief assignments that never seemed to produce substantial change, I found
something that worked for my class. Perhaps it was the more in-depth nature of the assignment;
perhaps it was the requirement that students examine closely the scientific basis of the learning
strategies they read about. I do not know exactly why, but that is one focus of ongoing investigation.
Introduction to Psychology is a natural course to target for interventions to improve students’ study
behaviors. It is often one of the most popular courses on campus that draws students from a wide
variety of majors, it enrolls largely first-year students who might struggle to study and learn effectively
during the college transition, and it already includes content on cognitive principles such as learning and
memory that can help students see why some learning strategies are more effective than others.
However, even upper-level courses may present excellent opportunities to help boost students’ use of
effective learning strategies. Cognitive psychology includes extensive coverage of learning and memory,
developmental psychology covers changes in cognitive abilities and skills over time, brain and behavior
courses can draw on neurological bases of learning and behavior, and social psychology could
incorporate social aspects of studying and learning.
It is also worth noting that both of my studies involved incremental, evidence-based changes rather than
major course overhauls that can be overwhelming and lead to instructor burnout. I would encourage
other educators to consider making simple switches like this and look at the data to inform further
changes and refinements. I would like to reiterate Bernstein’s (2018) advocacy for incremental change in
pedagogical revisions:
[The] incremental approach has the advantage of allowing teachers to empirically evaluate the
impact of each new [component of a course] and to retain or discard it depending on its value
for promoting learning, critical thinking, student satisfaction, or other valued outcomes (p. 299).
By embedding this term paper assignment or similar assignments into your courses, you may improve
students’ learning strategies and boost their learning and course performance using evidence-based
practices. At the same time, you can support your own well-being by making small, sustainable changes
that add up to big improvements.
463
Author Note
Please address correspondence to: [email protected]
References
exam score. Scholarship of Teaching and Learning in Psychology, 1(3), 219–228.
Bernstein, D. A. (2018). Does active learning work? A good question, but not the right one. Scholarship
of Teaching and Learning in Psychology, 4(4), 290-307. https://doi.org/10.1037/stl0000124
Annual Review of Psychology, 64(1), 417–444. https://doi.org/10.1146/annurev-psych-113011-
143823
Comparing intended versus actual study behaviour. Memory, 25(6), 784–792.
how to learn” assignment. Teaching of Psychology, 48(1), 48-54.
Brown-Kramer, C. R. (2022). Teaching students to learn better in Introductory Psychology: A replication
and extension study. Teaching of Psychology. Advance online publication.
Cathey, C. L., Visio, M. E., Whisenhunt, B. L., Hudson, D. L., & Shoptaugh, C. F. (2016). Helping when they
are listening: A midterm study skills intervention for introductory psychology. Psychology
Learning and Teaching, 15(3), 250–267. https://doi.org/10.1177/1475725716646319
Chen, P., Chavez, O., Ong, D. C., & Gunderson, B. (2017). Strategic resource use for learning: A self-
administered intervention that guides self-reflection on effective resource use enhances
academic performance. Psychological Science, 28(6), 774–785.
Dunlosky, J., & Rawson, K. A. (2015). Practice tests, spaced practice, and successive relearning: Tips for
classroom use and for guiding students’ learning. Scholarship of Teaching and Learning in
Rowell, S. F., Frey, R. F., & Walck-Shannon, E. M. (2021). Intended and actual changes in study behaviors
in an introductory and upper-level psychology course. Teaching of Psychology, 48(2), 165-174.
464
In Their Own Words Peer Discussions Improve Student Learning

Jonathan G. Tullis
You to Know About Why and How University of Arizona,
Editors
Robert L. Goldstone
Lauren L. Kordonowy
Indiana University
Victor A. Benassi
This chapter is based on: Tullis, J. G., & Goldstone, R. L. (2020). Why does peer instruction benefit
student learning? Cognitive Research: Principles and Implications. 5:15.
Building a large and versatile toolkit of teaching strategies that engage and efficiently teach students is
crucial for effective teachers. Many university instructors are trying to move away from staid didactic
lectures to more mentally lively activities. One engaging, active instructional strategy that is gaining use
across many university classrooms is called peer instruction (Mazur, 1997). In fact, more than 25% of
university physics professors report using peer instruction (Henderson & Dancy, 2009). During peer
instruction, teachers present a challenging problem to students, students answer the question on their
own, then students discuss their answers with a partner in the class, and finally students answer the
question on their own again.
In our research, we examined whether and why peer instruction benefits learning in undergraduate and
graduate psychology classes. Students’ answers following discussion are typically more accurate than
their answers before discussion. However, one concern with using peer instruction is that the more
knowledgeable partner just tells the less knowledgeable student what the correct answer is. This kind of
direct transmission of knowledge from the more knowledgeable student to the less knowledgeable
student may involve shallow learning and not produce long-lasting learning benefits. Alternatively, peer
instruction may prompt students to actively engage with each other to test ideas and yield a new
understanding that neither student possessed prior to their interaction. We tested whether peer
instruction encourages knowledge transmission or knowledge generation by assessing students’
answers and their confidence in their answers before and after discussion. More specifically, we
analyzed whether students just choose the answer of the more confident partner during discussion or
whether the discussion between the partners generates novel information.
What We Did
We tested when and why peer instruction benefits learning across 6 different courses. These courses
ranged from large undergraduate introductory courses in psychology to small master’s level graduate
courses in educational psychology at two public universities. In each of the six courses, we posed a
multiple-choice question related to course content and students had to answer it on their own first. We
required students to write down the answer to the question on their own (or submit it through a clicker
device). We also required students to report their confidence in their answer on a scale of 1 (not at all
confident) to 10 (very confident). After that, students discussed the question and answer with a
neighbor in the class. After a brief (less than five minutes) discussion, students answered the question
again on their own and reported their confidence in their second answer. We measured confidence to
test whether who is more confident played an important role in students’ final answers. (When
implementing peer discussion in your classes, you do not need to have students report their
465
confidence.) After students answered the second time, the instructor provided the correct answer and
its justification.
Minor procedural details differed between courses. For example, in the large enrollment courses,
students reported their answers and confidences through clicker systems, while in smaller courses
students wrote their answers and confidences on paper. Most of the questions posed, regardless of
their correctness, contributed to students’ participation grades, but some of the questions were graded
for accuracy.
Questions were designed to test the application and transfer of conceptual knowledge. An example of a
question was:
Your online exam software randomizes your Exam 2 questions and answers. However, when you take
the test, 4 questions in a row are "A". You think that the next answer CANNOT be "A". What is this an
example of?
a. Illusion of control
b. Gambler’s fallacy [the correct answer]
c. Hot Hand fallacy
d. Regression to the mean
Overall, we collected over 1,600 full answers to 86 different questions from more than 200 students.
What We Found
The average accuracy in answering the questions improved from pre-discussion to post-discussion in
each of the 6 courses, as shown in Figure 1.
Figure 1
Average Proportion Correct in Each of the Six Courses in Our Study Before and After Peer Discussion
466
Almost all of the students benefited from peer instruction. In fact, when averaged across all the
questions in our study, only 12% of students showed a decrease in their accuracy from pre- to post-
discussion. This is important because it shows a wide variety of students, including both high-performing
and low-performing students, benefited from engaging in discussions with their peers. Similarly,
students’ accuracy improved on the vast majority of the questions; only 9 questions (out of 86) showed
worse accuracy following discussion than before discussion. These data are shown in Figure 2. The data
show that accuracy improved just as much on very difficult questions as on easier questions.
The horizontal axis of Figure 2 represents the average proportion of correct responses to a question
before discussion, while the vertical axis shows the average proportion correct after discussion. Each
point on the graph represents a single question; the green diamonds show questions on which
performance improved from pre- to post-discussion, the orange circles show questions where
performance did not change (along the diagonal), and the red squares show questions where
performance decreased after discussion. The large majority of questions show improvements from pre-
to post-discussion (green diamonds). Further, questions ranging from difficult (left side of the graph) to
easy (right side of the graph) benefited from peer discussions.
Figure 2
A plot of performance on individual questions
Improvements in accuracy are driven by students appropriately shifting answers away from incorrect
answers and towards correct ones. Across all answers, 28% of incorrect answers were switched to
correct following discussion, while only 5% of correct answers were switched to incorrect, even though
there were multiple ways for an answer to be incorrect and only one way for it be correct. Switching
answers usually happened when peers initially disagreed; when peers agreed, they changed their
answers less than 3% of the time. However, students were reluctant to switch their answers even when
467
they disagreed with their peer. Students kept their answer 66% of the time when they disagreed with
their partner. These results show that students trust in their own answers more strongly than their
partner’s answer.
One of our central questions was whether accuracy improves because students just switch their answers
to the more confident partner in the pair. The data showed that students were doing much more than
choosing the answer of the more confident peer. For example, when partners disagreed, peers only
chose the answer of the more confident partner 58% of the time (which is far below 100%). Further, if
students always chose the more confident answer, their final accuracy would have been 69% on
average; instead, it was 72% (a small, but significant advantage to not always choosing the answer of the
more confident partner). Further analyses showed that students were drawn towards the correct
answer, even after we accounted for initial confidences. Discussion productively helped students find
the correct answer. In other words, peer discussion allowed students to test the coherence of their
answers and generate new knowledge. Students did not simply transmit answers from the more
confident partner to the less confident partner. One impressive result in this regard was that when
students originally agreed on the same incorrect answer, they were more than four times more likely to
switch to the correct answer than they were to switch to an incorrect answer when they originally
agreed on the correct answer. Interactive discussion apparently helped students to forge better
understandings even when neither student showed proper understanding at first.
We also examined how peer discussions impacted students’ confidence in their answers. Students
became more confident in their answers, particularly when their answers were correct (although their
confidence also increased in incorrect answers), through discussions with partners. Further, discussion
prompted students to more precisely judge the accuracy of their answer. Students were more confident
in correct answers (and less confident in incorrect answers) following discussion than before discussion.
Why Peer Instruction Works

There are several reasons why this format of peer discussions may benefit student learning. First,
students are required to answer the question first on their own. Answering practice questions prompts
students to bring to mind relevant prior knowledge and use it to solve the novel problems. Considerable
research shows the benefits of asking practice questions for the long-term retention of information.
Specifically, trying to access information from memory helps us remember that information more than
just rereading it or listening to it (Yang et al., this volume). Further, answering practice questions can
help students recognize what they know and do not know, organize existing knowledge, and learn from
subsequent instruction. Finally, by answering the question first on their own, they enter into discussion
with thoughts about the answers and an alternative for which they can advocate.
Second, beyond just the retrieval of information, peer instruction may be additionally helpful because it
prompts peer interactions. Peer interactions, especially when there are disagreements within a pair, are
likely to prompt argumentation and discussions of reasoning (Trouche et al., 2014). During
argumentation, students must explain their answers to their partner. The process of explaining an
answer may prompt students to notice gaps in their understanding, detect and correct errors in their
explanations, and help construct new knowledge (e.g., Schwartz, 1995). In fact, self-explanations, in
which a student verbalizes justification and reasons to themselves, can support student learning by
themselves (Chi et al., 1994).
Third, discussing with a peer may be more beneficial than receiving an explanation from the teacher,
who may not understand the perspective of the novice student. Peers may better understand the
knowledge of their classmates and be able to provide clearer reasoning in simple, familiar terms than
experts can (Brown & Palincsar, 1989; Noddings, 1985; Tullis, 2018; Vygotsky, 1981). In addition,
468
students may be particularly motivated to have their peers like and respect them. People are heavily
influenced by the opinions of others, and social psychologists have distinguished between expectations
that come from one’s peers (descriptive norms) versus higher authorities (injunctive norms) (Schultz et
al., 2007). Descriptive norms that come from one’s peers are especially influential to learning (Cialdini,
2007). For example, people are more likely to litter when they observe a lot of other litter on the
ground, even though they know that littering is against the official rules. Messages such as “Many
people litter. Don’t be one of them!” may have the paradoxical effect of increasing littering because it
suggests a descriptive norm that littering is common among fellow citizens (Cialdini et al., 1990). Given
that students are likely to relate to each other better than they do their teacher and that they care a lot
about what their peers think of them, giving and receiving explanations with peers often leads to
particularly strong learning.
Finally, beyond the cognitive benefits of peer instruction expressed above, peer instruction may allow a
greater number of students to mentally wrestle with the material. In a large class, time limits preclude
most students from explaining their ideas to the teacher. Peer instruction gives every student time to
verbalize their thoughts. Further, students who may be hesitant to volunteer their ideas in front of an
entire class may be more likely to voice their ideas and argue for their answer when debating with a
single, fellow student. In fact, we have ongoing research that compares learning from peer instruction
with a class-wide discussion of problems. This research suggests that peer instruction supports student
learning better than class-wide discussion. Further, other current research suggests that peer instruction
is most beneficial when students work in pairs. As the size of the group increases, students take less
responsibility for answering the question, engage with the material less deeply, and forget the
information more quickly.
Benefits of Peer Instruction Beyond Our Study

Other research shows that peer instruction benefits the application of learning to new problems. For
example, Smith et al. (2009) examined peer instruction in college biology classes. Students’ accuracy
improved from about 50% correct before discussion to 70% correct following discussion. Then, students
were asked to answer a new, similar question on their own. Students showed that they could apply their
knowledge learned during peer discussions to new problems. On new problems that were
approximately as difficult as the original problems, students answered about 75% of the new problems
correctly on their own (much better than the 50% correct they achieved originally on their own). That
study shows that students were able to transfer what they learned during peer instruction to solving
new problems on their own, which is a difficult and remarkable feat! Further, we have some preliminary
data which shows that peer instruction helps students remember the information for a longer time than
just answering a question on their own or having a class-wide discussion.
There are countless variations to the general procedures of peer instruction that can be adapted to a
specific teacher’s preferences. For example, in classes where students answer the questions via digital
clickers, some teachers opt to show the distribution of the class’s answers before peer discussion
(Nielsen et al., 2012; Perez et al., 2010). Some teachers grade the accuracy of student answers for
correctness (James, 2006), while other teachers use answering as a simple measure of participation.
Some instructors use multiple-choice questions during peer instruction, while others administer short
answer questions or complex problems. Some teachers choose to use peer instruction as a bell work
problem at the beginning of every class to refresh students’ minds about material covered in prior
classes, while others choose to use it in the middle of class to break up lecture and emphasize key ideas.
Peer instruction can be implemented in many different ways and still promote student learning. We
strongly advocate that peer instruction have at least the following components: 1. a requirement that
469
students answer the question first on their own so that they can retrieve relevant knowledge by
themselves and take responsibility for their answer (they thereby have made a commitment upon
entering the discussion phase), 2. discussion with a partner, so that they can verbalize their reasoning,
hear a partner’s reasoning, and possibly debate the partner, and 3. after submission of the final answer,
clarification and justification of the correct answer so that misconceptions or incorrect answers are
addressed (debate can help students prepare to learn from an explanation).
Peer instruction is a more specific strategy than “think-pair-share” (e.g., Kaddoura, 2013). Think-pair-
share can have multiple correct answers and does not necessarily encourage students to debate or
challenge each other. A key feature of peer instruction is that students need to argue and justify their
own reasoning when arguing in favor of a correct answer. This set-up likely works best when there is
only a single correct answer in order to foster debate among partners.
Peer instruction has broader learning benefits than just improving conceptual learning. Centering
classroom activities around peer instruction (and away from didactic lectures) improves student
retention in difficult courses (Lasry et al., 2008), increases passing rates (Porter, Bailey-Lee, & Simon,
2013), improves student attendance (Deslauriers et al., 2011), and bolsters student engagement (Lucas,
2009) and attitudes toward their course (Beekes, 2006).
Research shows that these benefits of peer instruction can be found in many different domains,
including physics (Mazur, 1997), biology (Knight, Wise, & Southard, 2013), chemistry (Brooks & Koretsky,
2011), physiology (Cortright et al., 2005), calculus (Lucas, 2009), computer science (Porter et al., 2013),
entomology (Jones et al., 2012), and philosophy (Butchart et al., 2009). We believe that the benefits of
peer instruction would extend even more broadly and likely benefit learning regardless of the domain of
the content. Further, benefits of peer instruction have been found in many different student
populations, from students in high school (Cummings & Roberts, 2008) to those in private universities
(Lasry et al., 2008).
Peer instruction has shown learning benefits in a wide range of domains, with many types of questions,
and across varied kinds of learners. Incorporating some peer instruction throughout classes is a
relatively simple, flexible, and quick method for encouraging student participation and learning.
Teachers can and should incorporate peer instruction into their teaching toolkit because peer
instruction encourages students to process information deeply, test their ideas, and derive new
knowledge.
Author Note
If you have any questions or comments, please contact the first author at [email protected]
References
Beekes, W. (2006). The ‘‘millionaire” method for encouraging participation. Active Learning in Higher
Education, 7(1), 25–36. https://doi.org/10.1177/1469787406061143
Brooks, B.J., Koretsky, M.D. (2011). The influence of group discussion on students’ responses and
confidence during peer instruction. Journal of Chemistry Education, 88(11), 1477–1484.
https://doi.org/10.1021/ed101066x
Brown, A. L., & Palincsar, A. S. (1989). Guided, cooperative learning and individual knowledge
acquisition. In L. B. Resnick (Ed.), Knowing, learning, and instruction: Essays in honor of Robert
Glaser (pp. 393-451). Erlbaum.
470
Butchart, S., Handfield, T., & Restall, G. (2009). Using peer instruction to teach philosophy, logic and
critical thinking. Teaching Philosophy, 32, 1–40. https://doi.org/10.5840/teachphil20093212
Chi, M. T. H., De Leeuw, N., Chiu, M., & Lavancher, C. (1994). Eliciting self-explanations improves
understanding. Cognitive Science, 18(3), 439-477.
Cialdini, R. B. (2007). Descriptive social norms as underappreciated sources of social control.
Psychometrika, 72(2), 263-268. https://doi.org/10.1007/s11336-006-1560-6
Cialdini, R. B., Reno, R. R., & Kallgren, C. A. (1990). A focus theory of normative conduct: Recycling the
concept of norms to reduce littering in public places. Journal of Personality and Social
Cortright, R.N., Collins, H.L. & DiCarlo, S.E. (2005) Peer instruction enhanced meaningful learning: ability
to solve novel problems. Advances in Physiology Education, 29, 107–111.
https://doi.org/10.1152/advan.00060.2004
Cummings, K. & Roberts, S. (2008). A study of peer instruction methods with school physics students. In
C. Henderson, M. Sabella, & L. Hsu (Eds.), Physics Education Research Conference (pp. 103–106).
American Institute of Physics.
Deslauriers, L., Schelew, E., & Wieman, C. (2011). Improved learning in a large-enrollment physics class.
Science, 332(6031), 862–864. https://doi.org/10.1126/science.1201783
Henderson, C., & Dancy, M. H. (2009). The impact of physics education research on the teaching of
introductory quantitative physics in the United States. Physical Review Special Topics: Physics
Education Research, 5(2), 020107. https://link.aps.org/doi/10.1103/PhysRevSTPER.5.020107
James, M. C. (2006). The effect of grading incentive on student discourse in peer instruction. American
Journal of Physics, 74(8), 689–691. https://doi.org/10.1119/1.2198887
Jones, M. E., Antonenko, P.D., & Greenwood, C. M. (2012). The impact of collaborative and
individualized student response system strategies on learner motivation, metacognition, and
knowledge transfer. Journal of Computer Assisted Learning, 28(5), 477-487.
Kaddoura, M. (2013). Think pair share: A teaching learning strategy to enhance students’ critical
thinking. Educational Research Quarterly, 36(4), 3-24.
Knight, J. K., Wise, S. B., & Southard, K. M. (2013). Understanding clicker discussions: Student reasoning
and the impact of instructional cues. CBE-Life Sciences Education, 12(4), 645-654.
Lasry, N., Mazur, E., & Watkins, J. (2008). Peer instruction: from Harvard to the two-year college.
American Journal of Physics, 76(11), 1066–1069. https://doi.org/10.1119/1.2978182
Lucas, A. (2009). Using peer instruction and i-clickers to enhance student participation in calculus.
Primus, 19(3), 219-231. https://doi.org/10.1080/10511970701643970
Mazur, E. (1997). Peer instruction: A user’s manual. Prentice Hall.
Nielsen, K.L., Hansen-Nygård, G., & Stav, J. B. (2012). Investigating peer instruction: how the initial
voting session affects students’ experiences of group discussion. ISRN Educ 2012, article 290157.
http://dx.doi.org/10.5402/2012/290157
471
Noddings, N. (1985). Small groups as a setting for research on mathematical problem solving. In E. A.
Silver (Ed.), Teaching and learning mathematical problem solving (pp. 345-360). Erlbaum.
Perez, K.E., Strauss, E.A., Downey, N., Galbraith, A., Jeanne, R., Cooper, S., & Madison, W. (2010). Does
displaying the class results affect student discussion during peer instruction? CBE Life Sciences
Education, 9(2), 133–140. https://doi.org/10.1187/cbe.09-11-0080
Porter, L., Bailey-Lee, C., & Simon, B. (2013). Halving fail rates using peer instruction: a study of four
computer science courses. In: SIGCSE ’13: Proceedings of the 44th ACM Technical Symposium on
Computer Science Education, ACM Press, 177–182.
Schultz, W.P., Nolan, J.M., Cialdini, R.B., Goldstein, N.J., and Griskevicius, V. (2007). The constructive,
destructive, and reconstructive power of social norms. Psychological Science, 18(5), 429–434.
Schwartz, D. L. (1995). The emergence of abstract representations in dyad problem solving. The Journal
of the Learning Sciences, 4(3), 321–354. https://doi.org/10.1207/s15327809jls0403_3
Smith, M. K., Wood, W. B., Adams, W. K., Wieman, C., Knight, J. K., Guild, N., Su, T.T. (2009). Why peer
discussion improves student performance on in-class concept questions. Science, 323(5910),
122–124. https://doi.org/10.1126/science.1165919
Trouche, E., Sander, E., & Mercier, H. (2014). Arguments, more than confidence, explain the good
performance of reasoning groups. Journal of Experimental Psychology: General, 143(5), 1958–
Tullis, J. G. (2018). Predicting others' knowledge: Knowledge estimation as cue-utilization. Memory &
Tullis, J. G., & Goldstone, R. L. (2020). Why does peer instruction benefit student learning? Cognitive
Research: Principles and Implications, 5(article 15). https://doi.org/10.1186/s41235-020-00218-
5
Vygotsky, L. S. (1981). The genesis of higher mental functioning. In J. V. Wertsch (Ed.), The concept of
activity in Soviet psychology (pp. 144-188). Sharpe.
472
In Their Own Words Using Online and Clicker Quizzes to Learn
Scientific and Technical Jargon
to Apply the Science of Learning in Steven C. Pan
National University of Singapore
Editors
Lauren L. Kordonowy
James E. Cooke
Victor A. Benassi
University of California, San Diego
Jeri L. Little
California State University, East Bay
Mark A. McDaniel
Washington University in St. Louis
Do you know what an “aliphatic hydrocarbon” is? Or a “ternary operator”? Here is a hint: The former is
an organic chemical compound, and the latter is a computer programming construct. In many science,
technology, engineering, and mathematics (STEM) courses, it is necessary to learn numerous such
jargon terms—that is, discipline-specific vocabulary words that are rarely used anywhere else (or are
used in a very particular way by a scientific or technical discipline). In fact, some high school science
textbooks contain well over 3,000 jargon terms (Groves, 1995; Yager, 1983), which exceeds the number
of words that is introduced in many foreign language courses!
Challenges Related to Learning Jargon

Learning jargon can be time-consuming and frustrating. However, in many STEM courses, mastering
jargon terms and their definitions is essential in order to understand assigned readings and lecture
materials. Without knowledge of jargon, it may be impossible to communicate effectively, attain deep
conceptual understanding, and perform well on course exams (Arons, 1973; Pan et al., 2019;
Wandersee, 1988). Accordingly, in order to help students, instructors should be aware of effective ways
in which students could learn jargon. Yet, rarely are they aware. Consequently, most—maybe all—
students are left to learn jargon with little guidance from their instructors, with mixed results.
Why is jargon learned ineffectively? A major reason is that there has been, until fairly recently,
surprisingly little research on productive ways to learn jargon. Instead, researchers have focused on
avoiding jargon, such as by rewriting introductory course materials using plain language (e.g., Brown &
Ryoo, 2008; McDonnell et al., 2015). Although that approach can improve reading comprehension, many
instructors do not have the time to remove and replace all of the jargon terms in their course materials.
Moreover, those terms still need to be learned at some point. The question remains: Are there effective
ways to help students learn jargon in the midst of learning course content? And if so, can any such
methods be used in a variety of courses, by different instructors, and without extensive amounts of
preparation?
We considered several possibilities. Perhaps students might study lists of jargon terms and their
definitions. Such lists are usually easy to make. However, students are not necessarily sophisticated in
how to study such material, and they may rely on repetitive review of jargon lists. An abundance of
research has shown that simply reading and restudying materials is not an effective way to learn (e.g.,
Callender & McDaniel, 2009). Another option might be having students look up and copy jargon terms
473
and definitions from a textbook. Such activities would also be relatively easy to implement. However, as
with studying, copying is not necessarily effective for learning (e.g., Bretzing & Kulhavy, 1979).
Ultimately, we felt that a learning technique with a stronger track record would have a greater chance of
success. After careful deliberation, we chose to explore one of the more promising techniques known to
learning science: retrieval practice.
Retrieval practice involves attempting to recall previously learned information from memory. Common
implementations such as practice quizzes, low-stakes tests, and flashcards can yield a host of learning
benefits, including better long-term retention, improved understanding and, in some cases, successful
transfer of learning (Pan & Rickard, 2018; Roediger & Karpicke, 2006). These benefits have been
demonstrated for materials ranging from history facts to science concepts (e.g., Carpenter et al., 2009;
McDaniel et al., 2007; Pan et al., 2018; for discussion, see Yang et al., this volume), and the benefits
obtain with both short-answer and multiple-choice test formats (McDaniel & Little, 2019). Collectively,
the research suggests that tests can be used to enhance learning in classroom settings (Trumbo et al.,
2021) (as opposed to using tests to primarily measure learning). Until our focus on the issue, however,
there had not been any examination of the use of retrieval practice to help students to learn jargon.
Retrieval Practice and Jargon Learning

Our examination involved two experiments (Pan et al., 2019). The first experiment occurred in a
graduate-level clinical neuroanatomy course at Washington University in St. Louis (OT 5782,
“Neuroscience Principles of Performance I”) and involved 85 occupational therapy students; the second
experiment occurred in two sections of an undergraduate-level physiology course at the University of
California, San Diego (UCSD) (BIPN 100, “Human Physiology I”) and involved 277 students. Each week,
both courses covered a different topic via course readings and lectures, and within each topic there
were numerous jargon terms (e.g., “anterior lateral,” which refers to an anatomical location, and
“aquaporin,” which refers to proteins that facilitate the transport of water between cells). That volume
of jargon is typical for courses in the biological sciences, which is among the most jargon-filled of all
scientific disciplines (Groves, 1995; Yager, 1983).
Both experiments featured the same basic design. During the course, we had students take practice
quizzes on jargon terms, which we called jargon quizzing, during some weeks and not during other
weeks. To ensure that any effects of such quizzing could not be attributed to the choice of any particular
topic, we varied the topics that were quizzed for each student. We then measured the effectiveness of
quizzing on course exams occurring between one to twelve weeks later. These exams assessed
knowledge of jargon terms and concepts from topics that had been quizzed and not quizzed. If effective,
we expected that jargon quizzing would improve exam performance and especially on questions
involving terms that had been quizzed.
Jargon quizzing occurred in two forms: online quizzing, which we implemented in the first experiment,
and in-class quizzing with clickers (also called clicker quizzing), which we implemented in the second
experiment. Both forms of quizzing were intended to be simple to set up, easy to use, and palatable to
students. Both forms were also intended to integrate well within existing course structures, interfere
minimally with other learning activities, and use commercially available instructional tools.
In the first experiment (Pan et al., 2019), online quizzing occurred via Blackboard©, a web-based learning
management system. In consultation with the course instructor, we created a quiz covering 25-27 jargon
terms for each to-be-quizzed neuroanatomy topic. Each quiz featured a series of fill-in-the-blank and
short answer questions. The correct answers were all jargon terms (e.g., “posterior parietal cortex” and
“inferior temporal cortex” were the answers to the respective blanks in the fill-in-the-blank statement,
“The dorsal pathway ends in the _______ (part of the cortex), whereas the ventral pathway ends in the
474
_____ (part of the cortex)”). Each online quiz included 16 fill-in-the-blank statements, some of which
included multiple blanks (25-27 jargon terms, total for each quiz).
When a student in the neuroanatomy course engaged in online quizzing for a given topic, they
completed the quiz for that topic three times: once prior to the first lecture on the topic, a second time
prior to the second lecture, and finally after both lectures. Quizzing on materials more than once can
foster more robust learning (Butler, 2010; McDaniel et al., 2011). Each quiz was self-paced and took
about ten minutes. To enable the review of any terms that might have been missed, the correct answers
were displayed immediately after all responses had been submitted (which constituted a form of
feedback). Each student completed the quizzes on their own, outside of class, in exchange for course
credit. In keeping with our emphasis on using quizzes for learning rather than assessment, credit was
awarded for quiz completion rather than performance.
In the second experiment (Pan et al., 2019), in-class quizzing with clickers occurred via the iClicker©
system from Macmillan Learning. iClicker enables students to transmit answers to multiple-choice
questions using handheld remotes. For each to-be-quizzed physiology topic, the course instructor
created a multiple-choice quiz covering ten jargon terms from that topic. Half of the questions involved
selecting the correct definition for a given term (e.g., “An end-plate potential is…?”, for which the
answer was, “a graded potential that occurs in a skeletal muscle cell”), and the other half involved the
reverse arrangement (i.e., selecting the correct term when given its definition). These questions
facilitated practice in recalling terms and their definitions, which can be more beneficial than recalling
either alone (Pan & Rickard, 2017). Each quiz included both types of questions per jargon term (with 10
jargon terms, 20 questions total for each quiz).
Students in the physiology course were apprised of the date of each clicker quiz via the course syllabus.
Each weekend before a quiz, they were also given a list of the terms to be quizzed, advised to pay close
attention to those terms during the course readings, and told to bring their clicker remotes to class. The
quiz occurred during the first ten minutes of the first lecture period for a given to-be-quizzed topic
(which always occurred at the beginning of the week). The instructor projected each question
individually on a screen at the front of the class, gave students a few moments to respond, and then
showed the correct answer before moving on to the next question. To encourage participation, course
credit was allotted for each quiz, and as in the neuroanatomy course, credit was not affected by
performance.
In both experiments, the implementation of jargon quizzing occurred quite smoothly. Quiz preparation
took no more than a few hours, and when the course instructors were involved, there was little impact
on other preparatory work. No course materials were changed. In the case of the second experiment,
the lecture period was truncated by ten minutes whenever there was a clicker quiz, but the instructor
was still able to cover the same content. Students also accepted the quizzing procedure with few
complaints. To encourage student “buy-in,” the quizzes were billed as a way to help them learn
important information, and their low-stakes nature helped to allay concerns. In addition, some students
had prior experience with taking practice quizzes (e.g., low-stakes quizzing is not uncommon in
undergraduate biology courses at UCSD), although those quizzes did not specifically target jargon.
The course exams assessed not just the effectiveness of jargon quizzing for students’ ability to recall
jargon terms, but also the extent of any benefits of quizzing across the course materials more generally.
Each exam included two types of questions: definition-focused questions, which assessed knowledge of
jargon terms, and conceptual questions, which involved the application of theories, processes, or other
information. The definition-focused questions addressed potential benefits of quizzing for its intended
goal, learning jargon, whereas the conceptual questions addressed any effects of quizzing on other
related course materials, including content that did not necessarily involve jargon.
475
The exams revealed that quizzing was indeed effective for learning jargon. As can be seen in Table 1, in
the first experiment, students scored an average of 10% better on definition-focused questions targeting
jargon terms that had been quizzed, versus questions on terms that had not been quizzed. In the second
experiment, quizzing improved performance on definition-focused questions by nearly the same
average amount, 11%. If translated to letter grades, these improvements were equivalent to the
difference between an A and a B. The effects of quizzing were all the more impressive when accounting
for the elapsed time between quizzing and the exams, which was up to twelve weeks in the first
experiment and three weeks in the second experiment.
Table 1.
Accuracy on definition-focused exam questions that assessed knowledge of jargon terms
Experiment 1: Neuroanatomy Experiment 2: Physiology
Mean Standard error Mean Standard error
Quizzed 53% 2% 87% 1%
Not Quizzed 43% 2% 76% 1%
Quizzing also produced other learning benefits (see Table 2). In the first experiment, on conceptual
questions that required jargon terms in the correct answer, students scored an average of 6% higher
when the required terms had previously been quizzed (e.g., “Henry recognizes a hammer sitting on the
table, but is unable to reach for it. He doesn’t have general motor impairment. Henry likely has
impairment in what part of the cortex?” The answer is the “posterior parietal cortex”). On conceptual
questions that did not require the use of jargon, there was no significant impact of quizzing, positive or
negative, in either experiment. (Note: we included conceptual questions that required jargon in the first
experiment and questions that did not require jargon in both experiments.) Thus, there was evidence
that quizzing improved transfer of learning to application questions, but only when there was an explicit
connection between those questions and previously quizzed content.
Table 2.
Accuracy on conceptual exam questions that required or did not require jargon terms
Experiment 1: Neuroanatomy Experiment 2: Physiology
Jargon terms required Jargon terms not required Jargon terms not required
Mean Standard error Mean Standard error Mean Standard error
Quizzed 38% 2% 46% 2% 44% 1%
Not Quizzed 32% 2% 44% 2% 46% 1%
To take stock, across two experiments, we observed consistent and long-lasting benefits from quizzing
on jargon terms. These benefits occurred across a considerable range of circumstances, including in two
separate courses at two universities, with different instructors and students, with neuroanatomy or
physiology jargon, and with online or clicker quizzes. Thus, our findings do not appear to have been due
to a fortuitously-chosen set of conditions. Rather, it appears likely that quizzing can improve the learning
of scientific and technical jargon across a variety of situations, including in such STEM domains as
chemistry, computer programming, physics, and beyond.
476
It is important to emphasize that jargon quizzing was effective in actual courses and not in highly-
controlled laboratory environments. Some pedagogical techniques can appear to be useful in the
laboratory, only to fall short when implemented in real-world settings where other learning activities are
also taking place. In our experiments, students were regularly engaged in independent study activities,
including cramming for exams (given that they were earning a grade for their overall performance).
Nevertheless, quizzing was able to boost learning over and above such activities.
Practical Recommendations for Instructors

More generally, our examination of jargon quizzing is part of a growing body of research showing the
benefits of low-stakes quizzing in the classroom (Trumbo et al., 2021). Based on this body of research as
well as on research in the laboratory, we offer several practical recommendations. First, instructors
should consider using practice quizzes to help students learn jargon. Such quizzes might occur online or
in-class via clickers (for further discussion of different methods of practice quizzing, see McDaniel &
Little, 2019; Yang et al., 2021). In all cases, quizzing should require students to engage in genuine,
effortful attempts to retrieve information from memory. Second, when used to promote learning,
quizzes should typically be low-stakes. There should be no punitive consequences for incorrect answers
and students should be allowed to practice retrieval without fear of failure (Hinze & Rapp, 2014; Pan et
al., 2020). Third, there should be feedback after quizzing that includes the correct answers. Such
feedback can serve as an additional learning opportunity (Pashler et al., 2005). Finally, to bolster the
effectiveness of quizzing, it can be helpful to engage in repeated quizzing that involves recalling terms
and their corresponding definitions. Quizzing on information more than once generates greater memory
improvements (Pyc & Rawson, 2009; Roediger & Butler, 2011), and quizzing information in more than
one way can also aid learning (Pan & Agarwal, 2018).
Although we did not fully test all of these recommendations, we implemented each recommendation
across one or both experiments (e.g., we included feedback in both experiments and multiple rounds of
testing in the first experiment) and observed positive results. Thus, when it comes to learning jargon,
repeated low-stakes quizzing with feedback appears to be a winning combination. However, it is
possible that alternative implementations of jargon quizzing may ultimately prove to be effective as
well. Additionally, although quizzing typically yields better learning than non-quizzing activities such as
studying, rereading, and concept mapping (e.g., Carrier & Pashler, 1992; Karpicke & Blunt, 2011), we did
not directly compare jargon quizzing against such activities. Hence, our research does not directly
address the possibility of other techniques that may be useful for learning jargon but have yet to be
identified.
In terms of expected results, jargon quizzing can be expected to generate improved knowledge of and
better memory for jargon terms and their definitions. The extent of such improvements may vary
depending on the difficulty level of the jargon terms being learned, as well as how much, how often, and
how recently students are asked to retrieve. Jargon quizzing may also not necessarily affect the learning
of other course content, but it can improve the ability to use jargon terms elsewhere, such as to answer
application questions.
Although our examination of jargon quizzing is not the final word on how to learn jargon effectively, our
findings provide compelling evidence for the use of a learning science-backed technique, retrieval
practice, to help students better acquire the jargon needed to do well in jargon-heavy disciplines
(courses). Both online and clicker quizzing can be implemented In a variety of courses, without extensive
preparation, and with a strong likelihood of positive results. Going forward, anyone seeking to improve
the learning of jargon terms and other specialized vocabulary need not look too far—a solution is now in
sight.
477
Author Note
Thanks to Erin R. Foster, Lisa Tabor Connor, and Timothy C. Rickard for their contributions to the original
study. Completion of the first experiment was supported by a James S. McDonnell Foundation
Collaborative Activity Award. Correspondence concerning this chapter should be addressed to Steven C.
Pan: [email protected]
References
Arons, A. (1973). Toward wider public understanding of science. American Journal of Physics, 41(6), 769-
782. https://doi.org/10.1119/1.1987384
Bretzing, B. H., & Kulhavy, R. W. (1979). Notetaking and depth of processing. Contemporary Educational
Psychology, 4(2), 145-153. https://doi.org/10.1016/0361-476X(79)90069-9
Brown, B. A., & Ryoo, K. (2008). Teaching science as a language: A “content‐first” approach to science
teaching. Journal of Research in Science Teaching, 45(5), 529–553.
studying. Journal of Experimental Psychology: Learning, Memory, and Cognition, 36(5), 1118–
Callender, A. A., & McDaniel, M. A. (2009). The limited benefits of rereading educational texts.
Contemporary Educational Psychology, 34(1), 30–41.
Carrier, M., & Pashler, H. (1992). The influence of retrieval on retention. Memory & Cognition, 20(6),
633-642. https://doi.org/10.3758/BF03202713
Carpenter, S. K., Pashler, H., & Cepeda, N. J. (2009). Using tests to enhance 8th grade students' retention
of US history facts. Applied Cognitive Psychology: The Official Journal of the Society for Applied
Research in Memory and Cognition, 23(6), 760-771. https://doi.org/10.1002/acp.1507
Groves, F. H. (1995). Science vocabulary load of selected secondary science textbooks. School Science
and Mathematics, 95(5), 231–235. https://doi.org/10.1111/j.1949-8594.1995.tb15772.x
Hinze, S. R., & Rapp, D. N. (2014). Retrieval (sometimes) enhances learning: Performance pressure
reduces the benefits of retrieval practice. Applied Cognitive Psychology, 28(4), 597-606.
McDaniel, M. A., Agarwal, P. K., Huelser, B. J., McDermott, K. B., & Roediger, H. L. III. (2011). Test-
enhanced learning in a middle school science classroom: The effects of quiz frequency and
placement. Journal of Educational Psychology, 103(2), 399–414.
McDaniel, M. A., & Little, J. L. (2019). Multiple-choice and short-answer quizzing on equal footing in the
classroom: Potential indirect effects of testing. In J. Dunlosky & K. A. Rawson (Eds.), The
Cambridge handbook of cognition and education (p. 480–499). Cambridge University Press.
478
McDaniel, M. A., Roediger, H. L., & McDermott, K. B. (2007). Generalizing test-enhanced learning from
the laboratory to the classroom. Psychonomic Bulletin & Review, 14(2), 200-206.
McDonnell, L., Barker, M. K., & Wieman, C. (2016). Concepts first, jargon second improves student
articulation of understanding. Biochemistry and Molecular Biology Education, 44(1), 12–19.
https://doi.org/10.1002/bmb.20922
Pan, S. C., & Agarwal, P. K. (2018). Retrieval practice and transfer of learning: Fostering students’
application of knowledge. RetrievalPractice.org.
http://pdf.retrievalpractice.org/TransferGuide.pdf
Pan, S. C., Cooke, J., Little, J. L., McDaniel, M. A., Foster, E. R., Connor, L. T., & Rickard, T. C. (2019).
Online and clicker quizzing on jargon terms enhances definition-focused but not conceptually
focused biology exam performance. CBE—Life Sciences Education, 18(4), ar54.
Pan, S. C., Hutter, S. A., D'Andrea, D., Unwalla, D., & Rickard, T. C. (2018). In search of transfer following
cued recall practice: The case of process‐based biology concepts. Applied Cognitive Psychology,
33(4), 629-645. https://doi.org/10.1002/acp.3506
Pan, S. C., & Rickard, T. C. (2017). Does retrieval practice enhance learning and transfer relative to
restudy for term-definition facts? Journal of Experimental Psychology: Applied, 23(3), 278–292.
Pan, S. C., Sana, F., Samani, J., Cooke, J., & Kim, J. A. (2020). Learning from errors: students’ and
instructors’ practices, attitudes, and beliefs. Memory, 28(9), 1105-1122.
Pashler, H., Cepeda, N. J., Wixted, J. T., & Rohrer, D. (2005). When does feedback facilitate learning of
words? Journal of Experimental Psychology: Learning, Memory, and Cognition, 31(1), 3–8.
Roediger III, H. L., & Butler, A. C. (2011). The critical role of retrieval practice in long-term retention.
Trends in Cognitive Sciences, 15(1), 20–27. https://doi.org/10.1016/j.tics.2010.09.003
Roediger III, H. L., & Karpicke, J. D. (2006). The power of testing memory: Basic research and implications
for educational practice. Perspectives on Psychological Science, 1(3), 181–210.
https://doi.org/10.1111%2Fj.1745-6916.2006.00012.x
Trumbo, M. C., McDaniel, M. A., Hodge, G. K., Jones, A. P., Matzen, L. E., Kittinger, L. I., Kittinger, R. S., &
Clark, V. P. (2021). Is the testing effect ready to be put to work? Evidence from the laboratory to
the classroom. Translational Issues in Psychological Science 7(3), 332-355.
https://doi.org/10.1037/tps0000292
Wandersee, J. H. (1988). The terminology problem in biology education: A reconnaissance. The
American Biology Teacher, 50(2), 97–100. https://doi.org/10.2307/4448654
479
Yager, R. E. (1983). The importance of terminology in teaching K‐12 science. Journal of Research in
Science Teaching, 20(6), 577–588. https://doi.org/10.1002/tea.3660200610
480
In Their Own Words The Value of Quizzing Students to Support
Transfer of Learning
to Apply the Science of Learning in Lou Ann Griswold
Editors
Lauren L. Kordonowy
Victor A. Benassi
Students in most education programs must acquire knowledge to prepare them for their career but also
must apply that knowledge to address real-life situations. This is especially true for students in
professional programs. Therein lies the challenge for me as an educator: how do I know that the students
are acquiring the basic knowledge and that they are able to apply that knowledge in a single course?
I teach in an occupational therapy program and often taught a graduate course to prepare students for
practice with young children that included evaluation and intervention used in occupational therapy
practice. The course was fast paced: students needed to learn new terms and concepts and then apply
them to case examples. However, I found that students were not always ready for application, or transfer,
of information. I knew I needed to help them get to that point.
Even as an experienced faculty member, I knew that I could learn more about effective teaching
strategies. I wanted to support my students’ learning and to guide them through application of content in
a supportive class environment. When students experienced challenge with application, as measured by
exam questions, I wondered if my exam questions were at too high of a level or perhaps the learning
activities did not prepare students for my exams. I questioned whether students had acquired the
information I had expected so that they would be ready to apply it in an appropriate context. Because I
needed to ensure that the students were prepared to work as occupational therapists with children, I
wanted to improve my skill in writing exam questions to indicate students’ ability to apply course content.
My needs as an educator led me to reach out to my university’s Center for Excellence and Innovation in
Teaching and Learning (CEITL) as a resource. Working with the CEITL staff (Catherine Overson and Victor
Benassi) led to a research study for my course the following semester.
I realized that I was asking students to quickly move through Bloom’s taxonomy of educational objectives
(1956, as cited in Anderson and Krathwohl, 2001). In my course, students needed to understand and
remember terms and concepts, apply these concepts to clinical examples, and analyze and evaluate
nuances to select from appropriate options. In a previous semester in which the course had been offered,
students watched pre-recorded lectures online and then applied the information to case examples during
class time, where I provided guidance and support. Previous use of pre-recorded lectures had been
somewhat successful, but only if students viewed them and treated them as a class lecture. (I had one
student tell me she watched my lectures while she made dinner!)
My CEITL colleagues introduced me to the idea of quizzing students after they had been introduced to
new information to facilitate their learning. After reading more about the benefits of quizzing, I was eager
to try this approach. Quizzing has been shown to enhance learning, retention, and transfer of information
to apply it to new situations (Dunlosky et al., 2013). The benefits of quizzing new material, known as the
testing effect, have been demonstrated by multiple researchers (Carpenter & Pashler, 2007; Dunlosky et
al., 2013; Karpicke, 2012; McDaniel et al., 2012; Roediger & Karpicke, 2006; Shobe, 2022; Yang & Luo, this
481
volume). Research has demonstrated the efficacy of quizzing in controlled laboratory situations (e.g.,
Roediger & Karpicke, 2006) and in college courses (e.g., Benassi et al., 2014; McDaniel, et al., 2012). Using
quizzing made sense to me because it reinforced students’ understanding of the terms and concepts they
were learning.
Despite research supporting the testing effect, Karpicke et al. (2009) found that students did not typically
use quizzing on their own as a way to enhance their learning. Rather, to study, students more frequently
re-read material, highlighted what they thought was important, or copied their notes. In other words, they
were revisiting the material but not quizzing themselves on it. Review of Karpicke et al.’s study led me to
realize that faculty are in a position to incorporate quizzing into their courses and individual lectures as
students are learning new material. Because most previous research on the testing effect had been done
with students in controlled situations, the current study would be unique, with quizzing embedded into
online lectures in an actual course. The online lecture platform made it easier to allow students to go at
their pace while prompting them to be quizzed on the material. The approach supported our research
methods. Furthermore, I wanted to assess how well the testing effect would support students’ transfer of
their knowledge to apply it to case examples. Transfer of information was important for my course and
added a different outcome than typically examined in the literature. Our research question was, “Would
students better retain and transfer information when they were asked to actively answer questions as
they are learning new content in an online course module?”
Study Context (Griswold et al., 2017)

The course had 61 graduate students, all of whom identified as female. Two students opted out of the
study at the end of the semester and data from three students were eliminated because they did not
complete all the study requirements, resulting in 56 study participants.
The occupational therapy course was divided into three units, one on evaluation and two on
intervention to address different types of pediatric conditions. The course included twice weekly lecture
classes for one-and-a-half hours and one three-hour lab for hands-on activities that complemented the
lecture material. I wanted to use the lecture class time for application of material with my support. To
prepare students for class time, I recorded lectures for students to watch online before class to replace
many of the in-class lectures. Each unit consisted of at least three pre-recorded lectures. Each lecture
had learning objectives that I shared with students so that they knew what I anticipated they would
learn from the lecture. The objectives guided my recorded lecture content and subsequently the in-class
application.
Research Study Methods

In the online lectures, I embedded what I called “Learning Moments.” Twice during the lecture and at the
end of the lecture, I asked the students to complete a Learning Moment. The Learning Moments included
one of three conditions: a quiz question on the content presented in the lecture, a statement that
students read regarding content just presented, or a question about how well students understood the
material just presented in the lecture (as a control condition, what we called “not asked.”) The three types
of Learning Moments, respectively, reflected quizzing, studying, and a control condition.
For each Learning Moment, I wrote a question that quizzed students on the lecture material just
presented, reflecting one of the lecture objectives. I submitted my questions to Catherine Overson, who
randomly assigned one of the three conditions to each Learning Moment. All students received the same
condition for these Learning Moments, which allowed us to use a within-subject comparison to examine
the benefits of quizzing as a learning strategy. The design eliminated any concerns of sampling error that
482
could be attributed to differences in students’ ability and to other sources of error. Table 1 illustrates the
three conditions presented to students for three different questions that I had written.
Table 1
Sample Learning Moments for the Three Conditions
Condition Original Quiz Question Written Condition Presented to Students
Question When do we administer assessments of client Question as originally written
factors?
a) At the beginning of the evaluation
process
b) After analyzing performance to
determine the underlying cause of
noted problems
c) At any point in the evaluation process
d) After doing an occupational profile and
learning about the child’s challenges
Study What principles are important to use when “When we position a child to engage
positioning a child for occupational in an occupation, we need to
engagement? promote a neutral pelvis, symmetry
at midline, slightly protracted
a) Neutral pelvis, asymmetrical upper
scapula, and chin tuck.”
extremities, differentiation of shoulders
and hips, and vertical alignment of the
head
b) Neutral pelvis, symmetry at midline,
slightly protracted scapula, and chin
tuck
c) Slight anterior tilt of the pelvis,
symmetry of upper body, and hips at 90
degrees
d) Slight posterior tilt of the pelvis,
asymmetrical upper extremities,
differentiation of shoulders and hips,
and chin tuck
Not Asked Recognizing that the letter “g” is a “g” is based Question not shown to students;
(control) on what type of visual perception? students asked if they understood
the material just presented.
a) Position in space
b) Figure ground
c) Visual memory
d) Visual form constancy
Note. Correct answers are in bold type.
483
As noted, lab activities allowed for hands-on learning to support the learning objectives for the modules.
For example, students would examine different assessment tools, interpret assessment results provided to
them for a sample child, or analyze games that might be used during an occupational therapy
intervention. During the scheduled class time, students worked in small groups of three to four to apply
the content to case examples, giving them practice at transferring their knowledge into practice. Table 2
provides an example of an application done during class.
Table 2
In-class Application of Content
____________________________________________________________________________
Scenario: You are working in an elementary school and Heidi, a second grader, has just been referred to
you.
1. Discuss the evaluation process that you would use with Heidi. What order would you gather
evaluation information?
2. Who would you want to gather information from? What questions might you ask?
______________________________________________________________________________
The exam questions required students to transfer the information as they applied the content to case
examples. To evaluate the students’ ability to do so and to support the research, I created an exam
question for each of the Learning Moment questions that I had written. The exam question provided a
context in which to transfer and apply knowledge. Table 3 provides an example of a Learning Moment quiz
question and a corresponding exam question reflecting application and analysis of the provided potential
answers.
Table 3
Learning Moment Quiz Question and Corresponding Exam Question
Learning Moment Quiz Question Exam Question

1. Activities that provide good proprioceptive input Colleen is an 8-year-old, with a learning disability,
include what type of actions? who you are seeing for occupational therapy in a
private clinic. Which of the following activities
a) Changing speed and direction of movement
provide good proprioceptive input to Colleen to
b) Deep pressure through the hands or feet support sensory integration function?
c) Discrimination of different textures a) Being pushed in a tire swing, changing
d) Heavy work such as pushing or pulling speed and direction of movement
b) Climbing on the jungle gym and crossing
the monkey bars
c) Playing hand clapping games (“Miss Lucy
had a baby….”)
d) Finding small dolls and accessories
pressed into play dough
Note. Correct answers are in bold type.
484
Students’ performance on the in-class exam questions provided the dependent measure for the study.
After the course had ended, my CEITL colleagues unveiled the condition for each Learning Moment. We
compared the mean scores on the exam questions for each of the learning conditions using paired sample
t-tests.
Study Results
Based on a repeated measures analysis of variance, we found an overall significant difference among the
mean exam scores for the three conditions, Wilks’s Lambda F (1, 54) 3.26, p < .05. The mean scores are
shown in Figure 1. An a priori planned contrast showed that the mean scores for the Not Asked and Study
conditions were not significantly different (p = .67), so we combined the results from these conditions to
form a Not Asked/Study condition. Another a priori contrast showed that students in the Question
condition scored significantly higher than those in the Not Asked/Study Condition, t (55) = 2.54, p < .007
one sided, d = .34. These results provide evidence that when quiz questions were embedded into the
lectures as a Learning Moment, students performed better on the exam questions than when they were
asked to study the content or merely asked if they understood the content (neither study nor quizzed). Of
note is that students did not perform better when they were given a study statement than when they did
not revisit the content in any way (Griswold et al, 2017).
Figure 1
Mean Percent of Correct Answers on Exam by Condition
90
Mean Percent Correct on Exam
85
80
75
70
65
60
Not Asked Study Question
Question Condition
Note. Error bars represent 95% confidence intervals

The difference on the exam questions, based on our analysis of the three conditions, was statistically
significant and also indicated a meaningful improvement in students’ overall exam score. The average
score was almost 82% for questions for which students had received quiz questions, versus the average
score of 77% for questions related to study statements or the not asked condition. The difference
corresponded to earning a B versus a C based on the grading scale for this course. Certainly, my students
would prefer to earn a grade of a B!
485
The results from this study support previous research on the testing effect and expand its application to
recorded lectures. We assessed the effectiveness of quizzing when the quizzed information is assessed for
transfer to application questions. My work on this project has convinced me of the value of quizzing
students to support their learning. I tell students why I frequently quiz them. Since completing this study, I
have embedded quiz questions in face-to-face lectures as well as recorded lectures. I usually make the
quizzes low stakes, making quiz scores count 5% or less toward students’ course grade. Sometimes
students receive the quiz points by merely taking the quiz, which acknowledges their active participation
in the course. Although I have not conducted another research study to examine the effects of quizzing on
students’ ability to retain and transfer information for application, students’ course grades have gone up,
indicating their preparation for their next professional step.
Participating in the research study answered the questions I had when I sought assistance from my
university’s CEITL Office. I have become much better at developing questions to ask during lectures and
exams. More importantly, I became more aware of the value of having clear learning objectives for each
lecture. Now, I share the lecture objectives with students so they understand what they are to learn
during a specific lecture. The objectives also help me stay focused and to not go off on a tangent. The
objectives allow me to prepare quiz questions to ask as I lecture. Furthermore, I find that I write exam
questions more easily, based on the previously written quiz questions. The interconnected process to
develop lectures based on objectives, followed by quizzing, leading to applied exam questions, provided a
critically cohesive framework that I believe supported the study results. The same process has led to
student success as I have continued to use it in other courses. Although this research study was conducted
in a graduate level course to prepare students for a professional degree in health care (occupational
therapy), the results suggest the value of the testing effect to support transfer of knowledge for other
types of courses when the instructor connects quizzing and exams to clearly identified objectives.
Author Note
This research was supported, in part, by a grant awarded to the University of New Hampshire from the
Davis Educational Foundation (Yarmouth, ME), which was established by Stanton and Elisabeth Davis
after Mr. Davis's retirement as chairman of Shaw’s Supermarkets, Inc.
I thank Catherine Overson and Victor Benassi, my co-authors on the target article for this chapter, and I
extend my appreciation for their contributions and support during this project.
Correspondence concerning should be addressed to Lou Ann Griswold: [email protected]
References
Anderson, L. E., & Krathwohl, D. (Eds.). (2001). A taxonomy for learning, teaching and assessing.
Longman.
Benassi, V. A., Tappin, E. M., Overson, C. E., Lee, M. J., O’Brien, E. J., Prudhomme White, B., & Hakala, C.
M. (2014). Applying the science of learning: The Cognition Toolbox. In V. A. Benassi, C. E.
science into the curriculum (pp. 194–205). Society for the Teaching of Psychology.
486
Carpenter, S. K., & Pashler, H. (2007). Testing beyond words: Using tests to enhance visuospatial map
learning. Psychonomic Bulletin & Review, 14(3), 474-478. https://doi.org/10.3758/BF03194092
psychology. Psychological Science in The Public Interest, 14(1), 4-58.
Griswold, L. A., Overson, C. E., & Benassi, V. A. (2017). Embedding questions during online lecture
capture to promote learning and transfer of knowledge. American Journal of Occupational
Therapy, 71(3), 1-7. https://doi.org/10.5014/ajot.2017.023374
Karpicke, J. D. (2012). Retrieval-based learning: Active retrieval promotes meaningful learning. Current
Directions In Psychological Science, 21(3), 157-163. https://doi.org/10.1177/0963721412443552
Karpicke, J. D., Butler, A. C., & Roediger, H. I. (2009). Metacognitive strategies in student learning: Do
students practice retrieval when they study on their own? Memory, 17(4), 471-479.
McDaniel, M. A., Wildman, K. M., & Anderson, J. L. (2012). Using quizzes to enhance summative-
assessment performance in a web-based class: An experimental study. Journal of Applied
Research in Memory and Cognition, 1(1), 18-26. https://doi.org/10.1016/j.jarmac.2011.10.001
Roediger, H. L., III, & Karpicke, J. D. (2006). The power of testing memory: Basic research and
implications for educational practice. Perspectives on Psychological Science, 1, 181–210.
Shobe, E. (2022). Achieving testing effects in an authentic college classroom. Teaching of Psychology,
487
In Their Own Words Implementing Exam Wrappers

John E. Edlund
You to Know About Why and How Rochester Institute of Technology
Editors
Lauren L. Kordonowy
Victor A. Benassi
Background
One of the challenging issues that instructors encounter is helping students prepare for exams. There
are numerous reasons why students end up underprepared for exams, such as not understanding
textbook materials (e.g., Schnotz & Wagner, 2018), issues with the generation of quality notes (e.g.,
Heijne-Penninga et al., 2015), cramming for exams (e.g., Rawson et al., 2013), poor processing of
information by the students themselves, which may result from lack of relevant background (e.g.,
Smiderle & Weigel Green, 2011), test anxiety (e.g., Nelson et al., 2015), motivational factors (e.g., Bahri
& Corebima, 2015), and more.
Another reason why students end up underprepared for exams is that they receive very little (or no)
training on how to actually study for an exam (see Fleming, 2002 for a notable exception). Although I
would argue that instruction and practice in how to take a test is invaluable on its own (and would be
helpful across numerous courses and disciplines), in the case of taking exams in a course, experience in
and of itself can be a valuable teacher. For instance, I have consistently found across all my courses that
my students typically perform the poorest on their first exam. There are a number of reasons for this
fact, ranging from students not knowing the idiosyncrasies of the professor’s approach to testing to
students not knowing what material should be studied for a particular exam. I believe that students can
learn a great deal from their first test in a course --they can learn to think about their own test-taking
abilities and how to adapt their studying to improve their exam performance. In my courses, I have
found that a brief exercise called an exam wrapper can facilitate improvements in students’ exam
performance with minimal class time being taken away from covering other material.
Exam wrappers have been demonstrated to be successful in a number of different disciplines, ranging
from computer science (e.g., Craig et al., 2016), engineering (e.g., Chew et al., 2016), food science and
nutrition (Gezer-Templeton et al., 2017), to physics (Lovett, 2013). In my study (Edlund, 2020), I
demonstrated that exam wrappers can be successful in a psychology course. Before describing my
implementation of an exam wrapper, I will briefly describe a generalized exam wrapper activity.
In the simplest form, when a teacher assigns an exam wrapper activity, students reflect on their
performance on the exam after the teacher returns a graded exam to students. This reflection is
metacognition. In its simplest form, metacognition is thinking about one’s thinking (Falvell, 1979).
Training in metacognition has been shown to improve scores in classes (Tanner, 2012). Importantly,
metacognition involves recognizing one’s own weaknesses in knowledge and addressing these
weaknesses (the opposite of the Kruger-Dunning effect: Dunning et al., 2003). As such, to engage in
metacognition one must first recognize that there are deficiencies in one’s knowledge base and then
engage in reflection about where the deficiencies originated and what could be done to remedy them.
488
For exam wrappers, the specific forms of reflection vary based on their implementation. In my study
(Edlund, 2020), as I returned their exams with the incorrect items marked (thereby, identifying gaps in
knowledge), I proceeded to ask the students five questions about their performance (see Appendix for
the exam wrapper as implemented). Other studies have had somewhat different questions, but the goal
is always to have students engage in metacognitive thinking about their exam preparation and
performance after identifying gaps in their knowledge base.
Study 1
In my first implementation of the exam wrapper intervention (Edlund, 2020), I compared two sections of
my social psychology course taught in different semesters. At my institution, this is a 200-level course
with a single prerequisite (introduction to psychology). As such, we have a number of psychology majors
in the course, but we also have a number of psychology minors and immersions/concentrations, as well
as students who just want to learn more about the topic. In this course, I typically offer three non-
cumulative exams (although certain core course concepts such as reliability and validity appear in all
exams, as they are part of the field more generally). The exams are a mix of multiple-choice questions
(25 in total) and five short-essay questions. I tell students that their responses should be 4-5 sentences
per question. After grading the first exam, I returned the exam to students in class and reviewed the
answers. After discussing the exam, I gave the exam wrapper to students in my experimental section
Students in the comparison group did not receive anything. Students were asked to immediately fill out
the exam wrapper and turn it in. Exactly one week before the second exam, I returned the exam
wrappers to the students (ideally allowing the students to remind themselves what they thought
immediately after the last exam).
In this study, I found that both sections showed increases in exam performance from the first exam to
the second, but the increase in exam performance in the exam wrapper section (Mgain = 9.37) was larger
than the mean gain in the comparison group (Mgain = 2.97), Finteraction (1, 82) = 6.15, p = .015, ηpartial2 = 07.
Although this study was an excellent proof of concept, it was not definitive evidence that the exam
wrapper had a beneficial impact on exam performance. After all, I could have been a more effective
teacher after I implemented the exam wrapper, the students may have tried harder knowing that I was
implementing a new teaching technique, I could have scored the subsequent exams less stringently, etc.
As such, I conducted a second study that addressed these potential issues.
Study 2
In a second study (Edlund, 2020), to address a number of potential confounds, I implemented things a
bit differently. The first difference was that I implemented the study in a single section of my course. By
taking this step, I reduced the possibility that my actual teaching effectiveness would be different across
semesters and course sections. I also knew that if some of the students completed the exam wrapper
and other students did not, several potential problems could occur (including the ethics of knowing that
some students were being denied a treatment designed to help them). Therefore, when I handed out
the exam wrapper to the experimental group, I handed out a sheet that discussed test taking strategies
to the control group. That way, all students received an intervention that could potentially improve their
performance. Finally, per my usual grading practice, I did not know my students’ identities while grading
free responses (the students only list their name on the front sheet which includes only multiple-choice
items). Thus, I was both unaware of students’ names and exam wrapper conditions during exam
grading. As in the first study, I handed out the sheets (exam wrapper or test-taking strategies) to the
students randomly (based on last name). After discussing the exam, students read and completed the
sheets and handed them back to me. The students in the control group put their name at the top of the
test taking strategy sheet and signed the bottom. The exam wrappers or test-taking strategies were
489
returned to the students one week before the second exam. Finally, given the success in my first study
and the early success in study two, I gave all students an exam wrapper after the second exam (which
was then handed back one week before the third exam).
In this study, the two groups did not differ significantly on exam one (both groups scored in the low 70s).
On exam two, both groups had an increase in exam score, although the increase was higher for the
exam wrapper group, Finteraction (1, 55) = 4.87, p = .031, ηpartial2 = 08. On the third exam, the difference
between the groups disappeared as both groups now had the benefit of the exam wrapper, FMaineffectexam3
(1, 54) = 0.55, p > .15. Further, the overall interaction term across all three exams was significant,
Finteraction (2, 108) = 4.97, p = .009, ηpartial2 = 08 (Table 1).
Table 1
Mean Exam Scores Across Three Exams
Exam 1 Exam 2 Exam 3

Control group (test taking strategies handout before exam 2, exam 73.55 82.96 91.48
wrapper before exam 3) (11.04) (8.60) (10.72)
Experimental Group (exam wrappers before exams 2 and 3) 73.56 90.22 90.78
(12.50) (9.20) (11.78)
Note. Standard deviations in parentheses
Conclusions and Future Directions

My take-home message from these two studies is that the exam wrapper is an easy-to-implement and
beneficial intervention. It takes roughly ten minutes of class time when combining the passing out of the
exam wrappers with the exams. In my implementations, I saw quite appreciable improvements in exam
performance. However, there are a number of remaining questions about exam wrappers that still need
to be investigated.
• At what course level will exam wrappers be most successful? In my class (Social Psychology), I do
not, by definition, have first-year students enrolled because of the prerequisite; rather, I have a
number of 2nd, 3rd, and 4th year students taking the class. Reviewing the literature more
generally, there does not seem to be a clear consensus regarding what levels of courses may be
most likely to show a benefit. Some studies, such as Gezer-Templeton et al (2017), looked at
introductory courses (where they showed a benefit of exam wrappers in an introduction to food
science course); whereas, other studies, such as Chew et al. (2016), examined more advanced
courses (they showed a benefit in an advanced statistics course).
• What kinds of students would show the greatest benefit from this sort of an intervention in
terms of individual differences? My courses certainly had a wide variety of students, ranging
from 2nd year to 4th year, but I believe that the sooner this intervention is introduced to the
students the better. As such, students in first-year courses may show the greatest benefit (if
nothing else, from having more years of education knowing about this metacognitive activity).
• Which disciplines may show the greatest improvement? To date, there have been no
investigation into this issue; this remains an interesting and valuable question to be addressed.
• How long-lasting is this intervention? My hope is that it will carry over from one course to
another over the course of an academic career; however, there is no evidence to be evaluated
with respect to this question.
490
• Regarding scalability of this intervention, could this intervention be implemented across first-
year students more generally and then show benefits across the years? Will the students
naturally adopt the strategies associated with the exam wrapper without needing a formal exam
wrapper activity? Obviously, this question would be very ambitious to answer and, in some
ways, would depend on answers to the earlier questions I posed.
References
Bahri, A. & Corebima, A.D. (2015). The contribution of learning motivation and metacognitive skill on
cognitive learning outcome of students within different learning strategies. Journal of Baltic
Science Education, 14(4), 487-500. http://dx.doi.org/10.33225/jbse/15.14.487
Chew, K. J., Chen, H. L., Rieken, B., Turpin, A., & Sheppard, S. (2016). Improving students' learning in
statistics skills: Using homework and exam wrappers to strengthen self-regulated learning. Paper
presented at the American Society for Engineering Education (ASEE) 123rd Annual Conference,
New Orleans, LA.
Chew, K. J., Chen, H. L., Rieken, B., Turpin, A., & Sheppard, S. (2016). Improving students’ learning in
statistics skills: Using homework and exam wrappers to strengthen self-regulated learning.
American Society for Engineering Education, New Orleans, LA.
Craig, M., Horton, D., Zingaro, D., & Heap, D. (2016). Introducing and evaluating exam wrappers is CS2.
Proceedings of the 47th ACM Technical Symposium on Computing Science Education, Memphis,
TN.
Dunning, D., Johnson, K., Ehrlinger, J., & Kruger, J. (2003). Why people fail to recognize their own
incompetence. Current Directions in Psychological Science, 12(3), 83-87.
https://doi.org/10.1111/1467-8721.01235
Edlund, J.E. (2020). Exam wrappers in psychology. Teaching of Psychology, 47(2), 156-161.
Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new area of cognitive–developmental
inquiry. American Psychologist, 34(10), 906-911. https://doi.org/10.1037/0003-066X.34.10.906
Fleming, V. M. (2002). Improving students' exam performance by introducing study strategies and goal
setting. Teaching of Psychology, 29(2), 115-119.
https://doi.org/10.1207/S15328023TOP2902_07
Gezer-Templeton, P. G., Mayhew, E. J., Korte, D. S., & Schmidt, S. J. (2017). Use of exam wrappers to
enhance students’ metacognitive skills in a large introductory food science and human nutrition
course. Journal of Food Science Education, 16(1), 28-36. https://doi.org/10.1111/1541-
4329.12103
Heijne-Penninga, M., Kuks, J.B.M., Hofman, W.H.A, & Cohen-Schotanus, J. (2015). Directing students to
profound open-book test preparation: The relationship between deep learning and open-book
test time. Medical Teacher, 33(1), e16-e21. https://doi.org/10.3109/0142159X.2011.530315
Lovett, M. (2013). Make exams worth more than the grade. In M. Kaplan, N. Silver, D. LaVague-Manty, &
D. Meizlish (Eds.), Using reflection and metacognition to improve student learning: Across the
disciplines, across the academy (pp. 18-52). Stylus.
491
Nelson, J. M., Lindstrom, W., & Foels, P. A. (2015). Test anxiety among college students with specific
reading disability (dyslexia): Nonverbal ability and working memory as predictors. Journal of
Learning Disabilities, 48(4), 422-432. http://dx.doi.org/10.1177/0022219413507604
Rawson, K. A., Dunlosky, J., & Sciartelli, S. M. (2013). The power of successive relearning: Improving
performance on course exams and long-term retention. Educational Psychology Review, 25,
523–548. http://dx.doi.org/10.1007/s10648-013-9240-4
Smiderle, D. & Weigel Green, P.L. (2011). How should students prepare for exams: A knowledge
management approach. International Journal of Knowledge Management, 7(4), 55-69.
https://doi.org/10.4018/jkm.2011100104
Soicher, R. N., & Gurung, R. A. R. (2017). Do exam wrappers increase metacognition and performance? A
single course intervention. Psychology Learning & Teaching, 16(1), 64-73.
http://dx.doi.org/10.1177/1475725716661872
Tanner, Kimberly D. (2012). Promoting student metacognition. CBE—Life Sciences Education, 11(2), 113-
120. 120. https://doi.org/10.1187/cbe.12-03-0033
492
Appendix
Social Psychology Post-Exam Reflection Paper
This activity is designed to give you a chance to reflect on your exam performance and, more
importantly, on the effectiveness of your exam preparation. Please answer the questions sincerely. Your
responses will be collected to inform your instructor regarding your experiences surrounding this exam
and how I can best support your learning. I will hand back your completed sheet in advance of the next
exam to inform and guide your preparation of that exam.
1) Approximately how much time did you spend preparing for this exam? ___________________
2) What percentage of your test-preparation time was spent on each of these activities (make sure
the percentages add up to 100%)
a. Reading the textbook the first time _____
b. Rereading the textbook _____
c. Reading the lecture slides the first time _____
d. Rereading the lecture slides the first time _____
e. Reading your notes from the class _____
f. Other practice (flash cards, etc.) _____
i. Please detail: _______________________________________
g. Other? _____
i. Please detail: _______________________________________
3) Now that you have looked over your graded exam, estimate the percentage of points you lost
due to each of the following (make sure the percentages add up to 100%)
a. Studied wrong topics: _____
b. Trouble with terminology: _____
c. Not clear what professor was asking for: _____
d. Careless mistakes: _____
e. Other: ____
i. Please specify: _______________________________________
4) Based on your responses to the questions above, name at least three things that you plan to do
differently in preparing for the next exam. Please be as specific as possible.
a. ___________________________________________________________________
b. ____________________________________________________________________
c. ____________________________________________________________________
5) What else can I do (as your instructor) to help support your learning and preparation for the
next exam?
493
In Their Own Words Wrapping up Lessons With Closed-Book and
Open-Book Tests
to Apply the Science of Learning in Judith Schweppe
University of Passau
Editors
Lauren L. Kordonowy
Ralf Rummer
Victor A. Benassi
University of Kassel
Imagine you are looking back on a successful lesson in your psychology class last week with active
students whose contributions gave you the impression that the learning goals you had set for that
lesson were largely achieved. Now you want to build on last week’s work and activate your students’
recently acquired knowledge, but your introductory question addressing one of last week’s central
topics is met with silence and blank stares. This frustrating experience – for teachers and students alike
– and the observation that initial comprehension is not sufficient for later access to that knowledge even
just a week later is common in college and university classrooms. As teachers (and researchers), this
shifts our focus from looking for the best ways to help students acquire knowledge in the first place to
finding ways to support them in consolidating that knowledge and enabling them to retrieve it from
memory when needed.
Overview
Among the methods supporting long-term memory consolidation and knowledge retrieval, the most
widely recommended approach is called retrieval practice (also known as testing or quizzing, e.g.,
Dunlosky et al., 2013; Pashler et al., 2007). The idea is that by trying to retrieve information from long-
term memory early in the learning process, learners improve later access to that information. In addition
to this direct effect of retrieval practice, testing is thought to improve long-term retention indirectly
because it enables students to better judge their learning and calibrate restudy opportunities. An easy
way to implement retrieval practice in one’s teaching is to use it for wrapping up lessons. Instead of
summarizing the most important points at the end of a lesson, teachers could set up brief quizzes in
which students answer questions addressing central points covered in that lesson. Another
implementation that requires little change to one’s regular teaching is to yet again replace one’s own
summary but in a review section at the beginning of a lesson with a quiz covering the previous lesson or
other prior knowledge that is relevant for today’s topic (see Retrieval Practice for more implementation
ideas).
Research on testing as a learning technique was first investigated in the early 20th century (e.g., Witasek,
1907). This area of inquiry has surged in the past 15 years with a plethora of studies conducted not only
in the laboratory but also in school and university classrooms. Previous meta-analyses indicating robust
testing effects primarily covered studies conducted in laboratory settings and with rather short
interventions (Pan & Rickard, 2018; Rowland, 2014). A recent meta-analysis on classroom-based
retrieval practice activities also demonstrated that testing substantially improved long-term learning
compared to more passive revision methods. When assessing for the impact of stakes, the authors
found that low- or no-stakes tests are as effective as high-stakes tests (Yang et al., 2021; Yang et al., this
volume). A factor that influenced the size of the testing effect in classroom studies was the length of the
intervention, with larger effects when testing was implemented in more than a single class (Yang et al.,
494
2021). There is thus indeed good reason to implement retrieval practice in one’s teaching and to do so
on a regular basis.
While the evidence compared to passive revision techniques is compelling, questions remain as to how
best to implement retrieval practice. One question is whether practice tests that are open-book, where
students are allowed to consult learning materials while answering questions, are equally effective in
supporting consolidation of relevant memory traces compared to closed-book tests. Open-book tests
could be an attractive alternative for students averse to testing, as they provide a fallback option when
the answers do not come to mind and may thus make for a less frustrating learning experience. Yet,
given the assumption that the act of practicing retrieval from long-term memory is beneficial in itself,
one would expect open-book practice tests to be less helpful in the long run, as in such a setting,
students typically answer fewer questions from memory and thus practice retrieval to a lower degree
(e.g., Waldeyer et al., 2020). Surprisingly, however, previous lab studies comparing open-book and
closed-book tests have not found any differences (Agarwal et al., 2008, 2011; Wenzel et al., 2022). Both
promoted long-term learning better than repeated studying to a similar degree.
Our Study
In the study we describe here (Rummer et al., 2019), we investigated this question further, and we did
so in a classroom setting with longer treatment duration and longer delays than in previous studies.
A colleague of ours, Annett Schwede, taught two parallel introductory seminars in cognitive psychology
(title: “Cognition”) at the University of Erfurt (Germany) for undergraduate first-year students of
psychology. The seminars accompanied the lecture “Introduction to cognitive psychology” (taught by
Ralf Rummer), with which they formed the module “Cognitive psychology.” The module concluded with
a mandatory final exam which consisted of questions referring to matters exclusively covered in the
lecture as well as matters taught in both the lecture and the seminar. All students received the same
questions. Annett assigned seats in either of the two courses randomly and took care to follow the exact
same pre-determined script in both courses. This allowed us to experimentally manipulate the degree of
retrieval practice during wrap-up practice questions by varying the procedure between the two courses.
We conducted the experiment, described below, during eight weeks of the course.
To wrap up the lessons, students in both courses spent the final ten minutes of each of the first seven
lessons answering between two and three short-answer questions individually on a sheet of paper. The
questions covered central topics of that lesson and could be answered with a few sentences. To
manipulate the degree of retrieval practice, students in the control group could consult handouts and
their own notes while answering the questions (open-book tests) and students in the experimental
group could not (closed-book tests). No explicit feedback was provided but students could ask questions
regarding the practice questions in the first five minutes of the following lesson or could restudy the
materials at home. The remainder of each lesson consisted of student presentations (closely supervised
by the instructor to ensure that the input in the two groups was of similar quality and informativeness)
and time for discussion.
At the beginning of the semester, students were told that the two seminars were part of a research
project investigating students’ learning behavior. Participation in the study was voluntary and students
could opt out of having their data used for research at any time. All students agreed to participate (30 in
the open-book group and 29 in the closed-book group).
Our main dependent variable was performance on a surprise quiz in week eight, which comprised the
questions from the practice tests. It took place during a lesson announced as preparation for the module
exam. For this lesson, 27 students from the open-book course and 19 from the closed-book course
495
attended the lesson. In addition, we compared students’ performance in the two courses on the final
module exam administered eight weeks later. As the exam focused on the lecture, only a small
proportion of the questions addressed content that was also covered in the seven experimental lessons
of the seminar and were thus analyzed here (9 out of 30 multiple-choice questions, with additional
short-answer questions not analyzed for the study). 25 students from the open-book group and 26 from
the closed-book group took part in the final module exam.
Table 1 gives an overview of students’ performance on the surprise quiz and on the questions of the
final module exam that addressed content covered in the critical seven lessons of the seminar. Students
who performed the practice tests in the closed-book condition performed significantly better on the
surprise quiz than those in the open-book condition. The findings were similar for the questions of the
module exam on overlapping topics, with better performance for the closed-book group than for the
open-book group. In contrast, the two groups did not differ on the remaining multiple-choice questions
of the exam and also did not differ in total grades.
Table 1
Mean Percentage Correct (And Standard Deviations) On the Surprise Quiz and the Final Module Exam as
a Function of Type of Practice Test (Closed-Book vs. Open-Book)
Closed-Book Tests Open-Book Tests
Surprise Quiz 45.09 (SD = 14.71) 36.73 (SD = 9.30)
Final Module Exam 61.11 (SD = 13.05) 48.00 (SD = 17.19)
(Overlapping Topics)
Both results indicate that answering wrap-up questions without consulting notes or other materials
helped students’ long-term retention more than answering the same questions with use of any
materials. What is particularly encouraging is that this advantage translated to questions other than
those on practice quizzes.
Our findings corroborate previous evidence that practice tests can alleviate the problem of forgetting
between successive lessons. The testing effect has been attributed to direct effects of practicing
retrieval from long-term memory as well as to indirect effects on metacognitive monitoring and
regulation. Both could have been at play here: students in the open-book group probably tried to
answer fewer questions from memory and thus practiced retrieval to a lower degree, though we did not
receive any student reports on this. Looking up the answers, however, is less likely to provide for
students’ insight into their ability to answer similar questions later without external sources, as is the
case in typical closed-book exams. Students who successfully answer the questions with the help of their
materials may be bolstered with unwarranted confidence that they will be able to be successful in a
later exam, and thus spend less time on further study than their peers who tried to answer the
questions without the aid of course materials.
However, our findings are at odds with previous lab studies in which open-book tests were just as
beneficial as closed-book tests (Agarwal et al., 2008, 2011; Wenzel et al., 2022). Several specifics of our
setting may have affected the outcomes. For instance, students in the open-book course may have been
too strongly discouraged to answer the questions from memory. Discarding open-book practice tests
would thus attach too much importance to our study. In particular, findings from a later lab study
(Waldeyer et al., 2020) suggested an intervention that might make open-book tests more effective than
closed-book tests: Their participants answered practice questions by default in a closed-book fashion
but could access the materials on demand, which then disappeared again. Participants in this condition
496
outperformed both a pure open-book and a pure closed-book group. This procedure of automatically
limiting the availability of materials is hard to implement in paper and pencil classroom quizzes.
However, one could try to follow this in principle by encouraging students to answer as many questions
as possible from memory and to only consult their materials for checking their answers or answering
particularly challenging questions. For this option to be effective, it should also be beneficial to tell
students about the power of retrieval practice – something about which our psychology students, who
were in their very first weeks, did not know. Raising awareness of retrieval practice as an effective
technique might also alter students’ attitudes towards the closed-book tests in our study. We did not
record any student perceptions but, anecdotally, Annett told us that student evaluations were clearly
better in the seminar with the open-book tests than in the one with the closed-book tests, which may
have been due to the less aversive implementation of the practice tests. All these speculations indicate
that there are still a lot of open questions on how to best implement practice tests in educational
settings that are worth investigating.
Recommendations
A wrap-up phase at the end is typically recommended for well-structured lessons. We implemented it in
a way to maximize retrieval practice by handing out two to three questions on paper that students
answered individually. Such a low-cost intervention alters more typically wrap-up procedures like
teacher summaries and takes a bit more time in the lesson (ten minutes in our study) and also some
time outside the lesson for the teacher to prepare the questions. Apart from these requirements,
however, closed-book test wrap-ups could be easily included in all types of courses in which the central
outcomes of a lesson can be anticipated so that good questions can be prepared in advance. While we
did not provide direct feedback (but gave students the opportunity to ask questions in the next lesson
or, of course, to check their answers themselves), this is something you might want to add if you think
about implementing something similar in your courses, as feedback is known to increase the benefits of
testing (Rowland, 2014; Yang et al., 2021).
References
Agarwal, P. K., Karpicke, J. D., Kang, S. H. K., Roediger, H. L. III, & McDermott, K. B. (2008). Examining the
testing effect with open- and closed-book tests. Applied Cognitive Psychology, 22(7), 861–876.
Agarwal, P. K., & Roediger, H. L. III. (2011). Expectancy of an open-book test decreases performance on a
delayed closed-book test. Memory, 19(8), 836–852.
synthesis. Psychological Bulletin, 144, 710. https://doi.org/10.1037/bul0000151
Organizing instruction and study to improve student learning. IES practice guide. NCER 2007-
2004. National Center for Education Research. https://files.eric.ed.gov/fulltext/ED498555.pdf
497
Rummer, R., Schweppe, J., & Schwede, A. (2019). Open-book versus closed-book tests in university
classes: A field experiment. Frontiers in Educational Psychology, 10, 463.
Waldeyer, J., Heitmann, S., Moning, J., & Roelle, J. (2020). Can generative learning tasks be optimized by
incorporation of retrieval practice? Journal of Applied Research in Memory and Cognition, 9(3),
355-369. https://doi.org/10.1016/j.jarmac.2020.05.001
Wenzel, K., Schweppe, J. & Rummer, R. (2022). Are open-book tests still as effective as closed-book tests
even after a delay of 2 weeks? Applied Cognitive Psychology, 36(3), 699-707.
Witasek, S. (1907). Über lesen und rezitieren in ihren beziehungen zum gedächtnis. [On reading and
recitation in their relations to memory.]. Zeitschrift für Psychologie, 44, 161–185.
498
In Their Own Words Throw Them in the Deep End? Quizzing With
Factual Versus Application Items
to Apply the Science of Learning in Danney Rasco
Salem State University
Editors
Lauren L. Kordonowy
Victor A. Benassi
Students regularly ask, “How can I do well in your course?” They are surprised when I provide a simple
answer: Take advantage of the quizzes. Quizzes are not a silver bullet, but they improve students’
recollection of course information and, importantly to students, increase performance on exams. Pre-
lecture reading quizzes encourage students to arrive prepared (pre-training), ongoing quizzes promote
regular studying (spacing), and review quizzes help students revisit material from previous topics
(interleaving). Central to the present discussion, all of these types of quizzes require students to retrieve
information to answer items, which improves performance on later exams (testing effect, retrieval
practice). Still, questions remain about how to use quizzes most effectively. In particular, should we use
harder application quizzes or easier factual quizzes to help students do well in the course? That is to say,
should we throw students in the deep end early in the learning process or not?
Students, across a range of grades and subjects, perform better on a final exam when they answer quiz
questions over the material as opposed to re-reading information or engaging in similar re-exposure
activities (Little & McDaniel, 2015; McDaniel et al., 2007; Pan & Rickard, 2018; Roediger et al., 2011;
Shobe, 2022; Yang & Luo, this volume). Researchers have found this outcome – referred to as the testing
effect, test-enhanced learning, and retrieval-based learning – regularly for free-recall, cued-recall, short-
answer, and multiple-choice quizzes (Glover, 1989; Kang et al., 2007; Karpicke, 2017; Little & Bjork,
2015; McDaniel et al., 2007; McDaniel et al., 2012). However, questions remain about the attributes of
an ideal quiz. Some research on the transfer-appropriate-processing (TAP) perspective supports
matching the acquisition (i.e., encoding) activities with the final assessments, and other research on the
levels-of-processing (LOP) perspective supports requiring deeper processing early in the course of
learning (Lockhart, 2002). These perspectives frame our discussion.
The TAP perspective asserts that aligning the level of the items on quizzes and exams so they match will
result in better performance on the follow-up exam (Morris et al., 1977). As an example, different types
of information can be emphasized depending on the format of practice materials (e.g., cued-recall or
free-recall items), and this format increases the likelihood of recalling different types of information
(target, cue) consistent with the type of information emphasized during the practice session (de
Winstanley et al., 1996). Based on the TAP perspective, we would expect better performance on exams
with application items if we used quizzes with application items, and we would expect better
performance on exams with factual items if we used quizzes with factual items. Consequently, from this
perspective, we should only throw students in the deep end if we intend to test them in the deep end,
assuming test performance is our learning outcome of interest.
The LOP perspective asserts that deeper initial processing will result in better recall at a later time (Craik
& Lockhart, 1972; Craik & Tulving, 1975). Bjork (1975) extended this idea and asserted that deeper
retrieval processing on a quiz or similar activity will result in better recall at a later time as well.
Preliminary research supported this assertion and found that difficulty of retrieval was associated with
499
depth of processing, and ease of immediate recall was negatively associated with delayed recall. For
example, using the serial-position framework and noting the relative placement of items in a list, recent
items were initially more likely to be recalled than items in the middle of the list (i.e., the recency effect
occurred). However, these items were less likely to be recalled at a delayed test (Bjork, 1975). Pyc and
Rawson (2009) continued to build on this idea using an approach similar to that used in studies on the
spacing effect, and they found a longer interstimulus interval, which should increase the difficulty of
initially retrieving information, resulted in a higher proportion of items correctly recalled on a final test.
From this result, they posited the retrieval effort hypothesis, which asserts that “given retrieval is
successful, more difficult retrievals are better for memory than less difficult retrievals” (Pyc & Rawson,
2009, p. 438). Consequently, the LOP perspective and subsequent related research suggest deeper items
(e.g., application items) will better prepare students for both factual and application exam items.
As noted earlier, the retrieval effort hypothesis assumes retrieval success. However, there is some
indication that failure to successfully retrieve the information may not be problematic in the classroom
setting (Kornell et al., 2015), and Kornell and Vaughn (2016) note that successful retrieval often is not
required to see benefits from retrieval attempts. Furthermore, in a classroom setting, quizzes and other
forms of dynamic assessment, especially when feedback is provided, influence how students re-study
material between the quiz and the final exam, which may result in indirect benefits from the quiz (i.e.,
test-potentiated learning; Arnold & McDermott, 2013; Hayes et al., 2013; Roediger & Karpicke, 2006;
Soderstrom & Bjork, 2014). Consequently, exam performance may be better following a more difficult
quiz, despite the fact that more errors in retrieval may occur during the quiz (Carpenter & DeLosh, 2006;
Glover, 1989; McDaniel et al., 2013). As an example, McDaniel et al. (2013) evaluated the impact of
term-response questions, which focused on selecting the term coinciding with a provided definition, and
application questions, which focused on selecting the correct term for a provided scenario or situation.
They found term-response questions resulted in more successful retrievals as indicated by higher exam
performance (i.e., they were easier) than application questions. Yet, application quizzes benefitted exam
performance on both types of questions, and term-response quizzes did not benefit exam performance
on application questions. Relevant to the present research, this outcome is consistent with expectations
from the LOP perspective.
How Were Quizzes Formatted in This Study?

The research described in this chapter (see Collins et al., 2018 for further details) explored how the
types of items (factual, application) used on quizzes influenced later exam performance. Based on the
TAP perspective, we would expect better performance on exams with application items if we used
quizzes with application items, and we would expect better performance on exams with factual items if
we used quizzes with factual items. Consequently, if the results support this perspective, we should only
throw students in the deep end if we intend to test them in the deep end. In contrast, the LOP
perspective supports deeper questions, such as application items, in preparing for both factual and
application exam items.
Students enrolled in an upper-level social psychology course (n = 40) were randomly assigned to
complete three multiple-choice online quizzes with either factual items or application items. Factual
items focused on the definition of the term: “Individuals who perceive outcomes as controllable by their
own efforts have a(n) internal locus of control.” In contrast, application items focused on the same term
using a real-life scenario: “Ryan is constantly pushing himself to improve and do better because he
believes that through these actions, he can have some influence over his future. Ryan seems to have
a(n) internal locus of control.”
500
In both conditions, quiz items covered the same 30 key terms (e.g., internal locus of control) from the
three chapters. Item order was randomly determined for each quiz but consistent across the two
conditions, so key terms appeared in the same order for both groups. Following each quiz, students
received a grade and item-level feedback with the correct answers (i.e., key terms). Later in the
semester, students completed an exam with factual and application multiple-choice items. Key terms
were randomly assigned to be factual or application items on the exam, and 30 new items (not
previously quizzed) were created – one for each key term.
What Were the Results?

The results are shown in Figure 1. On the 15 factual exam items, students who completed the factual
quizzes scored 5.6% lower than students who completed the application quizzes, although the
difference was not significant (t (38) = −1.46, p = .15, two-tailed). A similar, although significant,
difference was noted on the 15 application exam items. Students who completed the factual quizzes
scored 11.4% lower on the application items than students who completed the application quizzes (t
(38) = −3.09, p = .004, two-tailed).
Figure 1
Performance on Factual and Application Exam Items by Quiz Item Type
100.00
Percentage Correct on Exam
95.00
90.00
85.00
80.00
75.00
70.00
Factual Application
Exam Item Type
Factual Quiz Application Quiz
Note. Error bars represent 95% confidence intervals around the means.
What Does This Mean for Students in Your Course?

Throw them in the deep end! Although this statement would likely terrify a fledgling swimmer (and
possibly your students), it appears application quizzes result in better understanding on factual and
application exam items – consistent with the LOP perspective. Additionally, assuming the real goal of
education is transfer of learning and real-world application after graduation, testing students with
application items can potentially promote transfer (Pan & Rickard, 2018) and, based on the results
provided here, benefit later application.
Admittedly, there is evidence that supports the use of easier items under certain conditions. For
example, Adesope et al. (2017) performed a meta-analysis, which showed the testing effect was
501
stronger under TAP conditions. This finding supports the idea that later test performance can be
enhanced by aligning the quiz format with the exam format. If we assume that a factual, definition-
based final exam is the final learning objective, factual quizzes may be useful, and there are potentially
courses (e.g., introductory courses) where this level of knowledge is a final learning outcome. However,
if we assume that a higher-order goal of learning is transferring and applying the knowledge in a later
course or career, application quizzes and similarly challenging item types are supported.
Additionally, it may be beneficial to use items of varying degrees of difficulty on quizzes (see Kornell &
Metcalfe, 2006). However, challenging students on quizzes can increase the time spent re-studying the
content (Soderstrom & Bjork, 2014), aid metacognition by improving accuracy of judgments of learning
(Little & McDaniel, 2015), and ultimately, benefit student learning and, yes, exam performance, which
they appreciate.
Author Note
Results described in this chapter were originally reported in the following article: Collins, D. P., Rasco, D.,
& Benassi, V. A. (2018). Test-Enhanced learning: Does deeper processing on quizzes benefit exam
performance? Teaching of Psychology, 45(3), 235–238. https://doi.org/10.1177/0098628318779262
This research was supported, in part, by a grant awarded to the University of New Hampshire from the
Davis Educational Foundation (Yarmouth, ME), which was established by Stanton and Elisabeth Davis
after Mr. Davis's retirement as chairman of Shaw’s Supermarkets, Inc.
Correspondence concerning this chapter should be addressed to Danney Rasco, Assistant Dean, School
of Graduate Studies, Salem State University, 352 Lafayette St., Salem, MA 01970, United States. Email:
[email protected]
References
Adesope, O. O., Trevisan, D. A., & Sundararajan, N. (2017). Rethinking the use of tests: A meta-analysis
of practice testing. Review of Educational Research, 87(3), 659–701.
Arnold, K. M., & McDermott, K. B. (2013). Free recall enhances subsequent learning. Psychonomic
Bulletin & Review, 20(3), 507–513. https://doi.org/10.3758/s13423-012-0370-3
Bjork, R. A. (1975). Retrieval as a memory modifier: An interpretation of negative recency and related
phenomena. In R. L. Solso (Ed.), Information processing and cognition: The Loyola symposium
(pp. 123–144). Erlbaum.
Carpenter, S. K., & DeLosh, E. L. (2006). Impoverished cue support enhances subsequent retention:
Support for the elaborative retrieval explanation of the testing effect. Memory & Cognition,
34(2), 268–276. https://doi.org/10.3758/BF03193405
Collins, D. P., Rasco, D., & Benassi, V. A. (2018). Test-Enhanced learning: Does deeper processing on
quizzes benefit exam performance? Teaching of Psychology, 45(3), 235–238.
Craik, F. I. M., & Lockhart, R. S. (1972). Levels of processing: A framework for memory research. Journal
of Verbal Learning and Verbal Behavior, 11(6), 671–684. https://doi.org/10.1016/S0022-
5371(72)80001-X
502
Craik, F. I. M., & Tulving, E. (1975). Depth of processing and the retention of words in episodic
memory. Journal of Experimental Psychology: General, 104(3), 268–294.
de Winstanley, P. A., Bjork, E. L., & Bjork, R. A. (1996). Generation effects and the lack thereof: The role
of transfer-appropriate processing. Memory, 4(1), 31–48. https://doi.org/10.1080/741940667
Glover, J. A. (1989). The "testing" phenomenon: Not gone but nearly forgotten. Journal of Educational
Hays, M. J., Kornell, N., & Bjork, R. A. (2013). When and why a failed test potentiates the effectiveness of
subsequent study. Journal of Experimental Psychology: Learning, Memory, and Cognition, 39(1),
290–296. https://doi.org/10.1037/a0028468
Kang, S. H. K., McDermott, K. B., & Roediger, H. L., III. (2007). Test format and corrective feedback
modify the effect of testing on long-term retention. European Journal of Cognitive Psychology,
19(4–5), 528–558. http://doi.org/10.1080/09541440601056620
Karpicke, J. D. (2017). Retrieval-based learning: A decade of progress. In J. H. Byrne (Ed.), Learning and
memory: A comprehensive reference (2nd ed., pp. 487–514). Elsevier.
Kornell, N., Klein, P. J., & Rawson, K. A. (2015). Retrieval attempts enhance learning, but retrieval success
(versus failure) does not matter. Journal of Experimental Psychology: Learning, Memory, and
Kornell, N., & Metcalfe, J. (2006). Study efficacy and the region of proximal learning framework. Journal
of Experimental Psychology: Learning, Memory, and Cognition, 32(3), 609–622.
Kornell, N., & Vaughn, K. E. (2016). How retrieval attempts affect learning: A review and synthesis. In B.
H. Ross (Ed.), Psychology of learning and motivation (Vol. 65, pp. 183–215). Academic Press.
https://doi.org/10.1016/bs.plm.2016.03.003
Little, J. L., & Bjork, E. L. (2015). Optimizing multiple-choice tests as tools for learning. Memory &
Little, J. L., & McDaniel, M. A. (2015). Metamemory monitoring and control following retrieval practice
for text. Memory & Cognition, 43(1), 85–98. https://doi.org/10.3758/s13421-014-0453-7
Lockhart, R. S. (2002). Levels of processing, transfer-appropriate processing, and the concept of robust
encoding. Memory, 10(5-6), 397–403. https://doi.org/10.1080/09658210244000225
classroom, European Journal of Cognitive Psychology, 19(4-5), 494–513.
McDaniel, M. A., Thomas, R. C., Agarwal, P. K., McDermott, K. B., & Roediger, H. L., III. (2013). Quizzing in
middle-school science: Successful transfer performance on classroom exams. Applied Cognitive
Psychology, 27(3), 360–372. https://doi.org/10.1002/acp.2914
McDaniel, M. A., Wildman, K. M., & Anderson, J. L. (2012). Using quizzes to enhance summative
assessment performance in a web-based class: An experimental study. Journal of Applied
Research in Memory and Cognition, 1(1), 18–26. http://doi.org/10.1016/j.jarmac.2011.10.001
503
Morris, C. D., Bransford, J. D., & Franks, J. J. (1977). Levels of processing versus transfer appropriate
processing. Journal of Verbal Learning and Verbal Behavior, 16(5), 519–533.
Roediger, H. L., III, Agarwal, P. K., McDaniel, M. A., & McDermott, K. B. (2011). Test-enhanced learning in
the classroom: Long-term improvements from quizzing. Journal of Experimental Psychology:
Applied, 17(4), 382–395. https://doi.org/10.1037/a0026252
Roediger, H. L., III, & Karpicke, J. D. (2006). The power of testing memory: Basic research and
implications for educational practice. Perspectives on Psychological Science, 1(3), 181–210.
Shobe, E. (2022). Achieving testing effects in an authentic college classroom. Teaching of Psychology,
Soderstrom, N. C., & Bjork, R. A. (2014). Testing facilitates the regulation of subsequent study time.
Journal of Memory and Language, 73, 99–115. https://doi.org/10.1016/j.jml.2014.03.003
504
In Their Own Words The Science of Virtual Teaching and Learning for
Ophthalmology Medical Education
to Apply the Science of Learning in Tony Succar
Massachusetts Eye and Ear, Harvard Medical School
Editors
Lauren L. Kordonowy
John R. Grigg
Victor A. Benassi
Save Sight Institute Faculty of Medicine and Health, The University
of Sydney
The application of virtual technology has revolutionized teaching and learning in ophthalmology medical
student education. Advances in the science of learning have provided educators with a solid foundation
for designing strategies to address many of the educational challenges faced, with a range of practical
applications for effective teaching and optimized learning (Benassi et al., 2014). This chapter describes
science of learning principles which were applied to The Virtual Ophthalmology Clinic (VOC) to enhance
ophthalmology teaching for medical students (Succar et al., 2013; Succar et al., 2019).
The VOC aims to empower the student with skills in critical thinking, problem solving, and reasoning.
Medical students can sharpen their clinical reasoning skills by formulating a diagnosis and management
plan on virtual patients with simulated eye conditions in a safe learning environment before practicing
on real patients (Succar et al., 2019; Succar et al., 2013). Science of learning interventions which were
applied to VOC to optimize student learning include:
• Multimedia Instruction (Mayer, 2014). People learn more deeply from words and graphics than
from words alone. This assertion, called the multimedia principle, forms the basis for using
multimedia instruction, that is, instruction containing words (spoken text or printed text) and
graphics (illustrations, charts, photos, animation, or video) that is intended to foster learning
(Mayer, 2009). The theory of multimedia learning is founded upon guiding principles in cognitive
science relating to how people receive and process information (Overson, 2014). The VOC is a
multimedia computer-assisted learning module, which entails real patient scenarios, involving
simulated clinical cases delivered in a virtual learning environment. In ophthalmology, visual
diagnosis plays a crucial role in the ocular examination. Thus, a visual presentation of details in
ophthalmology can be performed in an easier way through multimedia technology. VOC
provides a user-friendly learning environment through interactive and multimedia content.
• Opportunities to make errors and learning from failures (Clark & Bjork, 2014). VOC provides
students with a virtual simulation or clinical problem, with the opportunity to make errors and
then learn from them in a safe environment before practicing on real patients. This simulation
also shows students where they went wrong and helps them correct their errors through timely
feedback. This aids student understanding of where in their thought process and clinical
reasoning they went wrong, and it can redirect their thinking to formulate a correct diagnosis
and treatment plan, especially in medicine where stakes of failure are high and inadequate
knowledge retention may result in detrimental consequences.
• Metacognition (Girash, 2014). VOC fosters self-reflection and self-questioning by providing
students with probing questions, which is beneficial in helping students understand their mental
processes. Self-analysis discussions can encourage students to think about their thinking
(metacognition). For example, VOC prompts questions which provides students opportunities to
505
dive deeper into why certain exams were chosen to support their diagnosis. Questions based on
a student’s response can help them dissect their learning and understanding in ophthalmology,
getting deeper into their own thought process and understanding.
• Asking students to justify and explain their thinking (Chiu & Chi, 2014). Self-explaining, or
making sense of new information by explaining it to oneself, can greatly enhance learning for
students across ages, domains, and instructional types. Over two decades of research have
demonstrated that explaining concepts while learning results in more effective problem-solving,
more robust conceptual understanding, and better monitoring of learning (Chiu and Chi, 2014).
• Prior knowledge (Ambrose & Lovett 2014). VOC helps students explore and understand how
ideas are connected and build upon past knowledge. Students apply what they have learned
from their pre-clinical years of studying anatomy, physiology, and pathologies of the eye and
visual system, and they apply this by interviewing and interpreting examinations of virtual
patients and formulating a diagnosis and treatment plan. Thus, prior knowledge plays a critical
role in VOC, which allows students to continually make connections among ophthalmology
concepts, principles, and theories, as well as connections among didactic teaching and clinical
examinations.
• Use of Feedback (Hattie & Yates, 2014). Once students complete and submit their report for a
VOC case, feedback with ideal answers is provided to promote learning and help students
develop an awareness of their learning strengths and weaknesses as well as the skill to
recognize and correct errors.
• Generating Active Learning (Bertsch & Pesta, 2014). VOC generates active learning by engaging
students with virtual patient interactions, allowing students to conduct an examination and
formulate a diagnosis and treatment plan, to promote higher-order thinking skills, and to
engage in deep learning.
These interventions, which form the basis of VOC, can be crucial in helping students improve their
critical thinking skills, clinical reasoning, comprehension, retention, retrieval, and self-regulation, which
are all essential to lifelong learning.
The Virtual Ophthalmology Clinic (VOC) Overview

The interactive VOC was designed to optimize student learning and decision making in ophthalmology.
The VOC is a multimedia computer-assisted learning module, which entails real patient scenarios
involving simulated clinical cases delivered in a virtual learning environment that focuses on training
students’ clinical reasoning, history taking, diagnosis, and management plans in ophthalmology. The
simulated consultations are introduced where virtual patients are chosen from a waiting room and are
based on clinical ophthalmic problems derived from real patient interviews (Succar et al., 2013; Succar
et al., 2019). Delivery of traditional ophthalmic knowledge occurs through lectures, theoretical and
practical courses, bedside teaching as well as case-oriented teaching in small groups. Problems
encountered with these forms of teaching have arisen due to limited staff and resources. A new
challenge that has significantly impacted medical student education has been the restrictions on clinical
placements that have arisen from the COVID-19 pandemic. Advances in Information Technology have
opened new possibilities in the training of medical students utilizing interactive multimedia. The VOC
provides a computer-based setting of the doctor-patient relationship in a virtual consulting room. It is
comprised of clinical cases designed to cover the major causes of blindness and acute visual loss, major
ophthalmology emergencies, and the impact of blindness on the community. These include Age-related
Macular Degeneration (AMD), blepharitis, diabetes, and eye problems, foreign body, glaucoma, herpes
simplex keratitis, myopia, scleritis, uveitis and watery eye, all of which are simulated as real patient
506
presentations performed by actors in a pre-recorded video interview (Succar et al., 2019; Succar et al.,
2013).
The program starts when a student selects a patient, and a simulated eye condition is randomly
assigned. The examination consists of four sections: medical interview, clinical examination, laboratory
investigations, and patient treatment. Medical history is the key to establishing a diagnosis, choosing
appropriate investigations for confirming the diagnosis and examination, and formulating a
management plan. A majority of diagnoses can be made by a thorough history which assists in directing
the examination as well as arranging additional investigations. VOC allows students to conduct a history
which provides clues to the site of pathology, facilitating which part of the eye to examine. The
Conversation Navigator TM (CN) is an interactive tool used for simulating history taking. It allows students
to take on the professional role of a medical practitioner and to interact with patients by probing further
history questions to formulate a diagnosis and management plan. This is made possible through specific
icons which represent symbols reflecting the students’ questions, including concepts of time, ocular
symptoms, and review of systemic health (Succar et al., 2013; Succar et al., 2019). The student
determines the line of inquiry by selecting icons that most closely reflect their response to patient
dialogue, thereby uncovering clues to the condition. The answers are then typed in an electronic form
constructed in the form of a medical record which are saved and submitted to an online tutor for
correction and feedback. The student can direct the questioning in a nonlinear manner. In summary,
there is no specific order which students need to follow; they choose the order of inquiry when
extracting the information from the patient’s history. A working diagnosis can then be made based on
the interview (Succar et al., 2019; Succar et al., 2013).
Quantitative and Qualitative Evaluation

The purpose of our research was to determine whether the implementation of VOC, traditional modes
of teaching, or some combination of the two is of greater educational effectiveness. A prospective
randomized controlled trial (RCT) was undertaken to evaluate the effectiveness of VOC, in comparison
with traditional modes of teaching (Succar et al., 2013).
To test this hypothesis, a pre- and post-test questionnaire was used. A secondary aim was to determine
students’ perception of the VOC program and assess the value of VOC as an additional educational
resource for teaching ophthalmology to medical students. One hundred and eighty-eight third-year
medical students commencing their clinical ophthalmology rotation were randomly divided into two
groups. The experimental group (n = 95) received the VOC intervention as an additional educational
resource while students in the control group (n = 93) took part in the traditional hospital-based
teaching. However, once the intervention was complete, the control group were then given access to
the virtual ophthalmology clinic. This was to provide equal access for all students to similar
ophthalmology teaching experiences and resources. The post-rotation knowledge test was conducted
immediately after the student had completed their ophthalmology rotation. Second, the researchers
assessed the ophthalmic knowledge in those starting their intern year. This analysis as well as examining
retained ophthalmic knowledge also addressed the potential bias of extra time spent on ophthalmic
education activities for the VOC group. The pre-post test scores were used to determine if there was a
significant statistical difference between the students using VOC and traditional ophthalmology
instructions, and the effect that VOC had on student learning outcomes. Statistical analysis was
performed using SPSS software package (Version 17.0, SPSS, Inc.). A p value of less than 0.05 was
considered statistically significant (Succar et al., 2013; Succar et al., 2019).
507
Results
There was a statistically significant (p < 0.001) within-subject improvement pre-to post rotation in the
number of correctly answered questions for both the control and experimental groups (mean control
group = 10%; experimental group = 17.5%). The improvement was significantly greater in the
experimental group (mean difference in improvement between groups = 7.5%, p < 0.001). At a 12-
months follow-up testing, students in the experimental group scored on average 8% higher than those
in the control group. The learning benefit of the VOC was maintained after 12 months (Succar et al.,
2013; Succar et al., 2019). A deeper understanding regarding the educational value of VOC is gained
when analysing students’ perceptions and attitudes towards the program, as measured by the
qualitative questionnaire in the second phase of the study. The VOC module was highly regarded and
enthusiastically received as seen in the students’ overall satisfaction and positive responses (Succar et
al., 2013, Succar et al., 2019).
Advantages of the VOC which utilizes the best of today’s eLearning capabilities include (Succar et al.,
2013; Succar et al., 2019):
• Emphasis on self-directed learning.
• Enhanced medical interviewing techniques.
• Students develop skills in ophthalmic diagnosis and management plans before practicing on real
patients.
• Enhanced medical decision making.
• Exposure to a broad range of ophthalmic conditions through virtual simulation, expanding what
students encounter in a clinical teaching environment.
• Students can work on their own time and pace, at off-site locations at any hour of the day.
• Enhanced revision opportunities by providing recurrent access to clinical case information.
• Course content can be quickly updated without causing stress to the patient.
Discussion and Future Developments

Today, VOC is an exciting innovative educational resource at the leading edge of medical education
technology. VOC effectiveness has been demonstrated and allows students to further reflect on their
findings and analyze critically their clinical reasoning processes with the opportunity for self and tutor
assessments. Revision and reinforcement will further enhance the skills which are required to formulate
a diagnosis and case management plans, building on their medical history taking and observation skills.
The VOC program will continually evolve with improvements based on formative evaluations, progress
assessments, and student feedback. There is an opportunity to further develop the program by adding
additional features. These include the use of 3D animations, teaching ocular examinations in which
students can perform on the virtual patients, such as measuring visual acuity, pupillary function, and
ocular movements, and demonstrating video recorded ophthalmic surgical procedures. The
Conversation Navigator responses could also be translated into additional languages. This application
could also be included in training ophthalmology residents, general medical practitioners, orthoptists,
optometrists, and ophthalmic nurses, who can utilise the program to learn and gain skills in
ophthalmology in an interactive environment. In addition, the icons can be programmed to reflect other
health-related disciplines where history taking forms the basis of a clinical examination.
508
Although ophthalmology is used as the primary subject, developing and applying the higher-order
cognitive processes needed in history taking, clinical decision making, and reasoning are critical skills
which can be transferred to other disciplines and life-long learning.
Author Note
Send correspondence to Tony Succar: [email protected]
References
Society for the Teaching of Psychology. http://teachpsych.org/ebooks/asle2014/index.php
Benassi, V. A., Overson, C. E., & Hakala, C. M. Eds. (2014). Applying science of learning in education:
Bertsch, S., & Pesta, B. (2014). Generating active learning. In V. A. Benassi, C. E. Overson, & C. M. Hakala
(Eds.), Applying science of learning in education: Infusing psychological science into the
Chiu, J. & Chi, M. (2014). Supporting Self Explanation in the Classroom. In V. A. Benassi, C. E. Overson, &
C. M. Hakala (Eds.), Applying science of learning in education: Infusing psychological science into
the curriculum (pp. 91-103). Society for the Teaching of Psychology.
Clark, C., & Bjork, R. (2014). When and Why Introducing Difficulties and Errors Can Enhance Instruction.
In V. A. Benassi, C. E. Overson, & C. M. Hakala (Eds.), Applying science of learning in education:
Infusing psychological science into the curriculum (pp. 20-30). Society for the Teaching of
Psychology. http://teachpsych.org/ebooks/asle2014/index.php
Hattie, J. & Yates, G. (2014). Using Feedback to Promote Learning. In V. A. Benassi, C. E. Overson, & C. M.
Hakala (Eds.), Applying science of learning in education: Infusing psychological science into the
Mayer, R. E. (2009). Multimedia learning (2nd ed). Cambridge University Press.
Mayer, R. (2014). Research-Based Principles for Designing Multimedia Instruction. In V. A. Benassi, C. E.
Overson, C (2014) Applying Multimedia Principles to Slide Shows for Academic Presentation. In V. A.
Benassi, C. E. Overson, & C. M. Hakala (Eds.), Applying science of learning in education: Infusing
Sawyer, R. K. (Ed.). (2006). Cambridge handbook of the learning sciences. Cambridge University Press.
509
SPSS Inc. (2008). SPSS Statistics for Windows (17.0). SPSS Inc.
Succar, T., Zebington, G., Billson, F., Byth, K., Barrie, S., McCluskey, P., & Grigg, J. (2013). The impact of
the Virtual Ophthalmology Clinic on medical students’ learning: A randomised controlled
trial. Eye, 27(10), 1151–1157. https://doi.org/10.1038/eye.2013.143
Succar, T. and Grigg, J. (2019), The Role of Educational Technology in Promoting the Inclusion of Rural
Clinical Schools for Ophthalmology Teaching Using Virtual Patients. In J. Hoffman, P. Blessinger,
& M. Makhanya. (Eds.), Strategies for fostering Inclusive classrooms in higher education:
International perspectives on equity and Inclusion (Innovations in Higher Education Teaching and
Learning, Vol. 16), Emerald Publishing Limited, pp. 167-181.
510
In Their Own Words Infusing Critical Thinking Skills Into Course
Content
to Apply the Science of Learning in Laura C. Edwards
Taylor University
Editors
Lauren L. Kordonowy
Victor A. Benassi
“The human foot was not built for ballet and only with discipline and training can it endure the strain
and produce beauty” (Daly, 2015, p. 8). It is the same with students’ minds; only with discipline and
training can students develop reflective, intellectual, and purposeful thinking.
If someone asked me if I taught my students to think critically, I answered with an emphatic yes. If
asked how, I would say that I challenged them to consider multiple perspectives, look for underlying
assumptions, and be aware of fallacies. How I approached the teaching of critical thinking (CT) was not
incorrect, and I believe my students had the opportunity to think critically.
There has been a historical concern to improve the quality of disciplined thinking (e.g., Bloom et al.,
1956; Dewey, 1933; Glaser, 1941) with little agreement regarding the type of instruction that results in
better outcomes (e.g., Ennis, 1987; McPeck, 1981; Tiruneh et al., 2018). Nonetheless, the importance
and challenge of fostering CT continue to command the attention of scholars worldwide (e.g., Halpern &
Butler, 2019; Willingham, 2020), but according to empirical evidence, college students still exhibit
inadequate CT achievement (Arum & Roksa, 2011, Willingham, 2020).
Tiruneh et al. (2014), in their systematic review of CT instruction in higher education, rendered, “The
main limitation in the current empirical evidence is the lack of systematic design of instructional
interventions that are in line with empirically valid instructional design principles” (p. 1). With that in
mind, I conducted a research study (Edwards, 2017) to learn if there were specific, empirically-based
learning strategies that fostered students’ higher-order thinking skills. CT skills refer to abilities such as
identifying parts of an argument, making inferences using reasoning, judging, evaluating evidence,
recognizing fallacies, and solving problems, among other competencies (Paul & Elder, 2009). My
research shows that the systematic application of science of learning principles to teach CT skills is
efficacious (as evidenced by pre-post test results) in enhancing students' thinking skills.
Ennis (1987) categorized four approaches to foster CT:
• the general approach introduces CT detached from specific content;
• the immersion approach presents the content in thought-provoking styles, but CT's general
principles and structure are not explicit;
• the mixed approach combines two methods: the general and either infusion or immersion;
• the infusion method integrates the instruction of CT skills into the framework of a specific
course content explicitly and embeds the skills in all curricular content. The results from a meta-
analysis (Abrami et al., 2008) indicated that the infusion approach appeared as the second most
effective method (following the mixed approach). Since the target population for this study was
511
faculty members trained in the infusion method, the focus of this inquiry was on this approach.
done
The purpose of my study (Edwards, 2017) was to investigate the training educators received in the
infusion approach and the strategies the educators employed to infuse specific CT skills into their course
content. Recall-based, application-focused pretest-posttest scores were used to discern the presence of
significant improvement in the students’ CT skills performance.
Overview of the Critical Thinking Training Program

A mini-grant from Taylor University’s Center for Teaching and Learning Excellence provided stipends to
professors who volunteered to participate in the “The Critical Thinking Project” (CTP). The training
expanded on Halpern’s four-part model of CT instruction (for review, see Halpern, 2014) and utilized
empirically supported pedagogical processes consistent with the science of learning (Mayer, 2008). The
pedagogical design for faculty instruction focused on the infusion of explicit instruction (Halpern &
Butler, 2019), critical dialogue (Mazur, 1997; Zhang et al., 2017), metacognitive processes (Harrington &
Zakrajsek, 2017), retrieval practice (Willingham, 2020), continuous feedback (Bain, 2021, Mazur, 2017),
transfer (Chew, 2014), and CT dispositions (Snyder et al., 2019). It is important to note that the
components of the infusion model are not linear as the elements of the process often overlap in the
process of teaching CT.
The training consisted of a three-day workshop during the summer, followed by a semester-long
infusion of specific CT skills. As part of the training, the instructor embedded the CT Skills into all course-
related materials (course objectives, learning outcomes, assessments, lectures, in-class activities, and
homework). The participants created a portfolio with the proposed interventions to present to the
workshop facilitator within three weeks of the end of the workshop.
Each professor was assigned an advanced undergraduate statistics student who received training to
observe the infusion process, provide respectful feedback, and score the pretests and posttests. The
student attended each class during the semester to support the proper implementation of the infusion
method, provide informative feedback consultations, and produce an end-of-semester feedback report
on students’ CT skills growth. The aspiration of the training was for faculty to continue for two more
semesters of infusion with class observations.
To measure students’ CT progress, the project director developed recall-based pretests and posttests for
each specific CT skill. Each test contained the same 3Ds process (Beyer’s, 1985) of defining, doing, and
discussing. Defining the CT skill helps develop a mental framework to perform it; doing involves the
appropriate implementation of the skill; and discussing requires the students to explain their train of
thought as they addressed the issues and questions presented. In addition, students had to apply that
same CT skill to a novel situation to assess the accurate transfer of the acquired CT skill. The students
took the pretest and posttest during the first and final weeks of the semester, respectively.
I participated in the project. After receiving feedback from my student observer, I realized that the
interventions used in the project had been effective (Edwards et al., 2016). I decided to interview seven
faculty members. Instructors from the Psychology, Exercise Physiology, Business, Theology,
Mathematics, Computer Science, and Research Design departments who received the infusion training
agreed to participate.
I checked the pre-post test results of all the students (n = 133) who had taken classes with the
interviewed instructors (for details, refer to Edwards, 2017). The results indicated that students’ CT skills
could improve within one semester when principles from the science of learning were used to embed
512
the CT skills into course content. One hundred percent of the students’ CT skills scores increased from
pretest to posttest, and results revealed growth in 88% of the CT skills to be both statistically (p < .05)
and practically significant, with medium (d > .5) to very large (d > 1.30) effects. Instructional
interventions with effect sizes of 0.4 or higher should be considered educationally relevant (Hattie,
2011).
Applying the Science of Learning

My interviews revealed seven strategies used to embed the CT skills. These strategies are consistent
with best practices from the science of learning. They are:
• explicit teaching;
• pedagogical design;
• retrieval-based learning,
• class discussions;
• modeling;
• transfer;
• metacognitive reflection.
Explicit Teaching
Proponents of explicit instruction argue that CT skills are not likely to develop if the teaching occurs in
an implicit manner, which implies the mention of a CT skill without the stipulation of the specific steps.
Explicit teaching has some of the same components of worked examples (Renkl, this volume), as step-
by-step demonstrations on how the CT skill is successfully applied support the initial acquisition of the
skills (Schalk et al., 2020). For instance, in addition to stating, “let’s look at the underlying assumptions
(identifying),” the instructor overtly helps students operationalize it and follow the steps of the cognitive
processes required to perform the task. All professors interviewed utilized handouts that included the
definitions and the steps necessary to execute the CT skills, and they often asked the students to refer to
the handout. Explicitly teaching the CT skills maximizes long-term retention and the transfer of the skills
to be applied in new situations (Fries et al., 2021). When instructors teach CT explicitly, students
become more proficient thinkers (Ambrose & Lovett, 2014; Halpern & Butler, 2019).
Designing Effective Pedagogy
How the learning environment is structured remarkably affects learning, retention, and knowledge
transferability (Benassi et al., 2014). Educators will want to consider the pedagogy to seamlessly
facilitate learning of both content knowledge and the application of the CT skills (Schell & Butler, 2018;
Willingham, 2020). Instructors must be deliberate and purposeful in infusing the CT skills in the whole
course design. This process includes adding the CT skills to the learning objectives and outcomes,
updating lectures, assignments, assessments, class activities, rubrics, and discussions, and instructors
adjusting their vocabulary to reflect the CT skills. This infusion strategy requires more than making the
skills explicit; it involves a deliberate effort in implementing the needed adaptations, and this should be
reflected in every aspect of the course (Chew, 2014).
Retrieval-Based Learning
Systematic and repeated practice is conducive to later recall of learned information and is a robust and
well-established intervention in the science of learning (Ariel & Karpicke, 2018; Karpicke, 2017; Schell &
513
Butler, 2018). It is also effective when teaching CT skills. For example, when professors teach the CT skill
of “Analyzing an Argument” through the infusion method, students learn to adhere to the following
steps systematically: 1. Recognize the premises. 2. Decide whether the premises are acceptable and
consistent. 3. Decide whether the premises are relevant to the conclusion and provide consistent
support for the conclusion. 4. Consider if there are missing components (Halpern, 2014, p. 203). When
utilizing the retrieval-based learning strategy, students are asked to recall and apply the above steps
immediately after learning them, which has invaluable benefits for students’ CT acquisition (Edwards,
2017). Engaging the students in the retrieval practice of a specific CT skill (e.g., analyzing an argument)
via repeated and successful retrieval of the steps through course content is vital, as this process fosters
a deeper understanding of content through the CT skill and enables the transfer of the thinking skill to
other contexts (Karpicke, 2017; Schell & Butler, 2018, Willingham, 2020). Retrieval practice provides
students with multiple opportunities to recall and apply a specific CT skill to enhance its mastery.
A mathematics professor who participated in the study chose math problems that employed the CT skills
of analogical thinking and looking for patterns. He used his two-hour labs to have the students problem-
solve, utilizing the 3-Ds steps of the CT skills “over and over again.” Repeated practice fosters better
retention than a single correct retrieval attempt (cf. Karpicke, 2017, on retrieval practice). It is also true
for teaching thinking skills.
Another professor expanded on the one-minute paper (for a review, see Lang, 2016) by asking students
to connect one of the CT skills to the material covered at the end of class. The professor called it H.O.T.
Wrappers (Higher-order Thinking Wrappers). Here is an example of a wrapper prompt:
We often hear these words on the news, “research says . . .” Please write one paragraph and
discuss one of the CT skills. Following the required CT steps, how can you become an informed
consumer of research?
With this intervention, instructors quickly notice if the students correctly utilize the CT skill. They can
then provide timely feedback (Darling-Hammond et al., 2020). This exercise also invokes students’ prior
knowledge (Fiorella & Mayer, 2016) and effectively engages students in ‘self-explaining.’ Self-explaining
is an active and constructive activity requiring students to explain newly acquired knowledge to
themselves (Chiu & Chi, 2014). When students use self-explaining to clarify why and how they are
employing the CT skill, it enhances their understanding and application of the steps required to think
critically while also promoting the skills’ transfer. For instance, the professor who chose the analyzing an
argument CT skill might ask students to first explain to themselves the processes (the why and how)
they would follow with specific course content (e.g., detecting stereotypes).
Discussion
Small group, larger group, or two-person discussions rendered powerful learning opportunities for
students to think critically. The discussions promoted engagement in multiple higher-order cognitive
processes while addressing course content, which is consistent with prominent findings from the science
of learning literature (e.g., Mazur, 1997). Instructors, however, need to frame the discussion prompts to
reflect the modes of inquiry desired (Darling & Hammond, 2020). The professors I interviewed
embedded the CT skills in students' peer discussions in multiple ways: they had students explore
controversial topics, assigned groups to seek alternate views to the ones presented, and asked students
to detect underlying assumptions. Students subsequently explained and discussed their findings with
the entire class.
Other discussion strategies mentioned during the interviews included students first individually working
through concept maps (Davies et al., 2019; Fiorella & Mayer, 2016; O'Day & Karpicke, 2020), after which
514
they gathered in small groups to explain and discuss their thinking. The students, as a group, then
produced one persuasive paragraph indicating which CT skills they employed and why they did so.
Guided class discussions are particularly effective in teaching higher-order thinking (Abrami et al., 2015).
Another useful strategy is Mazur's Peer Instruction (see Schell & Butler, this volume; Schell & Butler,
2018). Students think through the arguments they are developing, thus becoming familiar with the
required cognitive process. They then discuss their answers with their peers (a version of a self-
explanation strategy), reviewing their initial answers after the peer-sharing process. Finally, the
instructor provides students with feedback or further instruction. The mathematics, computer science,
and research design professors utilized this technique with the CT skill known as the Compensatory
Unequal Weights Decision Making Model used to make wise decisions by evaluating the possible options
and selecting the best one for the given situation (Rothrock & Yin, 2008). The use of these strategies to
foster CT is in line with the constructive mode of learning, which occurs when learners engage in self-
explaining, generating, and discussing ideas (Fiorella & Mayer, 2016; Zhang et al., 2017).
Modeling the Skills via Elaborative Interrogation
Elaborative Interrogation (EI) involves asking ‘why’ and ‘how’ questions about the learned material.
Research has provided strong evidence of EI’s effectiveness (Weinstein et al., 2018). In the process of
teaching the CT skill, several professors mentioned modeling the asking of why and how questions. For
example, why use this particular CT skill in this situation? How would you go about it? EI was also
present in how teachers modeled the process to engage in higher-order thinking as they verbalized their
decision-making process while teaching. Three professors expounded the merits of consistently
modeling good questions, thereby creating a context for quality thinking. For example, they selected
question stems, such as ‘explain why (the selected CT skill) is needed as we study’ or ‘explain how you
would apply the CT skill.’ Similarly, these teachers stated that they consistently modeled this line of
inquiry to the students—for example, some would say, ‘when I read this paragraph, I am asking why the
author chose this perspective or how the author arrived at this premise.’ The professors also stimulated
the students to generate good questions about that day’s class content by applying the CT skills.
Another faculty member described modeling the CT skills by working on solving a problem alongside his
students and asking them questions throughout the process. The instructors expounded on the merits
of consistently modeling their metacognitive process out loud through EI. Snyder et al. (2019) explained
that when instructors practice guided modeling, they mentor their students by employing the thinking
skills to work through problems explicitly and systematically. This type of modeling creates a context for
expert thinking (Zepeda et al., 2019).
Teaching for Transfer
Transfer refers to the application of knowledge or a skill effectively in a setting other than the one
learned initially or experienced (Halpern & Butler, 2019). It is well established through a long research
tradition that students do not routinely transfer learning to different contexts (e.g., Engle et al., 2012;
Hakala, 2018; Fiorella & Mayer, 2016). Along the same lines, the transferability of CT skills is not obvious
to students. However, its transfer across topics is essential for promoting and mastering higher-order
thinking skills (Halpern & Butler, 2019). Students are much more likely to transfer CT skills if instructors
provide opportunities for students to observe how a newly acquired skill can apply to novel situations
(Willingham, 2020).
When faculty explain that the CT skills are readily and efficiently utilized in multiple contexts, they
generate transfer-expectations. For example, some faculty in my study asked students for examples of
when they had previously employed a thinking skill, how they used it, and why they thought its use was
appropriate. This process promotes personalization (Mayer, 2014) and applies self-explanation
515
principles, which are both known to foster transfer (Chiu & Chi, 2014; Fiorella & Mayer, 2016). This
strategy is in line with the practicing connections framework (Fried et al., 2021), which proposes that
"connectedness is key to the coherent mental schemas that underlie deep understanding and
transferable skills" (p. 739).
Another way the professors promoted the students' transfer habit was by introducing variability during
repeated retrieval practice. Altering the context for retrieval of the CT skills aids students' awareness of
the skill's transferability (Butler et al., 2017; Halpern & Hakel, 2003). Another assignment was the
"Application of Concepts" used by the psychology professor. Throughout the semester, students from a
Psychology of Learning class were encouraged to find examples of how the CT skills learned in the
course could be applied to a novel situation, such as content from another course, a movie or series
they watched, or a book they were reading. For the assignment, students defined their chosen CT skill,
explained how they employed it in a different context, and described why they felt it was appropriate.
Fries et al. (2021) urged instructors to provide opportunities for students to practice creating
connections: "We want students to develop robust and flexible knowledge that they can take with them
into the world, coordinate with other knowledge, and apply to new problems and in new contexts" (p.
742).
Metacognitive Reflection
Metacognitive reflection draws from cognitive and affective information, allowing students time to
process what they are learning and thinking as well as to create meaning (Major et al., 2016). In this
study, reflective thinking and writing provided opportunities for students to expand their grasp and
enhancement of their thinking skills. Applying reflection in this manner is supported by research and
grounded in cognitive theory (Ambrose & Lovett, 2014; Case & Colgan, 2017; Chew, 2014, Fries et al.,
2021; Harrington & Zakrajsek, 2017; Zepeda et al., 2019). Exemplars of reflective assignments adopting
the personalization principle (see Mayer, 2014) included assigning journal logs and reflection questions.
Some students reflected on essays, checked for underlying assumptions, considered issues from
different perspectives, or recognized their stereotypical tendencies. The benefits add to the ones
achieved via self-explanation, which helps students reflect on their knowledge, their misconceptions,
and identify gaps in the correct application and execution of the metacognitive process (Fiorella &
Mayer, 2016; for review, see Zepeda et al., 2019).
One of the interviewed professors assigned one-page reflections eight times throughout the semester.
Some of the reflection questions included:
• How have you developed thinking skills that help you think more like a mathematician when you
now encounter problems?
• What does it mean for you to think and learn like a scholar-practitioner?
Phelan (2012) proposed that “A person’s capacity for reflective thought is at the heart of the
metacognitive process, and it is through reflection that individuals become aware of their knowledge
and cognitive strategies” (p. 16). The metacognitive reflection is similar to Fiorella & Mayer's (2016)
concept of reflective knowledge building, which invites the students to reflect on their understanding of
the concepts learned.
Implications for Educators

Research findings document that many college students are not learning CT skills (Halpern & Butler,
2019; Willingham, 2020). The science of learning offers a set of evidence-based interventions that
516
promote and advance learning. Incorporating these empirically validated principles into instruction for
the fostering of CT can be relatively straightforward, empower instructors, and benefit students.
Most college and university educators do not have the pedagogical background to foster CT among their
students (Schell & Butler, 2018). Educators who desire to emphasize thinking skills may benefit from
understanding the pedagogical process undertaken by colleagues (Schell & Butler, 2018). Instructors
may optimize their effectiveness in infusing CT skills by adopting the strategies presented here (and
described and discussed throughout this volume).
The process that emerged from the interviews includes the following: 1. Designate a few hours prior to
the beginning of an academic term to select a few relevant CT skills for the course content. 2. Prepare a
packet with selected CT skills, provide definitions, and outline the steps required to achieve them. 3.
Embed the selected skills and the strategies to promote them in every aspect of the course. Educators
may be encouraged to discover that a modest amount of preparation can substantially increase the
effectiveness of imparting CT skills.
Considering that faculty development plays a vital role in students' CT outcomes (Jaffe et al., 2019),
educational institutions can utilize centers for teaching and learning to promote effective CT pedagogy
(Willingham, 2020). The findings from my study suggest that faculty training incorporating the principles
and practices from the science of learning not only heightened the educators' ability to infuse CT skills, it
also served as a catalyst that promoted student growth in skillful thinking, a highly prized student
educational outcome.
Author Note
Send correspondence to Laura Edwards at [email protected]
References
Abrami, P. C., Bernard, R. M., Borokhovski, E., Waddington, D. I., Wade, A., & Persons, T. (2015).
Strategies for teaching students to think critically: A meta-analysis. Review of Educational
Research, 85(2), 275-291. https://doi.org/10.3102/0034654314551063
Abrami, P. C., Bernard, R. M., Borokhovski E., Wade, A., Surkes, M. A., Tamim, R., & Zhang, D. (2008).
Instructional interventions affecting critical thinking skills and dispositions: A stage 1 meta-
analysis. Review of Educational Research, 78(4), 1102-1134.
Society for the Teaching of Psychology. http://teachpsych.org/ebooks/asle2014/index.php
Ariel, R., & Karpicke, J. D. (2018). Improving self-regulated learning with a retrieval practice
intervention. Journal of Experimental Psychology: Applied, 24(1), 43–56.
Arum, R., & Roksa, J. (2011). Academically adrift: Limited learning on college campuses. University of
Chicago Press.
Bain, K. (2021). Super courses: The future of teaching and learning. University Press.
517
Benassi, V. A., Tappin, E. M., Overson, C. E., Lee, M. J., O'Brien, E. J., Prudhomme White, B., Stiegler-
Balfour, J. J., & Hakala, C. M. (2014). Applying the science of learning: The Cognition Toolbox. In
V. A. Benassi, C. E. Overson, & C. M. Hakala (Eds.), Applying science of learning in education:
Infusing psychological science into the curriculum (pp. 194–205). Society for the Teaching of
Psychology. http://teachpsych.org/ebooks/asle2014/index.php
Beyer, B.K. (1985). Critical thinking: What is it? Social Education, 49, 270-276.
Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956). Taxonomy of educational
objectives: Handbook I: Cognitive domain. David McKay.
Butler, A. C., Black-Maier, A. C., Raley, N. D., & Marsh, E. J. (2017). Retrieving and applying knowledge to
different examples promotes transfer of learning. Journal of Experimental Psychology:
Applied, 23(4), 433–446. https://doi.org/10.1037/xap0000142
Case, J. & Colgan, M. (2017). “Big idea” reflection assignments for learning and valuing mathematics. 21st
conference of the Association of Christians in the Mathematical Sciences. Charleston, S. C. 9-21.
https://pillars.taylor.edu/cgi/viewcontent.cgi?article=1006&context=acms-2017
Chew, S. L. (2014). Helping students to get the most out of studying. In V. A. Benassi, C. E. Overson, & C.
M. Hakala (Eds.), Applying science of learning in education: Infusing psychological science into
the curriculum (pp. 215–223). Society for the Teaching of Psychology.
Chiu, J. L., & Chi, M. T. H. (2014). Supporting self-explanation in the classroom. In V. A. Benassi, C. E.
Daly, W. T. (1995). Beyond critical thinking: teaching the thinking skills necessary to academic and
professional success (Monograph series/National Resource Center for the Freshman Year
Experience and Students in Transition, University of South Carolina, no. 17). National Resource
Center for the Freshman Year Experience & Students in Transition, University of South Carolina.
https://files.eric.ed.gov/fulltext/ED390338.pdf
Darling-Hammond, L., Flook, L., Cook-Harvey, C., Barron, B., & Osher, D. (2020). Implications for
educational practice of the science of learning and development. Applied Developmental
Science, 24(2), 97–140. https://doi.org/10.1080/10888691.2018.1537791
Davies, M., Barnett, A., & van Gelder, T. (2019). Using computer-assisted argument mapping to teach
reasoning to students. In J. A. Blair (Ed.), Studies in critical thinking (pp. 131–176). Windsor
Center for Studies in Reasoning.
Dewey, J. (1933). How we think, a restatement of the relation of reflective thinking to the educative
process. D.C. Heath.
Edwards, L. C. (2017). The craft of infusing critical thinking skills: A mixed-method research on
implementation and student outcome. Journal on Centers for Teaching and Learning, 9, 47- 72.
Edwards, L. C., Snyder, S. J., & Sanders, A. L. (2016). Faculty development for fostering students’ critical
thinking. Journal on Centers for Teaching and Learning, 8, 4-27.
Engle, R. A., Lam, D. P., Meyer, X. S., & Nix, S. E. (2012). How does expansive framing promote transfer?
several proposed explanations and a research agenda for investigating them. Educational
Psychologist, 47(3), 215-231. https://doi.org/10.1080/00461520.2012.695678
518
Ennis, R. H. (1987). A taxonomy of critical thinking dispositions and abilities. In J. Baron & R. Sternberg
(Eds.), Teaching thinking skills: Theory and practice (pp. 9-26). W. H. Freeman.
Fiorella, L.., & Mayer, R.E. (2016). Eight ways to promote generative learning. Educational Psychology
Review 28, 717–741. https://doi.org/10.1007/s10648-015-9348-9
Fries, L., Son, J.Y., Givvin, K.B., Stigler, J. W. (2021). Practicing connections: A framework to guide
instructional design for developing understanding in complex domains. Educational Psychology
Review, 33, 739–762. https://doi.org/10.1007/s10648-020-09561-x
Glaser, E. M., (1941). An experiment in the development of critical thinking, Teacher’s College, Columbia
University. https://doi.org/10.1177/016146814204300507
Hakala, C. (2018, December 3). The mystery of translation. E-xcellence in Teaching Blog.
https://teachpsych.org/E-xcellence-in-Teaching-Blog/6943695
Halpern, D. F. (2014). Thought & knowledge: An introduction to critical thinking (5th ed.), Erlbaum.
Halpern, D. F., & Butler, H. A. (2019). Teaching critical thinking as if our future depends on it, because it
does. In J. Dunlosky & K. A. Rawson (Eds.), The Cambridge handbook of cognition and education
(pp. 51–66). Cambridge University Press. https://doi.org/10.1017/9781108235631.004
Halpern, D. F. & Hakel, M. D. (2003) Applying the science of learning to the university and beyond:
Teaching for long-term retention and transfer, Change: The Magazine of Higher
Learning, 35(4), 36-41, https://doi.org/10.1080/00091380309604109
Harrington, C., & Zakrajsek, T. (2017). Dynamic lecturing: research-based strategies to enhance lecture
effectiveness. Stylus Publishing, LLC.
Hattie, J. (2011). Visible learning. Routledge.
Jaffe L. E., Lindell D., Sullivan A. M. & Huang G. C. (2019). Clear skies ahead: optimizing the learning
environment for critical thinking from a qualitative analysis of interviews with expert teachers.
Perspectives on Medical Education, 8(5):289-297. https://doi.org/10.1007/s40037-019-00536-5
Karpicke, J. D. (2017). Retrieval-based learning: A decade of progress. In J. T. Wixted (Ed.), Cognitive
psychology of memory, Vol. 2 of learning and memory: A comprehensive reference (J. H. Byrne,
Series Ed.) (pp. 487-514). Academic Press.
Lang, J. M. (2016). Small teaching: everyday lessons from the science of learning. Jossey-Bass.
Major, C. H., Harris, M. S., & Zakrajsek, T. (2016). Teaching for learning: 101 intentionally designed
education activities to put students on the path to success. Routledge, Taylor & Francis Group.
Margulieux, L.. & Catrambone, R. (2016). Improving Problem Solving with Subgoal Labels in Expository
Text and Worked Examples. Learning and Instruction, 42, 58-71.
http://dx.doi.org/10.1016/j.learninstruc.2015.12.002
Mayer, R. E. (2008). Applying the science of learning: Evidence-based principles for the design of
multimedia instruction. American Psychologist, 63(8), 760–769. https://doi.org/10.1037/0003-
066X.63.8.760
Mayer R. E. (2014). Research-Based Principles for Designing Multimedia Instruction. In V. A. Benassi, C.
E. Overson, & C. M. Hakala (Eds.), Applying science of learning in education: Infusing
519
Mazur, E. (1997). Peer instruction: A user’s manual. Prentice Hall.
McPeck, J. E. (1981). Critical thinking and education. St. Martin's Press.
Paul, R., & Elder, L. (2009). The miniature guide to critical thinking (5th ed.). The Foundation for Critical
Thinking. https://www.criticalthinking.org/files/Concepts_Tools.pdf
Rothrock L., & Yin J. (2008) Integrating Compensatory and Noncompensatory Decision-Making Strategies
in Dynamic Task Environments. In: Kugler T., Smith J.C., Connolly T., Son YJ. (Eds.), Decision
modeling and behavior in complex and uncertain environments. Springer Optimization and Its
Applications (pp. 125-141). Springer. https://doi.org/10.1007/978-0-387-77131-1_6
Schalk, L., Roelle, J., Saalbach, H., Berthold, K., Stern, E., & Renkl, A. (2020). Providing worked examples
for learning multiple principles. Applied Cognitive Psychology, 34(4), 813-824.
Schell, J.A., & Butler, A. C., (2018) Insights from the science of learning can inform evidence-based
implementation of peer instruction. Frontiers in Education (33) 3.
https://doi.org/10.3389/feduc.2018.00033
Snyder, S. J., Edwards, L. C., & Sanders, A. L. (2019). An empirical model for infusing critical thinking into
higher education. Journal on Excellence in College Teaching, 30(1), 127-156.
Tiruneh, D.T., De Cock, M. & Elen, J. (2018). Designing learning environments for critical thinking:
Examining effective instructional approaches. International Journal of Science and Mathematics
Education, 16, 1065–1089. https://doi.org/10.1007/s10763-017-9829-z
Weinstein, Y., Madan, C. R., & Sumeracki, M. A. (2018). Teaching the science of learning. Cognitive
Research: Principles and Implications, 3(2). https://doi.org/10.1186/s41235-017-0087-y
Willingham, D. T. (2020, September 25) Ask the cognitive scientist: How can educators teach critical
thinking? American Educator. https://www.aft.org/ae/fall2020/willingham
Zepeda, C. D., Hlutkowsky, C. O., Partika, A. C., & Nokes-Malach, T. J. (2019). Identifying teachers’
supports of metacognition through classroom talk and its relation to growth in conceptual
learning. Journal of Educational Psychology, 111(3), 522–
541. https://doi.org/10.1037/edu0000300
Zhang, P., Ding, L., & Mazur, E. (2017). Peer instruction in introductory physics: a method to bring about
positive changes in students' attitudes and beliefs. Physical Review Physics Education
Research, 13(1). https://doi.org/10.1103/PhysRevPhysEducRes.13.010104
520
About the Authors
Pooja K. Agarwal, Ph.D. is an Assistant Professor of Psychology at the Berklee College of Music in Boston
and conducts research on how students learn. Her research has been published in leading journals;
highlighted by The New York Times, NPR, and Scientific American; and recognized by the National
Science Foundation. She received her Ph.D. from Washington University in St. Louis. She is the author of
the book Powerful Teaching: Unleash the Science of Learning. Dr. Agarwal is the Founder of
RetrievalPractice.org, a source of research-based teaching strategies for more than 15,000 teachers
around the world.
Greg Ashman is a PhD candidate at the University of New South Wales. He is Deputy Principal at a
regional school in Victoria, Australia.
Sabrina Badali is a graduate student in the department of psychological sciences at Kent State
university. Her research interests are centered around students’ self-regulated use of different learning
strategies.
Felicitas Biwer is a PhD researcher at School of Health Professions Education, Maastricht University, the
Netherlands. She investigates how to support students in applying effective study strategies for long-
term learning. She is also project member of the Study Smart team at Maastricht University.
Elizabeth Ligon Bjork is Research Professor at UCLA, recipient of UCLA’s Distinguished Teaching Award,
past Chair of the Academic Senate and Equity Advisor for the Division of Life Sciences. She has chaired
numerous committees focused on teaching, such as the Committee on Instructional Improvement
Programs; TA Training; and Undergraduate Student Initiated Education. She is an elected member of
the Society of Experimental Psychologists, Fellow of the Association for Psychological Science (APS),
member of Editorial Boards and Review Panels for NIMH and NSF. With Robert A. Bjork, she has
received the APS James McKeen Cattell Fellow Award and the Mentor Award.
Robert A. Bjork is Distinguished Research Professor at UCLA. He has been President or Chair of APS
(American Psychological Society), WPA (Western Psychological Association), the Psychonomic Society,
SEP (Society of Experimental Psychologists), and Chair of UCLA’s Department of Psychology. He edited
Memory & Cognition; edited Psychological Review; and co-edited Psychological Science in the Public
Interest. He has received UCLA’s Distinguished Teaching Award; APA’s Distinguished Scientist Lecturer;
APA’s Distinguished Service to Psychological Science Awards; SEP’s Lifetime Achievement Award; and
(with Elizabeth Bjork) APS’s James McKeen Cattell and Mentoring Awards. He is a Fellow of the
American Academy of Arts and Sciences.
Nicole Boucher started working in the Learning and Cognition Lab under Dr. Michelene T. H. Chi's
supervision in January 2018. During her time in the lab, she has focused on the ICAP Theory--a theory
that defines active learning and aligns students' activities to four predicted learning levels. She
graduated Summa Cum Laude from Barrett, The Honors College at Arizona State University in May 2021
with a double major in Psychology (B.S) and Political Science (B.S.) with a minor in Philosophy.
Tamara Brenner is Executive Director of the Derek Bok Center for Teaching and Learning at Harvard
University, where she provides leadership for the Center’s work in enhancing teaching and learning. She
received a PhD in biochemistry and molecular biology from UCSF, and has taught an introductory life
sciences course at Harvard.
521
Judy Brooks is the Director of EdTech & Design at the Eberly Center (Carnegie Mellon University). Judy
directs a diverse team of Learning Engineers, Software Engineers, Multimedia Designers, and
Technology Consultants who support CMU’s Technology-Enhanced Learning (TEL) initiatives. Judy has a
long history of designing online learning environments and dashboards, and works with instructors to
develop evidence-based and data-informed TEL tools and resources.
Carolyn Brown-Kramer is an associate professor of practice at the University of Nebraska-Lincoln. She
primarily teaches courses in introductory psychology, social psychology, and motivation and emotion.
Her research interests include improving student learning and engagement in the classroom,
strengthening class discussion, and effective mentorship of undergraduate learning assistants (LAs) to
facilitate student learning and professional development.
Dr. Andrew C. Butler is chair and associate professor in the Department of Education at Washington
University in St. Louis, and associate professor in the Department of Psychological & Brain Sciences. He
earned a Ph.D. in cognitive psychology at Washington University and completed a postdoctoral
fellowship at Duke University.
Jennifer Calawa is a Ph.D. candidate at the University of New Hampshire. She has worked on the
Student Cognition Toolbox as a Research Associate for the UNH Center for Excellence and Innovation in
Teaching and Learning for the past three years. Her research interests are in improving utility value and
interest in STEM in non-STEM major undergraduate students.
Shana Carpenter is a cognitive psychologist at Iowa State University. She has interests in strategies to
promote learning and metacognition, and how this research can be applied toward improving
education.
Dr. Paulo Carvalho is a research scientist in the Human-Computer Interaction Institute at Carnegie
Mellon University. His research focuses on understanding how we acquire knowledge through repeated
practice and how we can improve students’ learning and outcomes in K-12 and Higher Education,
especially in self-regulated environments. He earned his Ph.D. in Psychological and Brain Sciences from
Indiana University in 2016.
Stephen L. Chew is a professor of psychology at Samford University in Birmingham, Alabama. Trained as
a cognitive psychologist, he studies the cognitive basis of effective teaching and learning. He works both
as a researcher and as a translator of research into effective teaching practice. He has created popular
resources for both teachers and students on how people learn. In addition to studying teaching, he is
the recipient of multiple national teaching awards.
Dr. Michelene Chi works at the intersection of the learning sciences and educational practice. Her ICAP
theory provides concrete definitions for classifying learning activities into four hierarchical modes that
predict four levels of learning outcomes. Her PAIR-C framework explains the robustness of students’
misconceptions for science process-concepts. She is also implementing her work on the advantages of
viewing tutorial dialog videos compared to a lecture monolog video. For her interdisciplinary
contributions, she was the recipient of the 2019 Rumelhart Prize in Cognitive Science, the 2020 McGraw
Prize in Education, and the 2021 APS William James Fellow Award in Psychology.
Clark Chinn is a Professor at the Graduate School of Education at Rutgers University in the USA. His
research focuses on epistemic cognition, improving reasoning and argumentation, learning from
multiple documents, and collaborative learning. His most recent work has focused on how to promote
better thinking in our so-called “post-truth” world. https://gse.rutgers.edu/faculty/clark-a-chinn/
Anne M. Cleary, PhD holds the title of Professor of Cognitive Psychology at Colorado State University.
She specializes in the study of human memory processes and metacognition, and developed an all-
522
university course on applying the science of learning to effective study strategies and habits for which
she and Matthew Rhodes co-authored a textbook, A Guide to Effective Studying and Learning: Practical
Strategies from the Science of Learning.
Dr. James Cooke is an Assistant Teaching Professor in the Department of Neurobiology at the University
of California, San Diego. His research group is focused largely on assessments. In particular, they are
interested in how students engage with open-ended test questions; how test timing affects student
outcomes; and whether collaborative exams. Following a PhD in Pharmacology at the University of
British Columbia, Dr. Cooke did a postdoctoral research fellowship at the Hospital for Sick Children in
Toronto, On.
Lauren Davidson is a frontend developer at Cast Iron Coding. Prior to this role, she worked as Assistant
Director of the Learning Lab at the Derek Bok Center for Teaching and Learning at Harvard University,
where she led the Media and Design Fellows Program, which positions graduate students to develop
multimedia assignments and learning objects for their departments. Lauren holds a PhD in Philosophy
from Harvard.
Anique de Bruin is a professor in Self-regulation in Higher Education at the School of Health Professions
Education, Maastricht University, the Netherlands. She examines how learners monitor and regulate
their learning of texts, problems, and clinical reasoning, in both classroom and workplace settings; and
how instructional design can support this. She is also project chair of the Study Smart team at
Maastricht University.
Jennifer L. Docktor is a Professor of Physics at the University of Wisconsin – La Crosse. Her research
focuses on the areas of physics learning and cognition and physics teacher preparation. She recently co-
authored a book with José Mestre on The Science of Learning Physics: Cognitive Strategies for Improving
Instruction. https://www.uwlax.edu/profile/jdocktor/
Dr. John Dunlosky is a Professor in the Department of Psychological Sciences and the Director of the
Science of Learning and Education Center at Kent State University. His research focuses on discovering
how to promote mastery learning and student achievement.
John E. Edlund is a Professor of Psychology at the Rochester Institute of Technology. He serves as the
Research Director of Psi Chi: The International Honor Society in Psychology. He serves as an Executive
Editor at the Journal of Social Psychology and an associate editor of Psychological Reports as well as
Collabra: Psychology. He received his doctorate from Northern Illinois University. He has won numerous
awards related to teaching and mentoring and is passionate about the dissemination of scientific
knowledge.
Laura C. Edwards, assistant professor of Psychology at Taylor University, has taught undergraduate and
graduate levels in the United States and Brazil. Her research interests lie in the SoTL, educational
psychology, and developmental issues. Edwards has authored peer-reviewed articles on educational
psychology, adolescent and student development, and the collaboration between clergy and
psychologists. She is the recipient of the 2020 Dr. Joe Burnworth Teaching Award.
Gil Einstein was the William R. Kenan, Jr., Professor of Psychology at Furman University until his
retirement in 2019. He studies learning and memory. He has received several teaching and mentoring
awards, and he has a long-standing commitment to student learning and, in particular, in how to
enhance students’ study strategies to prepare them for life-long learning.
Roberta Ekuni, Ph.D. is a Professor at the Graduate Program in Education at the Universidade Estadual
do Norte do Paraná, in Brazil. She has a keen interest in the science of learning. Her research focus on
retrieval practice and neuromyths in education.
523
Tino Endres is interested in the mechanisms of meaningful learning processes that can be applied in the
daily lives of instructors and learners. Following that interest, he earned a Ph.D. in Psychologies in 2020.
Right now, he is a postdoc at the University of Freiburg (lab of Prof. Alexander Renkl) and an associate
member of the University of Iowa (lab of Prof. Shana Carpenter). His research interests include self-
regulated learning, retrieval practice, and emotional design. All these areas of research were combined
to optimize the training that is introduced in this chapter, which hopefully inspires to use of the
provided tools.
Tian Fan is a Ph.D. candidate in the Collaborative Innovation Center of Assessment for Basic Education
Quality, Beijing Normal University, No. 19, Xinjiekouwai Street, Haidian District, Beijing 100875, China;
email: [email protected]. Her research interests lie in self-regulated learning, metamemory
monitoring, and the testing effect.
Dr. Barbara Fenesi is an Assistant Professor in the Faculty of Education at Western University, and a
Canada Research Chair in the Science of Learning. Her passion is rooted in understanding how the entire
human organism (both the brain and the body) plays a role in student success. She is currently
examining how physical activity can promote attention and learning in children with ADHD.
Logan Fiorella is an Associate Professor of Educational Psychology at the University of Georgia. Dr.
Fiorella’s interests are in the cognitive science of learning and its implications for instruction. He
received the Outstanding Early Career Scholar Award from Division C of the American Educational
Research Association, the Richard E. Snow Award for Early Contributions in Educational Psychology from
Division 15 of the American Psychological Association, and a National Academy of Education/Spencer
Postdoctoral Fellowship. He is co-author of the book Learning as a Generative Activity and co-editor of
The Cambridge Handbook of Multimedia Learning: 3rd edition.
Regina (Gina) F. Frey is the Ragsdale Professor of Chemical Education in chemistry at the University of
Utah. Her focus is on education research in introductory STEM courses and faculty development. Gina
studies the effectiveness of collaborative pedagogies and approaches to improve social belonging and
inclusivity for all students in STEM. url: https://chem.utah.edu/directory/frey/research-group/
Robert Goldstone is Distinguished Professor in the Psychological and Brain Sciences department and
Cognitive Science program at Indiana University. His research interests include concept learning and
representation, perceptual learning, educational applications of cognitive science, decision making,
collective behavior, and computational modeling of human cognition. He won the 2000 APA
Distinguished Scientific Award for Early Career Contribution to Psychology, and a 2004 Troland research
award from the National Academy of Sciences. He was the executive editor of Cognitive Science from
2001-2005, Director of the Indiana University Cognitive Science Program from 2006-2011, and is current
executive editor of Current Directions in Psychological Science.
Maren Greve is a graduate student at Kent State University in the department of psychological sciences.
She is broadly interested in student achievement, individual differences, and the interaction between
the two.
John Grigg is Head of the Discipline of Ophthalmology at the University of Sydney and Consultant
Ophthalmologist Sydney Eye Hospital and The Children’s Hospital Westmead. The Discipline of
Ophthalmology oversees all ophthalmic teaching and research at the University of Sydney including The
Save Sight Institute.
Dr. Lou Ann Griswold, a university educator for over 30 years, continues to hone her teaching skills. She
is currently Associate Professor and Department Chair of the Department of Occupational Therapy at
the University of New Hampshire.
524
Dr. Regan Gurung is Associate Vice Provost and Executive Director of the Center for Teaching and
Learning, and Professor of Psychological Science at Oregon State University. His research focuses on the
science of teaching with a focus on the introductory psychology class, personality characteristics
influencing learning, and student mental health. He is winner of the Charles L. Brewer Award for
Distinguished teaching in psychology and is the author, co-author, edition, co-editor of 15 books and
over 130 peer reviewed articles and book chapters. He has served as President of the Society for the
Teaching of Psychology and Psi Chi, the International Honor Society for Psychology.
Marissa Hartwig received her PhD in experimental psychology from Kent State University. Her research
focuses on student learning strategies and metacognition.
John Hattie is Emeritus Laureate Professor at the Melbourne Graduate School of Education at the
University of Melbourne, Chair of the Australian Institute of Teaching and School Leaders, and director
of the Hattie Family Foundation. His Visible Learning research is based on ¼ billion students and he
continues to update this research. He has published and presented over 1000 papers, and supervised
200 theses students, and 60 books – including 24 on Visible Learning.
Dr. Jennifer J. Heisz is a Canada Research Chair in Brain Health and Aging, Associate Professor in the
Department of Kinesiology at McMaster University, Director of the NeuroFit Lab (www.neurofitlab.com)
and author of Move the Body, Heal the Mind. Dr. Heisz's research examines the effects of physical
activity on brain function to promote mental health and cognition in young adults, older adults and
individuals with Alzheimer’s disease. Recent honours include receiving an Early Researcher Award from
the Government of Ontario and the Petro-Canada Young Innovator Award. Fellow her on twitter
@jenniferheisz
Chad Hershock, Ph.D., is the Director of Faculty & Graduate Student Programs at the Eberly Center for
Teaching Excellence & Educational Innovation (Carnegie Mellon University). Chad leads teams of
teaching and assessment consultants who support instructors through data-driven consultation services
and evidence-based educational development programming. Leveraging service-based, action research,
he also contributes to the Scholarship of Educational Development and Scholarship of Teaching.
Slava Kalyuga is Emeritus Professor of Educational Psychology at the School of Education, the University
of New South Wales. His research interests are in cognitive processes in learning, cognitive load theory,
and evidence-based instructional design principles. His specific contributions include detailed
experimental studies of the role of learner prior knowledge in learning (expertise reversal effect); the
redundancy effect in multimedia learning; the development of rapid online diagnostic assessment
methods; and studies of the effectiveness of different adaptive procedures for tailoring instruction to
levels of learner expertise. He is the author of four books and near 200 research articles and chapters.
Dr. Althea Need Kaminske (Twitter: @drsilverfox) is an Associate Professor in Psychology at St.
Bonaventure University where she is the Chair of the Psychology Department and Director of the
Behavioral Neuroscience Program. She is passionate about bridging the gap between research and
practice in education.
Sean Kang is a Senior Lecturer in the Science of Learning at the University of Melbourne Graduate
School of Education. His research focuses on applying the cognitive science of learning and memory
towards improving instruction and self-regulated learning.
Manu Kapur is currently a Professor and Chair of Learning Sciences and Higher Education at ETH Zurich,
Switzerland. Before this, he worked in Singapore and Hong Kong, heading learning sciences labs and
initiatives. Manu is widely known mainly for his research on learning from productive failure. For more
information, visit www.manukapur.com
525
Quentin King-Shepard is a Ph.D. student in the department of psychology and Learning, Research, and
Development Center at the University of Pittsburgh. His research interests include learning, problem
solving, and decision-making in cognitively demanding tasks.
Kenneth R. Koedinger is the Hillman Professor of Computer Science at Carnegie Mellon. He investigates
how people learn and provides learning engineering techniques and technology tools through his
leadership of LearnLab.org (http://learnlab.org) and LearnSphere.org (http://learnsphere.org/) and in
directing the Masters of Educational Technology and Applied Learning Sciences (METALS;
https://metals.hcii.cmu.edu/).
Jordan Koffman, Assistant Director of the Bok Center’s Learning Lab at Harvard University, works
primarily with undergraduates in running workshops and hackathons on multimodal academic
communication. She received a Masters in Arts in Education from Harvard and previously taught art and
science integration while simultaneously managing a cognitive psychology laboratory in Honolulu.
Dr. Carolina Kuepper-Tetzel (Twitter: @pimpmymemory) is a Senior Lecturer (Associate Professor) in
Psychology at the University of Glasgow, Scotland. She is an expert in applying findings from Cognitive
Psychology to education and an enthusiastic science communicator. She leads the interdisciplinary TILE
Network (https://tile.psy.gla.ac.uk/) that connects educational sectors through a speaker series.
Marlon Kuzmick is Director of the Learning Lab at the Derek Bok Center for Teaching and Learning at
Harvard University. Before joining the Learning Lab he served as Head Preceptor of Expository Writing in
the Harvard College Writing Program, and his work at the Bok Center has involved extending the
pedagogical practices of the Writing Program into new media.
Scott E. Lewis is an Associate Professor in chemistry at the University of South Florida. His research is in
chemistry education with a focus on evaluating pedagogical reforms, developing student assessments
and exploring the impact of representations on student success.
Jeri L. Little is an Assistant Professor of Psychology at California State University, East Bay. She earned
her Ph.D. in Psychology from University of California, Los Angeles. Her research focuses on educational
applications of cognitive psychology. Find out more at www.littlelearninglab.com .
Marsha C. Lovett is Senior Associate Vice Provost for Educational Innovation and Learning Analytics,
Director of the Eberly Center, and Teaching Professor of Psychology, at Carnegie Mellon University. She
studies learning in the laboratory and in real courses, develops data-informed educational technology
tools, and is committed to leveraging learning research methods and results to improve student
outcomes.
Liang Luo is a professor both in the Institute of Developmental Psychology and Collaborative Innovation
Center of Assessment toward Basic Education Quality, Beijing Normal University, No.19, Xinjiekouwai
Street, Haidian District, Beijing, 100875; email: [email protected]. His research focuses on the
influence factors of children’s learning ability and mental health. Besides, his research also explores the
potential internal mechanisms, such as metacognition, family SES, and learning strategies.
Emmanuel Manalo, PhD, is a Professor at the Graduate School of Education of Kyoto University in Japan.
His research focus is on the promotion of effective learning and instructional strategies, including the
educational uses of self-constructed diagrams, the linking of different dimensions/strands of learning,
and the cultivation of thinking competencies. He is Co-Editor-in-Chief of the journal, Thinking Skills and
Creativity.
Richard Mayer is Distinguished Professor of Psychology at the University of California, Santa Barbara.
His current research interests include multimedia learning, computer-supported learning, and computer
526
games for learning. His research is at the intersection of cognition, instruction, and technology, with a
focus on how to help people learn in ways so they can transfer what they have learned to new
situations. He is the winner of the Thorndike Award for career achievement in educational psychology,
the Scribner Award for outstanding research in learning and instruction, and the American Psychological
Association’s Distinguished Contribution of Applications of Psychology to Education and Training Award.
Jennifer A. McCabe is a psychology professor at Goucher College. Her research focuses on memory
strategies and metacognition in undergraduates. She has published multiple empirical articles on these
topics, as well as two teaching resources funded by STP. She is currently an Associate Editor for Teaching
of Psychology. https://www.goucher.edu/learn/academic-centers/psychology/faculty/jennifer-mccabe
Mark McDaniel is a Professor of Psychological and Brain Sciences and the Director of the Center for
Integrative Research on Cognition, Learning, and Education at Washington University. He has published
numerous papers related to cognition and education. To convey research findings pertinent to
education, he co-authored Make it Stick: The Science of Successful Learning.
Elizabeth A. McLaughlin is a senior researcher at Carnegie Mellon University whose interests lie within
educational research and learning sciences. She works on projects that investigate methods for
improving learning and pedagogy.
Dr. Michael Melville works as a Senior Data Science Research Associate for the Eberly Center at
Carnegie Mellon University, where he consults with faculty, graduate students, and post-docs to design,
implement, and assess research projects that relate to teaching and learning in their classrooms. He also
contributes to a variety of program-level assessment work on the CMU campus. Mike’s training includes
a PhD and M.A. in Social Psychology from the University of New Hampshire, and an M.Ed. in sport and
performance psychology from Springfield College. Mike previously worked at the University of New
Hampshire Center for Excellence and Innovation in Teaching and Learning from 2014-2017.
José P. Mestre is Professor Emeritus of Physics and Educational Psychology at the University of Illinois.
His research applies cognitive science methodologies to study expertise, problem solving and
instructional design. He has offered Congressional testimony and has published numerous articles and
books. He is a Fellow of the American Physical Society.
https://physics.illinois.edu/people/directory/profile/mestre
Janet Metcalfe, PhD, is a Professor of Psychology and of Neurobiology and Behavior at Columbia
University. Her research focuses on both behavioral and brain imaging studies concerned with how
people—children, young adults, older adults, and people with schizophrenia and Aspergers' syndrome—
know what they know and use that metaknowledge to control their behavior. She has a particular
interest in curiosity and how to provoke it. Her work addresses hot and cool emotional self-regulation,
the impact of error generation and correction on learning, the role of future projection in self-
regulation, and strategies of learning. The matter of inquiry that underpins all of Dr. Metcalfe's work is
to understand the mechanisms underlying how people are able to willfully control their own thoughts,
feelings, actions, and learning.
Dr. Cynthia Nebel (Twitter: @PsyDocCindy) is a Senior Lecturer in the Leadership & Organizations
Program at Vanderbilt University, and serves as Capstone Director, helping students to complete
doctoral work with organizational partners. She has broad interests in human learning and memory and
applying cognitive concepts to improve education. She is dedicated to bi-directional communication
among researchers and educators.
Nora S. Newcombe is Professor and James H. Glackin Fellow at Temple University. Her Ph.D. is from
Harvard University. Her research focuses on spatial development and the development of episodic
527
memory. Dr. Newcombe is the author of numerous chapters, articles, and books, including Making
Space (with Janellen Huttenlocher). Her work has been recognized by awards including the George A.
Miller Award and the G. Stanley Hall Award. She has served as Editor of the Journal of Experimental
Psychology: General and Associate Editor of Psychological Bulletin. She was PI of the NSF-funded Spatial
Intelligence and Learning Center 2006-18.
Timothy Nokes-Malach is a cognitive psychologist in the department of psychology and Learning,
Research, and Development Center at the University of Pittsburgh. He has interests in learning, problem
solving, and motivation, and how these contribute to promoting transfer of learning.
Michelle Ogrodnik is a PhD Candidate in the Department of Kinesiology at McMaster, where she also
works as a Teaching Assistant and Sessional Instructor. Under the supervision of Dr. Jennifer Heisz in the
NeuroFit Lab, she studies how exercise can benefit attention and memory. Her work is funded by a
SSHRC Joseph-Armand Bombardier Canada Graduate Scholarship. Alongside her research, Michelle
worked at the MacPherson Institute for Leadership, Innovation and Excellence in Learning for over five
years as a Lead Educational Development Fellow.
Steven C. Pan is an Assistant Professor of Psychology at the National University of Singapore. He
received a PhD in Psychology from the University of California, San Diego. His research investigates the
optimization of human learning and memory processes in educational contexts. Further information is
available at www.stevencpan.com.
Sabine Pompeia, Ph.D., is currently a Professor at the Department of Psychobiology at the Universidade
Federal de São Paulo, in Brazil. She has conducted research on the biological bases of human cognitive
abilities for over 30 years and also has a keen interest in the science of learning.
Danney Rasco is an assistant dean for the School of Graduate Studies at Salem State University. His
research focuses on the interpersonal processes (e.g., disclosure, relationship authenticity) that
influence romantic and classmate relationships and their outcomes (e.g., subjective well-being, student
retention).
Martina A. Rau is an associate professor in Educational Psychology at the University of Wisconsin -
Madison. Her research focuses on learning with visual representations. She is interested in interactions
among perceptual and conceptual learning processes. Her work has yielded instructional interventions
that help students use visual representations to learn STEM content.
https://website.education.wisc.edu/rau-lab/
Dr. Katherine Rawson is a Professor in the Department of Psychological Sciences at Kent State
University. Her research focuses on identifying techniques that promote durable and efficient student
learning and on investigating how students use these techniques during self-regulated learning.
Alexander Renkl finished his diploma degree in psychology in 1987 (RWTH Aachen, Germany). In 1991,
he received his Ph.D. (University of Heidelberg). As Assistant Professor, he spent several years (1991 to
1997) at the University of Munich. In 1997, he became a Professor of Educational Psychology at the
University of Education in Schwäbisch Gmünd. Since 1999, he has been working at the University of
Freiburg as Professor of Educational and Developmental Psychology. His main research areas are
example-based learning, instructional explanations and self-explanations, learning from multiple
representations, learning strategies, learning by journal writing, retrieval practice, and teachers'
knowledge.
Matthew Rhodes is a Professor of Psychology at Colorado State University. His research focuses on
memory, metacognition, and evidence-based approaches to learning. He is a Fellow of the American
Psychological Association (Division 3) and the Association for Psychological Science and has received
528
several awards for teaching and student mentoring. Dr. Rhodes is currently an Associate Editor at the
Journal of Experimental Psychology: Learning, Memory and Cognition and is also an author, with Anne
Cleary and Edward DeLosh, of a recent book on learning, A Guide to Effective Studying and Learning:
Practical Strategies from the Science of Learning (2020; Oxford University Press).
Doug Rohrer is a professor of psychology at the University of South Florida. He received his PhD in
psychology from the University of California, San Diego.
Ido Roll is an Associate Professor and the head of the Learning Analytics, Learning Design lab at the
Faculty of Education in Science and Technology at the Technion - Israel Institute of Technology. Ido
studies how interactive learning environments can support students in becoming better learners and
scientists, focusing on the development of scientific literacies, creativity, self-regulated learning, and
sense making. His research is rooted in frameworks and methodologies from education, cognitive
science, learning analytics, and human-computer interaction. More can be found on his website,
https://roll.net.technion.ac.il
Luke Rowe is a learning scientist and educator at Australian Catholic University’s (ACU) National School
of Education, Melbourne, Australia. Before embarking on his career at ACU, Luke was the human
research coordinator for the Hunt Lab at the school of psychological sciences and lectured on the topic
of the brain and the science of learning at the University of Melbourne. He holds a Ph.D. on the topic of
collective intelligence, which was based on his experimental research in the Science of Learning
Research Centre laboratory under the supervision of Professors John Hattie, Robert Hester, and John
Munro. Luke’s most recent research investigates how educators access, interpret, and apply evidence to
improve teaching and learning. He has broader interests in the science of learning, evidence-based
teaching, group performance, and cognitive science.
Ralf Rummer is a professor of cognitive psychology at the University of Kassel where he works since
2018. From 2008 to 2018, he was professor of cognitive and instructional psychology at the University of
Erfurt. His research interests are in the field of applied cognitive psychology (e.g., learning with
multimedia, desirable difficulties, metacognition) as well as in the field of basic cognitive research,
particularly concerning psychology of language (e.g., sound symbolism, sentence processing) and
memory (e.g., working memory, retrieval practice).
Dr. Faria Sana is an Associate Professor in Psychology at Athabasca University. She received her PhD in
Cognitive Psychology from McMaster University, and completed an Izaak Walton Killam Fellowship at
the University of Alberta. She is the director of the Cognitive Science of Learning Lab (cogscilearn.ca) and
the co-director of the EdCog Research Lab (edcog.ca). Dr. Sana’s research program is funded by the
Social Sciences and Humanities Research Council. Her research interests include evidence-based
pedagogy and cognitive science of learning.
Julie Schell is an award-winning expert in college teaching and learning. She is the Assistant Dean for
Instructional continuity and Innovation in the College of Fine Arts at The University of Texas at Austin.
She is also an Assistant Professor of Practice in the Department of Design and the Program in Higher
Education Leadership at UT. Julie was a post-doctoral fellow in the Mazur Group at Harvard University,
where she continues to serve as a research affiliate focused on Peer Instruction. She holds a Doctorate
in Higher and Postsecondary Education from Teachers College, Columbia University.
Judith Schweppe is a professor for psychology with a focus on digital learning at the University of Passau
in Germany. Before coming to Passau in 2020 she worked as a research scientist at Saarland University,
where she received her doctoral degree in cognitive psychology, and at the University of Erfurt. Her
research focuses on basic research on working memory and long-term memory and how it can be
529
applied to learning and teaching (e.g., with respect to multimedia learning and incorporating desirable
difficulties).
David R. Shanks is a cognitive psychologist and Deputy Dean of the Faculty of Brain Sciences at
University College London, England. His research aims to understand the basic mechanisms of learning,
memory and decision making and apply this knowledge in educational and other real-world settings. He
has developed many computational and mathematical models of basic psychological processes.
Together with Ben R. Newell and David A. Lagnado, he is author of Straight Choices: The Psychology of
Decision Making (3rd edition, 2022).
Dr. Tony Succar earned his Master of Science in Medicine (Ophthalmic Science) from The University of
Sydney, involving the development of a bionic eye for people with severe vision impairment. He was
subsequently awarded The University of Sydney Postgraduate Scholarship in Vision and Eye Health
Research, where he completed his PhD in the development of a Virtual Ophthalmology Clinic (VOC) on
which medical students can sharpen their clinical reasoning skills by formulating a diagnosis and
treatment plan on virtual patients with simulated eye conditions. Dr. Succar was appointed as a
Postdoctoral Scholar-Research Associate at the University of Southern California, Keck Medicine of USC,
and is currently a Clinical Research Project Manager at Massachusetts Eye and Ear, Harvard Medical
School.
Dr. Megan Sumeracki (Twitter: @DrSumeracki) is an Associate Professor of Cognitive Psychology at
Rhode Island College. Her research focuses on retrieval-based learning strategies and improving the
adoption of evidence-based practices in the classroom. In an effort to promote more conversations
between researchers and practitioners, she co-founded The Learning Scientists
(www.learningscientists.org).
John Sweller is an Emeritus Professor of Education Psychology at the University of New South Wales. His
research is associated with cognitive load theory. The theory is a contributor to both research and
debate on issues associated with human cognition, its links to evolution by natural selection, and the
instructional design consequences that follow. Based on many hundreds of randomized, controlled
studies carried out by many investigators from around the globe, the theory has generated a large range
of novel instructional designs from our knowledge of human cognitive architecture.
Elizabeth Tappin is the Student Learning Outcomes Assessment Coordinator for the UNH Center for
Excellence and Innovation in Teaching and Learning. Her work supports faculty and staff in assessing
student learning outcomes in courses, academic departments, and programs.
Jonathan Tullis studies how to structure education to improve student learning as an associate
professor of Educational Psychology at the University of Arizona. Jonathan earned his M.Ed. from Notre
Dame and taught high school physics and chemistry before earning his Ph.D. in cognitive psychology
from the University of Illinois.
Yuri Uesaka, PhD, is an Associate Professor at the University of Tokyo, Japan. Her interest is in using
psychological approaches to develop effective instructional environments for enhancing the quality of
student learning in Japan. She is also actively engaged in international collaborative research. She is a
former member of the research committee of the Japanese Association of Educational Psychology.
Dr. Kathryn Wissman is an Assistant Professor in the Department of Psychology and a core faculty
member in the Discipline-Based Education Research program at North Dakota State University. Her
research explores how to support student success and inform pedagogical instruction by examining best
learning practices for inside and outside of the classroom.
530
Dr. Veronica X Yan (PhD, University of California, Los Angeles) is an assistant professor of Educational
Psychology at the University of Texas at Austin where she directs the Science of Learning and
Metacognition lab. Her research bridges cognitive, social, and educational psychology fields to explore
how people can be empowered to become motivated and effectively self-regulated learners. In
particular, her work examines the mechanisms underlying 'desirably difficult’ strategies, their
interaction with individual differences, and learners’ motivational mindsets and metacognitive beliefs.
Chunliang Yang is an associate professor in the Institute of Developmental Psychology, Faculty of
Psychology at Beijing Normal University, No. 19, Xinjiekouwai Street, Haidian District, Beijing 100875,
China; email: [email protected]. His research interests lie in test-enhanced learning, spacing
and interleaving, comprehension and metacomprehension, metamemory monitoring and control,
development of meta-awareness, and translating principles from cognitive sciences into educational
practice.
Wenbo Zhao is a Ph.D. candidate in the Collaborative Innovation Center of Assessment Toward Basic
Education Quality, Beijing Normal University, No. 19, Xinjiekouwai Street, Haidian District, Beijing
100875, China; email: [email protected]. Her research interests lie in metamemory monitoring
and control processes, and the reactivity effect of making judgments of learning on memory itself.
531

In Their Own Words

Uploaded by

Copyright:

Available Formats

In Their Own Words

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

In Their Own Words

Uploaded by

Copyright:

Available Formats

In Their Own Words

What Scholars and Teachers Want

Division 2, American Psychological Association 2023

Copyright and Other Legal Notices

Suggested Reference Format

Individual chapters should be referenced in this fashion (an example):

Constructing a Canon for the Science of Learning

Part 2: Science of Learning: Principles and Approaches

Part 3: Preparing Faculty, Educational Developers, Student Success Professionals, and

Part 4: Preparing Students to Apply the Science of Learning

Part 5: Putting the Science of Learning into Practice

About the Authors 521

What Scholars and Teachers Want

Division 2, American Psychological Association 2023

Suggestions for Reading the Book

Other Resources That May be Useful to Teachers

Division 2, American Psychological Association 2023

Division 2, American Psychological Association 2023

The Instructional Toolbox

Division 2, American Psychological Association 2023

Increasingly, learning is happening outside of classroom settings, and computer-based technologies

Learning Versus Performance; Research Versus Intuition

Obstacles to Introducing Desirable Difficulties

Opportunities, Amid the Obstacles, to Introduce Desirable Difficulties

The Important Role of Public-Audience Books and Websites

Division 2, American Psychological Association 2023

The Culture of Teaching We Have

The Appeal of the Information-Driven Approach

The Culture of Teaching We Need

What the Best Teachers Know

The Cognitive Challenges of Effective Teaching

An Expanded View of Student Learning

An Expanded View of Successful Teaching

An Expanded Role on Teacher Preparation and Professional

A Contextual Approach for Conducting, Organizing, and Translating

What Scholars and Teachers Want

Research Is Fast, but Uptake by Teachers Is Slow – Why?

Improving Receptivity Toward the Use of Evidence From the Learning

Two Major Challenges, Seven Minor Challenges

The Origins and Evolution of SoTL

Changes in Perceptions of SoTL

The Future of SoTL

Division 2, American Psychological Association 2023

Introduction and Overview

“Pretrieval Practice”: The Benefits of Prequestions

Why Do Prequestions Benefit Learning?

What Kind of Benefits Can Be Expected from Prequestions?

Recommendations for Teachers

What is Successive Relearning?

The Promise of Successive ReLearning: Experimental Demonstrations

Barriers to Using Successive Relearning Effectively and

Some Frequently Asked Questions

Division 2, American Psychological Association 2023

ICAP’s Main Hypothesis

How to Use ICAP to Design Activities and Exercises, Choose Strategies

Passive Verbs Active Verbs Constructive Verbs Interactive Verbs

How to Select Which Strategies to Use

Passive Active Constructive Interactive

How to Ask Questions in the Classroom

Examples of Upgrades from Post-secondary and K-12 Classes