Smart Computing and Intelligence

Series Editors: Kinshuk · Ronghuai Huang · Chris Dede

Dejian Liu
Chris Dede
Ronghuai Huang
John Richards Editors

Virtual,
Augmented,
and Mixed
Realities
in Education
Smart Computing and Intelligence

Series editors
Kinshuk, Athabasca, Canada
Ronghuai Huang, Beijing, China
Chris Dede, Cambridge, USA
This book series aims to establish itself as a medium for the publication of new
research and development of innovative paradigms, models, architectures, concep-
tual underpinnings and practical implementations encompassed within smart
computing and intelligence.
The scope of the series includes but is not limited to smart city, smart education,
health informatics, smart ecology, data and computational analytics, smart society,
smart learning, complex systems-chaos, computational thinking, brain computer
interaction, natural/computer interaction, humanoid behaviour, and impact of
educational psychology on computing.
The cornerstone of this series’ editorial policy is its unwavering commitment to
report the latest results from all areas of smart computing and intelligence research,
development, and practice. Our mission is to serve the global smart computing and
intelligence community by providing a most valuable publication service.

More information about this series at http://www.springer.com/series/15620

Dejian Liu Chris Dede Ronghuai Huang
• •

John Richards
Editors

Virtual, Augmented,
and Mixed Realities
in Education

123
Editors
Dejian Liu Ronghuai Huang
Smart Learning Institute Smart Learning Institute
Beijing Normal University Beijing Normal University
Beijing Beijing
China China

Chris Dede John Richards

Harvard Graduate School of Education Harvard Graduate School of Education
Harvard University Cambridge, MA
Cambridge, MA USA
USA

ISSN 2522-0888 ISSN 2522-0896 (electronic)

Smart Computing and Intelligence
ISBN 978-981-10-5489-1 ISBN 978-981-10-5490-7 (eBook)
DOI 10.1007/978-981-10-5490-7
Library of Congress Control Number: 2017947472

© Springer Nature Singapore Pte Ltd. 2017

This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part
of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission
or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar
methodology now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publication does not imply, even in the absence of a specific statement, that such names are exempt from
the relevant protective laws and regulations and therefore free for general use.
The publisher, the authors and the editors are safe to assume that the advice and information in this
book are believed to be true and accurate at the date of publication. Neither the publisher nor the
authors or the editors give a warranty, express or implied, with respect to the material contained herein or
for any errors or omissions that may have been made. The publisher remains neutral with regard to
jurisdictional claims in published maps and institutional affiliations.

Printed on acid-free paper

This Springer imprint is published by Springer Nature

The registered company is Springer Nature Singapore Pte Ltd.
The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721, Singapore
Contents

1 Introduction: Virtual, Augmented, and Mixed Realities in

Education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Christopher J. Dede, Jeffrey Jacobson and John Richards

Part I Frameworks for the Design and Implementation of Immersive

Learning
2 Implicit Learning Through Embodiment in Immersive Virtual
Reality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Mel Slater
3 Authenticity in Immersive Design for Education . . . . . . . . . . . . . . . 35
Jeffrey Jacobson
4 The Immersive Power of Social Interaction. . . . . . . . . . . . . . . . . . . . 55
Nicole C. Krämer
5 Assessment for Learning in Immersive Environments . . . . . . . . . . . 71
Valerie Shute, Seyedahmad Rahimi and Benjamin Emihovich
6 Infrastructures for Immersive Media in the Classroom . . . . . . . . . . 89
John Richards
7 The Potentials and Trends of Virtual Reality in Education . . . . . . . 105
Dejian Liu, Kaushal Kumar Bhagat, Yuan Gao, Ting-Wen Chang
and Ronghuai Huang

Part II Case Studies of Immersive Learning

8 Virtual Reality as an Immersive Medium for Authentic
Simulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
Chris Dede, Tina A. Grotzer, Amy Kamarainen and Shari J. Metcalf

v
vi Contents

9 Systems to Support Co-creative Collaboration in Mixed-Reality

Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
Michael Robert Gardner and Warren W. Sheaffer
10 Massively Multiplayer Online Roleplaying Games and Virtual
Reality Combine for Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
Eric Klopfer
11 Embodied Education in Mixed and Mediated Realties . . . . . . . . . . . 193
Mina C. Johnson-Glenberg
12 Preparing Students for Future Learning with Mixed Reality
Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
Bertrand Schneider
13 Conclusion—Strategic Planning for R&D on
Immersive Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
Christopher J. Dede and John Richards
Glossary of Terms Related to Immersive Learning . . . . . . . . . . . . . . . . . 245
Contributors

Ting-Wen Chang Smart Learning Institute, Beijing Normal University, Beijing,

China
Chris Dede Harvard Graduate School of Education, Harvard University,
Cambridge, MA, USA
Christopher J. Dede Harvard Graduate School of Education, Cambridge, MA,
USA
Benjamin Emihovich Educational Psychology and Learning Technologies,
Florida State University, Tallahassee, FL, USA
Yuan Gao Smart Learning Institute, Beijing Normal University, Beijing, China
Michael Robert Gardner School of Computer Science and Electronic
Engineering, The University of Essex, Colchester, Essex, UK
Tina A. Grotzer Harvard Graduate School of Education, Harvard University,
Cambridge, MA, USA
Ronghuai Huang Smart Learning Institute, Beijing Normal University, Beijing,
China
Jeffrey Jacobson CEO, EnterpriseVR, Jamaica Plain, MA, USA
Mina C. Johnson-Glenberg Department of Psychology, Arizona State
University, Tempe, AZ, USA
Amy Kamarainen Harvard Graduate School of Education, Harvard University,
Cambridge, MA, USA
Eric Klopfer Scheller Teacher Education Program, Massachusetts Institute of
Technology, Cambridge, MA, USA
Nicole C. Krämer Department for Computer Science and Applied Cognitive
Science, University Duisburg-Essen, Duisburg, Germany

vii
viii Contributors

Kaushal Kumar Bhagat Smart Learning Institute, Beijing Normal University,

Beijing, China
Dejian Liu Smart Learning Institute, Beijing Normal University, Beijing, China
Shari J. Metcalf Harvard Graduate School of Education, Harvard University,
Cambridge, MA, USA
Seyedahmad Rahimi Educational Psychology and Learning Technologies,
Florida State University, Tallahassee, FL, USA
John Richards Harvard University Graduate School of Education, Cambridge,
MA, USA; Consulting Services for Education, Inc., Newton, MA, USA
Bertrand Schneider Harvard University, Cambridge, MA, USA
Warren W. Sheaffer Department of Mathematics and Computer Science, Saint
Paul College, Saint Paul, MN, USA
Valerie Shute Educational Psychology and Learning Technologies, Florida State
University, Tallahassee, FL, USA
Mel Slater Institució Catalana de Recerca I Estudis Avançats (ICREA),
Barcelona, Spain; University College London, London, UK; University of
Barcelona, Barcelona, Spain
Chapter 1
Introduction: Virtual, Augmented,
and Mixed Realities in Education

Christopher J. Dede, Jeffrey Jacobson and John Richards

Keywords Virtual reality Augmented reality Mixed reality Augmented virtuality





Virtual environment VR AR MR VE Cyberspace Immersion







Presence Haptics Constructivism Situated learning Active learning




Constructionism Education Schools Museums Informal education




Conceptual change Adaptive response Metaverse 360 video HMD







CAVE Dome Cybersickness Sensory conflict Interactive Interactivity





Multiuser virtual environment MUVE Massively multiple online roleplaying game




MMORPG MMO Avatar Panoramic Oculus rift HTC vive


Google cardboard GearVR 3D

1.1 Origin of This Book

We live at a time of rapid advances in both the capabilities and the cost of virtual reality
(VR), multi-user virtual environments (MUVEs), and various forms of mixed reality
(e.g., augmented reality (AR), tangible interfaces). These new media potentially offer
extraordinary opportunities for enhancing both motivation and learning across a range

C.J. Dede (&)

Harvard Graduate School of Education, 336 Longfellow Hall, 13 Appian Way,
Cambridge, MA 02138, USA
e-mail: [email protected]
URL: https://www.gse.harvard.edu/faculty/christopher-dede
J. Jacobson
CEO, EnterpriseVR, 333 Lamartine St., Jamaica Plain, MA 02130, USA
e-mail: [email protected]
J. Richards
Harvard University Graduate School of Education, 13 Appian Way,
Cambridge, MA 02138, USA
e-mail: [email protected]
URL: http://www.cs4ed.com
J. Richards
Consulting Services for Education, Inc., 22 Floral St., Newton, MA 02461, USA

© Springer Nature Singapore Pte Ltd. 2017 1

D. Liu et al. (eds.), Virtual, Augmented, and Mixed Realities in Education,
Smart Computing and Intelligence, DOI 10.1007/978-981-10-5490-7_1
2 C.J. Dede et al.

of subject areas, student developmental levels, and educational settings. However,

past generations of learning technologies have seldom fulfilled the promise they
offered, because of shortfalls in orchestrating research, development, policy, practice,
and sustainable revenue to achieve transformational change in education.
With the sponsorship of NetDragon Websoft, a Chinese gaming and education
company, in January, 2017 the Immersive Learning Group at the Harvard Graduate
School of Education, and the Smart Learning Institute at Beijing Normal University
co-convened an invitational research workshop of research experts in immersive
learning. Its goal was to describe the leading edge of research in this field, as well as
to push forward the evolution of next-generation immersive learning experiences.
This volume is based on chapters these experts presented at that workshop and later
refined based on feedback from rich discussions among participants. Overall, the
goal of the workshop and book is to develop a strategic vision for educational VR
and immersive learning. This will include evaluations of long term potential and
opportunities, as well as current problems and barriers. Further, the ideas in the
book can inform a research and development agenda for the field. Achieving this
agenda will require understanding and surmounting issues with implementation, as
well as the development of testbeds at scale.
The discussion at the workshop was scholarly and sophisticated, since we are a
select group of researchers deeply familiar with this field. However, this book is
written to be accessible to a broad audience, since we want to reach a much wider
spectrum of stakeholders in immersive learning: teachers, administrators, scholars,
policy makers, instructional designers, evaluators, industry leaders.
Also, even experts are still refining their definitions of terms in this field. For that
reason, we include a glossary towards the end of this book that provides our
definitions as used to frame the workshop. However, the authors of each chapter
may use somewhat different definitions and will describe why in their discussions.

1.2 A Brief History of Immersive Media in Education

Virtual Reality (VR) was invented in the 60s or 70s with the flight simulators
developed by military aerospace, although they might be better described as Mixed
Reality (MR). [The next section of this chapter provides detailed definitions for all
these terms.] The advance of research on educational applications of VR has been
uneven, with empirical studies rare (Jacobson, 2008, pp. 62–75). VR was shown to be
very effective for learning procedural tasks, in which students learns a sequence of
steps to accomplish a task requiring maneuvers in three-dimensional space. Examples
include as operating a vehicle, fixing on a complex piece of machinery, and finding
your way around an otherwise unfamiliar landscape. The scientific literature on this is
vast, but it never found significant use in K-12 education, which tends to emphasize
declarative knowledge, primarily facts and concepts. Also, until 2015 equipment of
usable quality was unaffordable at scale in classroom settings.
In the early 2000s multi-user virtual environments (MUVEs) and augmented
realities (AR) came on the scene, and soon educational research established their
1 Introduction: Virtual, Augmented, and Mixed … 3

effectiveness for learning. Further, these technologies were affordable at scale. But
again they did not penetrate the K-12 market, for reasons discussed in Richards’
chapter.
Today, VR AR and MR are all flourishing in the consumer market. Google,
Samsung, SONY, and Facebook all have Head Mounted Devices (HMDs) and were
joined by a half dozen new devices at the 2017 Consumer Electronics Show.
Pokemon GO© had 100 million downloads from its launch on July 6, 2016 through
December, 2016 and has been earning over $10M per day on IOS and Google Play
combined. After twenty-five years of educational research, the consensus of the
participants at the conference was that the time has come for these new technologies
to have a substantial impact in education.

1.3 A Conceptual Framework for VR in Education

Learning experiences designed to teach complex knowledge and sophisticated skills

are often based on “guided social constructivist” theories of learning. In this
approach, learning involves mastering authentic tasks in personally relevant, real-
istic situations. Meaning is imposed by the individual rather than existing in the
world independently, so people construct new knowledge and understandings based
on what they already know and believe, which is shaped by their developmental
level, their prior experiences, and their sociocultural background and context
(Palincsar, 1998). Instruction can foster learning by providing rich, loosely struc-
tured experiences and guidance (such as apprenticeships, coaching, and mentoring)
that encourage meaning-making without imposing a fixed set of knowledge and
skills. This type of learning is usually social; students build personal interpretations
of reality based on experiences and interactions with others.
Immersive media have affordances that enhance this type of learning.
Psychological immersion is the mental state of being completely absorbed or
engaged with something. For example, a well-designed game in a MUVE draws
viewers into the world portrayed on the screen, and they feel caught up in that
virtual environment. The use of narrative and symbolism creates credible, engaging
situations (Dawley & Dede, 2013); each participant can influence what happens
through their actions and can interact with others. Via richer stimuli, head-mounted
or room-sized displays can create sensory immersion to deepen the effect of psy-
chological immersion, as well as induce virtual presence (place illusion), the feeling
that you are at a location in the virtual world (see Slater’s chapter for more details).
Three types of immersive interfaces underlie a growing number of formal and
informal learning experiences:
• Virtual Reality (VR) interfaces provide sensory immersion, at present focusing
on visual and audio stimuli with some haptic (touch) interfaces. The participant
can turn and move as they do in the real world, and the digital setting responds
to maintain the illusion of presence of one’s body in a simulated setting.
4 C.J. Dede et al.

• Multi-user Virtual Environment (MUVE) interfaces offer students an engaging

Alice-in-Wonderland experience, going “through the screen” to a simulated
setting in which their digital avatars convey psychological immersion in a
graphical, virtual context. The participant represented by the avatar feels remote
presence inside the virtual environment: the equivalent of diving rather than
riding in a glass-bottomed boat.
• Mixed Reality (MR) interfaces combine real and virtual settings in various ways,
to enable psychological immersion in a setting that blends physical and digital
phenomena. For example, an outdoor augmented reality (AR) experience using
mobile devices can superimpose information, simulations, and videos on a
through-the-camera-lens view of natural phenomena (Dunleavy & Dede, 2013).
The range of options is complex, because new sub-types of VR, MUVEs, and
MR are constantly emerging. Each has unique strengths and limits for aiding
learning, so understanding how to choose the right medium for a particular edu-
cational situation is an important next step in realizing the potential of immersive
media in learning. However, some aspects of immersive learning apply across all
these media.

1.3.1 How Immersive Presence Enhances Motivation

and Learning

Immersion in a mediated, simulated experience (VR, MUVE, or AR) involves the

willing suspension of disbelief. Inducing powerful immersion for learning depends
on designs that utilize actional, social, and symbolic/narrative factors, as well as
sensory stimuli (Dede, 2009):
• Actional Immersion: Empowering the participant in an experience to initiate
actions that have novel, intriguing consequences. For example, when a baby is
learning to walk, the degree of concentration this activity creates in the child is
extraordinary. Discovering new capabilities to shape one’s environment is
highly motivating and sharply focuses attention.
• Symbolic/Narrative Immersion: Triggering powerful semantic associations via
the content of an experience. As an illustration, reading a horror novel at
midnight in a strange house builds a mounting sense of terror, even though one’s
physical context is unchanging and rationally safe. Narrative is an important
motivational and intellectual component of all forms of learning. Invoking
intellectual, emotional, and normative archetypes deepens the experience by
imposing a complex overlay of associative mental models.
• Sensory Immersion: This occurs when the student employs an immersive dis-
play, like a head-mounted display, a CAVE, or a digital dome. The display
presents a panoramic egocentric view of some virtual world, which the student
leverages to imagine him or herself to be there. This type of immersion has been
1 Introduction: Virtual, Augmented, and Mixed … 5

used extensively for vehicle training and other procedural learning applications.
There is also solid evidence that it can advantage students who need to learn the
declarative knowledge connected to three-dimensional structures (Salzman,
Dede, Loftin, & Chen, 1999; Jacobson, 2011, 2013).
• Social Immersion: As discussed in Gardner’s and Kraemer’s chapters, rich
social interactions among participants in a shared virtual or mixed reality
deepens their sense of immersion. In the real world, we participate in shared
processes of reasoning between people who leverage their environment to make
decisions and get things done. To the extent that a virtual or partially virtual
environment supports this, it draws the user in, makes him or her feel more a
part of it.
Psychological immersion is achievable in any of these interfaces by design
strategies that combine actional, social, symbolic, and sensory factors.
Immersion is intrinsically helpful for motivation and learning in some ways, but
not necessarily useful in others. In mastering complex knowledge and sophisticated
skills, students learn well in a Plan, Act, Reflect cycle (PAR), in which first they
prepare for an experience that involves doing something they want to master, then
they attempt that performance, and finally they assess what went well, what did not,
why, and what they need to learn in order to execute a more successful repetition of
the cycle. Immersion is great for the Act part of the cycle, but unless used carefully
can interfere with the Plan and the Reflect parts of the cycle. This—and numerous
other factors—make effective instructional design for immersive learning complex.

1.3.2 Situated Learning and Transfer via Psychological

Immersion

The capability of VR, MUVE, and MR interfaces to foster psychological immersion

enables technology-intensive educational experiences that draw on a powerful
pedagogy: situated learning.
Situated Learning: “Situated” learning takes place in the same or a similar context
to that in which it is later applied, and the setting itself fosters tacit skills through
experience and modeling. For example, in a medical internship, both the configu-
ration and the coordinated team activities in a hospital surgical operating room
provide embedded knowledge.
Situated learning requires authentic contexts, activities, and assessment coupled
with guidance from expert modeling, mentoring, and “legitimate peripheral par-
ticipation” (Wenger, 1998). As an example of legitimate peripheral participation,
graduate students work within the laboratories of expert researchers, who model the
practice of scholarship. These students interact with experts in research as well as
with other members of the research team who understand the complex processes of
scholarship to varying degrees. While in these laboratories, students gradually
6 C.J. Dede et al.

move from novice researchers to more advanced roles, with the skills and expec-
tations for them evolving.
Related to situated learning is embodied cognition, an instructional strategy that
posits retrieving a concept from memory and reasoning about it is enhanced by
creating a mental perceptual simulation of it (Barsalou, 2008). For example,
research shows that second grade students who acted out stories about farms using
toy farmers, workers, animals, and objects increased their understanding and
memory of the story they read. Steps involved in a grounded cognition approach to
learning something include having an embodied experience (which could be created
by immersive interfaces), learning to imagine that embodied experience as a mental
perceptual simulation, and imagining that experience when learning from symbolic
materials.
Potentially quite powerful, situated learning is seldom used in formal instruction
because creating tacit, relatively unstructured learning in complex real-world set-
tings is difficult. However, VR, MUVE, and MR experiences can draw on the
power of situated learning by creating immersive, extended experiences with
problems and contexts similar to the real world. In particular, all three types of
immersive interfaces provide the capability to create problem-solving communities
in which participants can gain knowledge and skills through interacting with other
participants who have varied levels of skills, enabling legitimate peripheral par-
ticipation driven by social and collaborative interactions.
Situated learning is important in part because of the crucial issue of transfer.
Transfer: Transfer is the application of knowledge learned in one situation to
another situation, demonstrated if instruction on a learning task leads to improved
performance on a transfer task, typically a skilled performance in a real-world
setting. For example, statistical reasoning learned in a classroom can potentially aid
with purchasing insurance, or with gambling.
A major criticism of instruction today is the low rate of transfer generated by
conventional instruction. Even students who excel in schooling or training settings
often are unable to apply what they have learned to similar real-world contexts.
Situated learning addresses this challenge by making the setting in which learning
takes place similar to the real-world context for performance in work or personal
life. Learning in well-designed digital contexts can lead to the replication in the real
world of behaviors successful in simulated environments (Fraser et al., 2012;
Mayer, Dale, Fraccastoro, & Moss, 2011; Norman, Dore, & Grierson, 2012).
Moreover, the evolution of an individual’s or group’s identity is an important
type of learning for which simulated experiences situated in immersive interfaces
are well suited (Gee, 2003; Turkle, 1997). Reflecting on and refining an individual
identity is often a significant issue for students of all ages, and learning to evolve
group and organizational identity is a crucial skill in enabling innovation and in
adapting to shifting contexts. Identity “play” through trying on various represen-
tations of the self and the group in virtual environments provides a means for
different sides of a person or team to find common ground and the opportunity for
1 Introduction: Virtual, Augmented, and Mixed … 7

synthesis and evolution (Laurel, 1993; Murray, 1998). As discussed in Slater’s

chapter, immersion is important in this process of identity exploration because
virtual identity is unfettered by physical attributes such as gender, race, and
disabilities.
Another attribute that makes immersive learning different, and potentially more
powerful than real world learning, is the ability to create interactions and activities
in mediated experience not possible in the real world. These include, for example,
teleporting within a virtual environment, enabling a distant person to see a real-time
image of your local environment, or interacting with a (simulated) chemical spill in
a busy public setting. Slater’s chapter categorizes opportunities for learning in
simulations of settings not possible in the real world. Jacobson’s chapter addresses
how to develop representations that are authentic for learning.
All these capabilities suggest that, to maximize the power of immersive learning
it’s important not to present isolated moments in which VR, MUVEs, and AR are
used to provide short-term engagement or fragmentary insight. Instead, extended
experiences that immerse students in rich contexts with strong narratives, authentic
practices, and links to real world outcomes are what truly unleash the transfor-
mational power of immersion. For example, while showing a 3-D model of a
human heart illustrating blood flow is useful, immersing students in a virtual setting
where they are applying knowledge of the heart to save the lives of computer-based
agents is much more motivating, as well as effective in fostering a wide range of
complex knowledge and sophisticated skills.
Schnieder references constructivism explicitly in his chapter, and the term is
used widely in the educational VR literature. Constructivism is nearly the same
thing as Situated Learning, because it revolves around a managed ecosystem within
which students build their own learning experience.

1.3.3 Approaches to Designing Immersive Educational

Media

1.3.3.1 Simulation

In this approach, the learning experience is an immersive simulation of an artifact,

environment, or situation that exists in real life. For example, the phenomenon
modeled at an appropriate level of fidelity for the instructional goals could be a
virtual garden, or a simulated first-responder crisis with a building on fire, or a
representation of some industrial process. Importantly, the simulation allows
learners to do a few things they could not do in real life (National Research Council,
2011). For example, they could change the season of a virtual forest with the touch
of a button, or move along a timeline for historical change, or operate dangerous
machinery that would be too risky to learn how to use (at first) in real life.
8 C.J. Dede et al.

In classroom settings, the instructional design is built around the simulation,

with students often using a PAR cycle to interact with the simulation, comple-
mented by the instructor teaching with knowledge and skills outside the simulation
to aid in developing effective performances. The design of the curriculum in which
the simulation is embedded is important, as is professional development for
effective use of simulations in classrooms.
Some advantages of simulations are that they are quick to deploy, compared to
developing more complex experiential environments, and relatively straightforward
to understand. They are well suited for teaching students procedural knowledge,
using skills to help them accomplish tasks that require some sequence of actions.

1.3.3.2 Constructionist Activities

In his chapter, Schneider discusses constructivism, the educational learning theory

of which constructionism is one approach. Constructionist learning theory is based
on the assumption that developing knowledge occurs best through building artifacts
(physical or digital) that can be experienced and shared (Papert, 1991). In this type
of learning, participants are given tools to build their own immersive environments,
or provided an immersive environment and told to build something within it. In a
classroom setting, an illustration would be a learning project where each student
designs a “monster truck” for each one of the planets in our solar system, and then
attempts to drive their truck on the surface of that planet. To be successful, they
have to learn about that planet’s characteristics (e.g., gravity, temperature) and
about the process of engineering a vehicle.
This approach can be very effective, because it empowers the learners to create
something in which they have an emotional investment. Further, participants learn
how to author in the immersive technology. As discussed later, all of the immersive
technology manufacturers go to great lengths to develop tutorials and educational
materials for people who want to informally create experiences that use their
products. In formal education, all those materials can be leveraged in a construc-
tionist curriculum. Further, research suggests that children can handle this approach
at an earlier age than parents and teachers might expect.
As with simulations, in classroom settings the teacher must manage the learning
process. She will need guidance in the technology itself and in instructional
strategies, as well as a curriculum surrounding the building experience that is linked
to academic objectives (Laurillard, 2009).

1.3.3.3 Embodied Cognition

As discussed earlier, embodied cognition learning experiences involving creating a

mental perceptual simulation useful when retrieving a concept or reasoning about it
(Barsalou, 2008). An embodied immersive experience via VR, MUVE, or MR can
1 Introduction: Virtual, Augmented, and Mixed … 9

develop such a mental perceptual simulation, especially when facilitated by cur-

ricular and instructional support.
Embodied experience with academically important situations and phenomena is
often limited, both by personal circumstances and by limitations of the real world.
For example, an impoverished inner city student may never visit a farm, and no one
now can have a physically embodied experience of living in the 17th century, or
seeing relativistic effects when moving close to the speed of light. Digitally
immersive learning experiences can bridge these gaps.
This approach to learning is not new; Montessori used analog artifacts as an
important part of her pedagogical method. With the emergence of multi-modal
interfaces that include gestures and similar physical movements, new forms of
digitally enhanced embodied cognition are now possible and practical. Research on
effective designs for immersive embodied learning is an exciting frontier for formal
and informal education.

1.3.3.4 Directed Immersive Narrative

In this type of instructional design, learners participate in—and shape—a narrative.

Participants are guided from beginning to end, but also choose their own path, with
meaningful choices along the way to help them learn. In gaming, this type of
immersive learning is usually a MUVE, although VR games are emerging. As
discussed in Klopfer’s chapter, the MUVE The Radix Endeavor is an example.
However, such a learning experience could also include MR, if the phenomenon
studied is some kind of activity that would make sense in a readily accessible type
of physical environment, like an ecosystem. As discussed in Dede’s chapter,
EcoMOBILE is an augmented reality that illustrates this.
Directed narratives provide a superstructure within which the other types of
immersive learning designs can be placed: simulations, constructionism (in part
through the participant’s choices), and embedded cognition. The story that emerges
is the vehicle for identity and transfer that makes the whole more than the sum of its
parts.
Transmedia narratives are emerging as a new form of entertainment and learn-
ing. Such a narrative can span immersive and non-immersive media, creating “al-
ternate realities” that interweave fact and fiction to form myth. In education, the
challenge is to immerse participants in the alternate reality for learning, but then
fade its attraction so they return to the real world empowered through what they
now know and can do.

1.3.3.5 Learning Simpler Material

This discussion has focused on learning complex knowledge and sophisticated

skills, but what is the role of immersion in learning simpler, foundational material?
It may seem counterintuitive given prevalent educational practice for centuries, but
10 C.J. Dede et al.

basic ideas are best learned in the context of attempting a relatively complicated
task that is engaging and has relevance to the real world. Learning involving rote
performances and low-level retention (e.g., math facts, vocabulary words) is not
intrinsically interesting, and many students quickly tire of music, animations,
simple games, and other forms of extrinsic rewards (the chocolate-covered broccoli
problem). This leads to apathy about mastering foundational content and skills,
especially when they have no perceived relevance to the learner’s life. This moti-
vational problem is exacerbated by a fundamental assumption of behaviorist
instructional design that no complex knowledge or skill is learnable until the stu-
dent has mastered every simple, underlying sub-skill. This tenet leads to long initial
sequences of low-level teaching by telling and learning by listening, followed by
rote practice with extrinsic bribery to keep going. In this common situation, stu-
dents often lose sight of why they should care about learning the material, which
may seem to them remote from the eventual goal-state of an engaging, complex
knowledge or skill with real-world utility.
Substantial theory, research, and experience documents that—in contradiction to
behaviorist theories of learning—students can master simple skills in the context of
learning a complex task that is engaging and relevant to them (Dede, 2008). In
contrast to conventional practice now, even when learning foundational material,
students will experience higher motivation and longer retention of simple skills
learned via the types of simulations, constructionist experiences, and directed
immersive narratives discussed above. While learning by guided social construc-
tivism seems inefficient compared to direct instruction, because more time is
required, in the long run this approach is more effective, because less re-teaching is
required due to problems with retention as un-engaging material is memorized,
immediately tested, then forgotten.
So, if one is using such an approach to foundational learning, what is the role of
immersion for the parts of instruction that involve simple skills and knowledge?
While the psychological aspects of immersion are always useful in learning, sen-
sory immersion in VR is necessary only for material that is intrinsically
3-dimensional (e.g., understanding the role of the ecliptic plane in the solar system)
or where embodied cognition is useful (e.g., becoming an animal to experience its
relationship to an ecological niche). 2-D simulations, non-immersive construc-
tionism, and non-digital narratives—even rote teaching and learning—may be as
effective and more efficient than immersive media if used for foundational learning
in the context of a guided social constructivist experience.

1.4 Overview of the Chapters

The book begins with this introductory chapter introducing terms and conceptual
frameworks, as well as providing a quick summary for the contents of each chapter,
grouped into two types of discussions. Frameworks for the design and imple-
mentation of immersive learning are delineated in chapters by Slater; Jacobson;
1 Introduction: Virtual, Augmented, and Mixed … 11

Kraemer; Shute, Rahimi, and Emihovich; Richards; and Liu and Huang. Then,
Case Studies of immersive learning are described in chapters by Dede, Grotzer,
Kamarainen, and Metcalf; Gardner and Sheaffer; Klopfer; Johnson-Glenberg; and
Schneider. Finally, a concluding chapter summarizes cross-cutting themes and
advances a proposed research agenda.

1.4.1 Frameworks for the Design and Implementation

of Immersive Learning

Slater’s chapter, Implicit Learning through Embodiment in Immersive Virtual

Reality, presents a framework for understanding how VR results in the illusion of
presence. The participant in a VR scenario typically has the illusion of being in the
virtual place and, under the right conditions, the further illusion that events there are
really occurring. Slater also describes a further illusion that can be triggered in VR,
referred to as body ownership. This can occur when the participant sees a life-sized
virtual body substituting her or his own, from first person perspective. This virtual
body can be programmed to move synchronously with the participant’s real body
movements, thus leading to the perceptual illusion that the virtual body is her or his
actual body. Slater surveys various experiments showing that the form of the virtual
body can result in implicit changes in attitudes, perception and cognition, as well as
changes in behavior. He compares this with the process of implicit learning and
concludes that virtual body ownership and its consequences may be used as a form
of implicit learning. He concludes by suggesting how the study of the relationship
between body ownership and implicit learning might be taken forward.
Jacobson’s chapter, Authenticity in Immersive Design, describes authenticity a
concept found in both media design and educational design, usually as a quality
needed for success. He develops a theory of authenticity for educational experi-
ences with immersive media (VR, MR, MUVEs) to help educators and designers in
this new field. In this framework, authenticity refers to the relationship between a
truth and its representation, guided by a purpose; A representation or an experience
is said to be authentic, when it successfully captures the fundamental truth of what
we are learning. This framework provides a practical way to look at one key
dimension of good educational design.
Kraemer’s chapter, The Immersive Power of Social Interaction, reviews new
technologies and their impact on learning and students’ motivation. The main
argument is that, in order to achieve immersion, social interactions should be
fostered. Three technologies are discussed that either inherently draw on social
interactions (pedagogical agents, transformed social interaction) or can be enriched
by including collaborative learning elements (augmented reality). For each of the
three technologies, a short overview is given on the state of current developments,
as well as on results from empirical. Also, discussed is to what extent these
developments have built on social interaction, how this usage might be extended
and whether beneficial outcomes can be expected from increased usage.
12 C.J. Dede et al.

The chapter by Shute, Rahimi, and Emihovich focuses on how to design and
develop valid assessments for immersive environments (IEs), particularly those
providing “stealth assessment,” an ongoing, unobtrusive collection and analysis of
data as students interact within IEs. The accumulated evidence on learning thus
provides increasingly reliable and valid inferences about what students know and
can do across multiple contexts, for both cognitive and non-cognitive variables. The
steps toward building a stealth assessment in an IE are presented through a worked
example. The chapter concludes with a discussion about future stealth assessment
research, how to move this work into classrooms to enhance adaptivity and
personalization.
Shifting from the design focus in prior chapters to implementation issues,
Richard’s chapter summarizes the distribution and availability of the infrastructure
needed for using VR and MR in the schools. Using immersive media requires a
technology infrastructure consisting of dependable high-speed Internet connectivity
to the classroom, a ratio of at least one-to-one computer to student, an interactive
white board, and curriculum materials that can be monitored and controlled by the
teacher. This infrastructure, the Digital Teaching Platform, is quickly becoming a
reality. However, a larger and more complex barrier remains: integrating the new
technologies with existing classroom systems and with existing and emerging
pedagogical practice. The evolving nature of digital curricula, formative assess-
ment, and classroom practice impact how teachers will be able to integrate these
new technologies. Richards also addresses how immersive media can work as
supplemental digital materials for instruction and assessment. In particular, he
focuses on issues of the sensory comfort and fidelity of interaction, as these impact
the viability of these technologies in the classroom.
Liu and Huang provide the last chapter in this section, The Potentials and Trends
of Virtual Reality in Education. This presents an overview of virtual reality research
in education, including a bibliometric analysis to evaluate the publications on
virtual reality from 1995 to 2016, based on the Thomson Reuters’s Web of Science
(WoS). A total of 975 related documents were analyzed based on their publication
patterns (documents types and languages, major journals and their publications,
most prolific authors, most productive journals and their publications, and inter-
national collaborations). Bibliometric results show that the number of article has
been increasing since 1995 exponentially. USA, UK, and Chinese Taipei are the top
3 most productive countries/regions that are involved in virtual reality research in
education. The findings can help researchers to understand current developments
and barriers in applications of virtual reality to education.

1.4.2 Case Studies of Immersive Learning

The Framework chapters are followed by chapters presenting case studies of

immersive media for learning. Virtual Reality as an Immersive Medium for
Authentic Simulations, by Dede, Grotzer, Kamarainen, and Metcalf, describes a
1 Introduction: Virtual, Augmented, and Mixed … 13

design strategy for blending virtual reality (VR) with an immersive multi-user
virtual environment (MUVE) curriculum developed by the EcoLearn design team at
Harvard University for middle school students to learn ecosystems science. The
EcoMUVE Pond middle grades curriculum focuses on the potential of immersive
authentic simulations for teaching ecosystems science concepts, scientific inquiry
(collaborative and individual), and complex causality. The curriculum is
inquiry-based; students investigate research questions by exploring the virtual
ecosystem and collecting data from a variety of sources over time, assuming roles
as ecosystems scientists. The implications of blending in VR for EcoMUVE’s
technical characteristics, user-interface, learning objectives, and classroom imple-
mentation are discussed. Then, research questions for comparisons between the VR
version and the “Classic” version are described. The chapter concludes with gen-
eralizable design heuristics for blending MUVE-based curricula with head-mounted
display immersion.
Gardner and Sheaffer’s chapter, Systems to Support Co-Creative Collaboration
in Mixed-Reality Environments, examines the use of mixed-reality technologies for
teaching and learning, particularly for more active and collaborative learning
activities. The basis for this work was the creation of the MiRTLE platform—the
Mixed Reality Teaching and Learning Environment. They report on some of the
lessons learnt from using this platform on a range of different courses and describe
how different active/collaborative approaches were used. They also provide evi-
dence of the effect of these different approaches on the overall student attainment
and discuss the implications on the use of this technology, describing some of the
technological research being done to develop these mixed reality learning spaces
and the affordances offered by this approach. Finally they reflect on the tensions
between the pedagogy and technology and consider the implications for the wider
systems that support teaching and learning and co-creative collaboration in
mixed-reality environments.
Klopfer’s chapter, Massively Multiplayer Online Roleplaying Games and
Virtual Reality Combine for Learning, argues that the way Virtual Reality (VR) can
really make a difference in learning are involve bringing a truly unique experience
to students. The simulated online world of games is an ideal genre for this, because
these games provide a set of structures that not only scaffold learners in solving
complex problems, but also provide a great deal of freedom to explore personally
interesting pathways. In particular, Massively Multiplayer Online Roleplaying
Games (MMOs) offer an environment that supports social learning and exploration
around increasingly challenging problems. VR can greatly enhance MMOs through
opportunities for more natural and expressive communication and collaboration, as
well as ways to visualize the complex information resulting from interactions in this
space. When this approach is applied in an educational context, learners can be
presented with challenging problems, requiring participation from multiple players
around realistic scientific concepts. As this genre moves forward, it can explore
interesting hybrid approaches that combine VR with Augmented Reality (AR) and
traditional displays to meet the needs of schools, teachers, and learners.
14 C.J. Dede et al.

Johnson-Glenberg’s chapter, Embodied Education in Mixed and Mediated

Realities, provides a summary of some of this lab’s immersive media and embodied
STEM learning research. This synthesis focuses on the integration of gesture in
learning, and a new gesture-based assessment. A taxonomy for embodiment in
education is included. The chapter concludes with several design principles that the
Embodied Games Lab has culled over the years while creating educational content
that maximizes the affordances of virtual and mixed reality technologies and
meshes those with best pedagogical practices.
Schneider’s chapter, Preparing Students for Future Learning with Mixed Reality
Interfaces, explores how new learning environments, such as mixed reality inter-
faces (i.e., interfaces that combine physical and virtual information), can prepare
students for future learning. He describes four controlled experiments in which
students learned complex concepts in STEM via a Tangible User Interface that
created a “Time for Telling”. This is followed by a summary the findings from this
research, a discussion of the possible mechanisms for the effects found in those
studies, and a suggestion of design guidelines for creating this type of constructivist
activities. He conclude by discussing the potential of mixed reality interfaces for
preparing students for future learning.
Following these sections on Frameworks and Case Studies, a concluding chapter
summarizes cross-cutting themes and advances a proposed research agenda. The
book ends with a glossary of terms related to immersive learning.

Acknowledgements The editors, authors, and workshop participants are grateful for the financial
support provided by NetDragon Websoft.

References

Barsalou, L. W. (2008). Grounded cognition. Annual Review of Psychology, 59, 617–645.

Dawley, L., & Dede, C. (2013). Situated learning in virtual worlds and immersive simulations.
In J. M. Spector, M. D. Merrill, J. Elen, & M. J. Bishop (Eds.), Handbook of research for
educational communications and technology (4th ed., pp. 723–734). New York, NY: Springer.
Dede, C. (2009). Immersive interfaces for engagement and learning. Science, 323, 66–69.
Dede, C. (2008). Theoretical perspectives influencing the use of information technology in
teaching and learning. In J. Voogt & G. Knezek (Eds.), International Handbook of Information
Technology in Primary and Secondary Education (pp. 43–62). New York: Springer.
Dunleavy, M., & Dede, C. (2013). Augmented reality teaching and learning. In J. M. Spector, M.
D. Merrill, J. Elen, & M. J. Bishop (Eds.), Handbook of research for educational
communications and technology (4th ed., pp. 735–745). New York, NY: Springer.
Fraser, K., Ma, I., Teteris, E., Baxter, H., Wright, B., & McLaughlin, K. (2012). Emotion,
cognitive load and learning outcomes during simulation training. Medical Education, 46(11),
1055–1062.
Gee, J. P. (2003). What video games have to teach us about learning and literacy. New York, NY:
Palgrave Macmillan.
Jacobson, J. (2008). Ancient architecture in virtual reality; Does immersion really aid learning?
Doctoral Dissertation, School of Information Sciences, University of Pittsburgh. URN:
etd-04222008-172117
1 Introduction: Virtual, Augmented, and Mixed … 15

Jacobson, J. (2011). Digital dome versus desktop display in an educational game: Gates of Horus.
International Journal of Gaming and Computer-Mediated Simulations, 3(4). http://publicvr.
org/publications/IJGCMS-PublicDraft.pdf.
Jacobson, J. (2013). Digital dome versus desktop display; Learning outcome assessments by
domain experts, International Journal of Virtual and Personal Learning Environments, 4(3).
http://publicvr.org/publications/JacobsonIJVPLE-12.pdf.
Laurel, B. (1993). Computers as theatre. New York, NY: Addison-Wesley.
Laurillard, D. (2009). The pedagogical challenges to collaborative technologies. International
Journal of Computer-Supported Collaborative Learning, 4(1), 5–20.
Mayer, B. W., Dale, K. M., Fraccastoro, K. A., & Moss, G. (2011). Improving transfer of learning:
Relationship to methods of using business simulation. Simulation & Gaming, 42(1), 64–84.
Murray, J. H. (1998). Hamlet on the holodeck: The future of narrative in cyberspace. Cambridge,
MA: MIT Press.
National Research Council. (2011). Learning science: Computer games, simulations, and
education. Washington, DC: Board on Science Education, Division of Behavioral and Social
Sciences and Education.
Norman, G., Dore, K., & Grierson, L. (2012). The minimal relationship between simulation
fidelity and transfer of learning. Medical Education, 46(7), 636–647.
Palincsar, A. S. (1998). Social constructivist perspectives on teaching and learning. Annual Review
of Psychology, 49(1), 345–375.
Papert, S. (1991). Situating constructionism. In I. Harel & S. Papert (Eds.), Constructionism:
Research reports and essays, 1985–1990 (pp. 1–11). Norwood, NJ: Ablex.
Salzman, M. C., Dede, C., Loftin, R. B., & Chen, J. (1999). A model for understanding how virtual
reality aids complex conceptual learning. Presence: Teleoperators and Virtual Environments, 8
(3), 293–316.
Turkle, S. (1997). Life on the screen: Identity in the age of the internet. New York, NY: Simon and
Schuster.
Wenger, E. (1998). Communities of practice: Learning, meaning, and identity. Cambridge,
England: Cambridge University Press.

Author Biographies

Christopher J. Dede is the Timothy E. Wirth Professor in Learning Technologies at Harvard’s

Graduate School of Education (HGSE). His fields of scholarship include emerging technologies,
policy, and leadership. From 2001–2004, he was Chair of the HGSE department of Teaching and
Learning. In 2007, he was honored by Harvard University as an outstanding teacher, and in 2011
he was named a Fellow of the American Educational Research Association. From 2014–2015, he
was a Visiting Expert at NSF, Directorate of Education and Human Resources. Chris has served as
a member of the National Academy of Sciences Committee on Foundations of Educational and
Psychological Assessment, a member of the U.S. Department of Education’s Expert Panel on
Technology, and a member of the 2010 National Educational Technology Plan Technical Working
Group. In 2013, he co-convened a NSF workshop on new technology-based models of
postsecondary learning; and in 2015 he led two NSF workshops on data-intensive research in the
sciences, engineering, and education. His edited books include: Scaling Up Success: Lessons
Learned from Technology-based Educational Improvement, Digital Teaching Platforms:
Customizing Classroom Learning for Each Student, and Teacher Learning in the Digital Age:
Online Professional Development in STEM Education.
16 C.J. Dede et al.

Dr. Jeffrey Jacobson, Ph.D., has investigated fully immersive virtual reality (VR) as a learning
medium for two decades, developing the technology and conducting experimental research. His
early technical work in affordable free software is widely cited in the literature. His experimental
trials on VR verses desktop displays were one of the few successful media comparison studies ever
conducted. His later work (NSF and NEH funded) is highly regarded among scholars of cultural
history and heritage. He has given hundreds of talks and demonstrations at top universities,
academic conferences, and industrial conventions. Today, Dr. Jacobson is a co-founder and leader
BostonVR, the fifth largest VR meet-up group in the world. He is currently consulting with
educators and professionals in several industries, including university graduate studies, architec-
tural design, large-vehicle piloting, and virtual reality display.

John Richards, Ph.D., is Adjunct Faculty at the Harvard Graduate School of Education teaching
Entrepreneurship in the Education Marketplace. He is founder and President of Consulting
Services for Education, Inc. (CS4Ed). CS4Ed works with publishers, developers, and educational
organizations, as they negotiate the rapidly changing education marketplace to improve business—
planning processes, and to develop, evaluate, and refine products and services.
John was President of the JASON Foundation, and GM of Turner Learning–the educational arm
of CNN and the Turner Broadcasting System. Over the years, John has served on boards for a
variety of education groups including NECC; Cable in the Classroom; Software Information
Industry Association (SIIA), Education Market section; and the Association of Educational
Publishers (AEP). John’s projects have won him numerous awards including two Golden Lamps
and several CODIEs, as well as several EMMY nominations. He is a respected keynote speaker,
has been responsible for the publication of over 1000 educational products, and is the author/editor
of over 100 chapters and articles, and four books, including Digital Teaching Platforms, Teacher’s
College Press (with Chris Dede). He is the primary author of the Software and Information
Industry Association’s annual U.S. Educational Technology Market: Pre K-12 report.
 Part I
Frameworks for the Design
and Implementation of Immersive
Learning
Chapter 2
Implicit Learning Through Embodiment
in Immersive Virtual Reality

Mel Slater

Abstract Virtual reality (VR) typically results in the illusion of presence. The
participant in a VR scenario typically has the illusion of being in the virtual place,
and under the right conditions the further illusion that events that are occurring
there are really occurring. We review how these properties are useful for the
application of VR in education. We present a further illusion that can be triggered in
VR referred to as body ownership. This can occur when the participant sees a
life-sized virtual body substituting her or his own, from first person perspective.
This virtual body can be programmed to move synchronously with the participant’s
real body movements, thus leading to the perceptual illusion that the virtual body is
her or his actual body. We survey various experiments that show that the form of
the virtual body can result in implicit changes in attitudes, perception and cognition,
and changes in behavior. We compare this with the process of implicit learning and
conclude that virtual body ownership and its consequences may be used as a form
of implicit learning. We conclude by suggesting how the study of the relationship
between body ownership and implicit learning might be taken forward.

Keywords Virtual reality
Education
Implicit learning
Embodiment

Body ownership

M. Slater (&)
Institució Catalana de Recerca I Estudis Avançats (ICREA),
Barcelona, Spain
e-mail: [email protected]; [email protected]
URL: http://www.melslater.me
M. Slater
University College London, London, UK
M. Slater
University of Barcelona, Edifici Teatre, Passeig de La Vall d’Hebron 171,
08035 Barcelona, Spain

© Springer Nature Singapore Pte Ltd. 2017 19

D. Liu et al. (eds.), Virtual, Augmented, and Mixed Realities in Education,
Smart Computing and Intelligence, DOI 10.1007/978-981-10-5490-7_2
20 M. Slater

2.1 Introduction

In this article we consider how Immersive Virtual Reality may be useful for implicit
learning, that is acquiring knowledge and skills without conscious effort, or without
explicitly having to learn specific information. First, we briefly recap what virtual
reality is (see also Chap. 1) and some essential concepts, then we review virtual
reality in education and implicit learning, and go on to provide some examples of
how it has been exploited to bring about changes in people. Note that for the
purpose of this article we extend the notion of implicit learning to mean accom-
plishing changes to the self that were not explicitly programmed, including changes
in attitudes, behaviors and cognition.

2.1.1 Immersive Virtual Reality

Some background and history of Virtual Reality (VR) is discussed in Chap. 1 of

this book, and further information and discussion of a range of applications is
presented in (Slater & Sanchez-Vives, 2016). VR incorporates participants bodily
into a virtual computer-generated world. A critical aspect is that as the participant
turns or moves his or her head the computer updates the images displayed at a very
high frame rate, ideally at least at 60 Hz. The participants therefore become
immersed in a completely surrounding virtual environment that they see when
turning in any direction with movement and motion parallax, and they perceive this
(ideally) in a wide field-of-view display. (The same can be done with spatialized
sound.) A way to grasp the difference between VR and other media comes from
noting that in VR we can go into a virtual movie theatre and watch a movie. We can
enter a virtual living room and watch a virtual TV. We can sit by a virtual computer
and play a computer game—all while in virtual reality. In fact in VR we can even
simulate the process of entering into a VR and thus enter realities within realities
(Slater, 2009). There is no other technology that has ever enabled this.

2.1.2 Presence—Place Illusion and Plausibility

The fundamental experience that VR delivers is referred to as ‘presence’. This is the

perceptual illusion of being in the place rendered by the VR system. Presence in
virtual environments was first elucidated in a set of papers in the early 1990s and
there followed many years of further research into this concept reviewed in
(Sanchez-Vives & Slater, 2005). Presence was deconstructed into two independent
concepts in (Slater, 2009), referred to as Place Illusion (PI) and Plausibility Illusion
(Psi). PI refers to the original idea of the illusion of being in the virtual place. It was
argued that a necessary condition for this illusion is that the virtual reality is
2 Implicit Learning Through Embodiment in Immersive Virtual Reality 21

perceived through natural sensorimotor contingencies, a theory referred to as the

active vision paradigm (O’Regan & Noë, 2001). This argues that we perceive
through using our whole body, via a set of implicit rules involving head turning,
leaning, reaching, looking around, bending an ear towards, and so on. The illusion
of ‘being there’ can be generated to the extent that the VR system affords perception
through natural sensorimotor contingencies. The argument is that if we perceive
using the same methods in an environment as we normally perceive, then we must
be in that environment, this being the simplest hypothesis for the brain to adopt.
Place illusion is the first effect of viewing a scene with a VR display. However,
the Plausibility Illusion (Psi) is the illusion that the events experienced in VR are
really happening (even though the participant knows that they are not). Psi requires
that the virtual environment respond to actions of the participant, generates spon-
taneous actions towards the participant, and is ecologically valid when the envi-
ronment is meant to depict real-life events. For example, when the environment
includes virtual human characters, they should acknowledge the presence of the
participant (e.g., by gaze) and respond to the participant’s actions (e.g., by main-
taining appropriate interpersonal distances).
When both PI and Psi operate then participants are likely to behave realistically
in the VR. This has far reaching consequences. For example, VR has been used
extensively for psychological therapy for 25 years. This is only possible because
patients exhibit sufficient affect in VR as to enable the clinicians to engage in the
therapeutic process.

2.1.3 Embodiment and Body Ownership

A participant in a VR wearing a HMD looking in any direction would always see

only the virtual world. What happens when the participant looks down towards his
or her own body? If it has been so programmed then they would see a virtual body
substituting their own. This virtual body would be life sized, approximately occupy
the same space as where the person feels their real body to be, in other words be
coincident in space with the real body. The body would be seen from first person
perspective, i.e., from the eyes of the virtual body providing the centers of pro-
jection from which the participant sees the virtual world.
Seeing the virtual body from first person perspective is already a cue to the brain
that it is the person’s actual body, thus providing an illusion towards this effect. The
illusion is enhanced if further multisensory feedback is applied. This follows from
the fundamental finding reported in the rubber hand illusion. Here a person seated
by a table has their real (say right) hand resting on the table but hidden behind a
screen, and a right-handed rubber arm placed in an anatomically plausible position
on the table in front of them, as if it were their real arm. Typically the hidden real
arm and the rubber one are approximately parallel. The experimenter touches both
the rubber hand and the hidden real hand synchronously so that the person sees the
touches on the rubber hand but feels them on their real hand (Botvinick & Cohen,
22 M. Slater

1998). After a few seconds of such stimulation proprioception shifts to the rubber
hand, so that although the person knows for sure that it is not their real hand, it feels
as though it is. When the seen and felt touch are asynchronous then the illusion
typically does not occur. Petkova and Ehrsson (2008) applied a similar technique to
the whole body, to produce a full body ownership illusion. In this case a stereo
camera was mounted on the head of a manikin pointing down towards its body, and
the video streamed to a person wearing a stereo HMD. Provided that the person was
looking down towards their real body, it would seem to them that their real body
had been substituted by the manikin body. The experimenter synchronously tapped
the abdomen of the manikin body (which would be seen through the HMD by the
participant) and the abdomen of the person. Thus the person would see the manikin
body being touched while feeling this on their real body, and integrate the two
percepts into one overall illusion that the manikin body was their body. As with the
RHI when the manikin body was threatened with a knife the participants exhibited
an increase in physiological arousal concomitant with the attack. A synchronous
tapping did not lead to these illusions.
It was demonstrated in (Slater, Perez-Marcos, Ehrsson, & Sanchez-Vives, 2008)
that an equivalent to the RHI could be achieved in VR, where the person saw a
virtual arm protruding from their shoulder, that was seen to be tapped by a virtual
ball, that was in fact controlled by a tracked wand touching their corresponding real
hand. Slater, Spanlang, Sanchez-Vives, and Blanke (2010) showed that a body
ownership illusion could be attained over a virtual body, and that the dominant
factor was seeing the body from first person perspective, although visuotactile
synchrony also contributed.
Transformed body ownership was first tried in the very early days of VR in the
late 1980s, although not as scientific research and therefore not published at the
time. Lanier (2006) later reported that at VPL, the company he led, in the late 1980s
and early 1990s they experimented with embodying people as virtual lobsters, and
used unusual combinations of human muscle movements as a means by which
people could move their lobster limbs. He termed this ‘Homuncular Flexibility’,
meaning that humans can quickly adapt to new bodies and new modes of bodily
control.
For the sake of terminology we refer to embodiment as the process by which the
person’s body is substituted by a virtual one—using the head-tracked stereo
head-mounted display, motion capture to track the person’s real movements and
map these to movements of the virtual body, or tactile stimulation on the person’s
body synchronous with virtual objects seen to touch the virtual body. Hence
embodiment refers to the actual setup, whereas ‘body ownership’ refers to the
perceptual illusion that the virtual body is the person’s own body (even though of
course they know that this is not the case). Later we will be discussing the con-
sequences of such virtual body ownership for implicit changes.
2 Implicit Learning Through Embodiment in Immersive Virtual Reality 23

2.2 Learning

2.2.1 Intentional Learning

As we have seen earlier VR has been developed, used, and studied for the past
25 years and there have been many applications in education. For recent reviews
see (Freina & Ott, 2015). There are at least five reasons why VR may contribute to
education: (i) Transforming the abstract to the concrete; (ii) Doing rather than only
observing; (iii) The infeasible or impossible becomes practical; (iv) explore
manipulations of reality; (v) go beyond reality to positive advantage. We consider
each in turn. This section is based on (Slater & Sanchez-Vives, 2016).

2.2.1.1 Transforming the Abstract to the Concrete

VR can transform abstractions into concrete perceptions and experiences. Hwang

and Hu (2013) used VR and showing that it is advantageous compared to standard
paper and pencil techniques in the learning of mathematics. Learning concepts
about vector algebra and spatial abilities was described by Kaufmann, Schmalstieg,
and Wagner (2000) using an HMD based augmented reality system. Roussou
(2009) used a ‘virtual playground’ for the teaching of mathematics by 50 eight to
twelve year olds, where children watched a virtual robot illustrating concepts
leading to enhanced enjoyment, better conceptual understanding, and reflection.

2.2.1.2 Doing Rather Than Observing

In general VR supports ‘doing’ rather than only observing. This is very important
for example in neurosurgery training—e.g. (Müns, Meixensberger, & Lindner,
2014)—or any kind of ‘hands on training’, especially that is too problematic or
dangerous to rehearse in reality.

2.2.1.3 Doing the Infeasible or Practically Impossible

VR can be used to carry out activities that may be infeasible in reality. A good
example here is learning geography, geology or archeology, where students would
typically be unable to visit real places, but could instead visit them virtually. This
idea of virtual field trips—e.g. (Lin et al., 2013)—has become popular and certainly
feasible inexpensively with today’s relatively low cost hardware.
24 M. Slater

2.2.1.4 Manipulating Reality

Einstein’s famous thought experiment about riding on a light beam can become a
concrete experience in VR. How would the world be if gravity were changed by a
fraction? How would it be like to play football in such a world? The theories of
relativity can be modeled and experienced in VR. Such ideas were propounded and
implemented by Dede, Salzman, Loftin, and Ash (1997). These ideas are worth
following up, since manipulating the parameters of reality is, course, not possible in
reality, but in VR this is possible.

2.2.1.5 Beyond Reality

VR is often thought of as a way to simulate and reproduce reality—for example,

visit ancient archaeological sites such as Qumran as part of a history lesson (Cargill,
2008), or manipulate parameters of reality as discussed in the previous section.
However, it is possible also to go quite beyond what is possible in reality in
unexpected and radical ways. For example, Bailenson et al. (2008b) showed how
teaching can be transformed where every student in a shared VR perceives that she
or he is the center of attention of the teacher through placement in the classroom
and through eye gaze feedback.
Generally with regard to VR and learning Fowler (2015) points out that too
much emphasis has been placed on the technical affordances of VR (such as pro-
viding an immersion in a 3D space) and not enough on the pedagogical aspects of
using VR. In particular VR applications must address (i) how the VR experience
advances explanation; (ii) deepening understanding, for example, through explo-
ration; (iii) taking account of the wider social context involved in learning. Any
system that uses VR cannot escape demonstrating how these three required prop-
erties are satisfied. In the next section we consider, however, an entirely different
approach that, it could be argued, obviates the need for such classifications, since
the aim is learning, but not explicit learning. Moreover, all of the above examples
rely essentially on the presence inducing aspects of VR: they transform learners to
another world, and make use of the properties and affordances of that world for
some type of learning. Instead we consider a more radical approach that while still
necessitating presence also relies on body ownership.

2.2.2 Implicit Learning

Implicit learning is the process whereby individuals learn complex information

unconsciously and gain abstract knowledge through this process (Reber, 1989). It
has been applied, for example, to the learning of artificial grammars (Reber, 1967)
where subjects are exposed to grammatically correct sentences over a set of sym-
bols and asked to reproduce them while receiving feedback about which were
2 Implicit Learning Through Embodiment in Immersive Virtual Reality 25

correct or incorrect, without reasons why. There is no explicit learning of rules, and
yet subjects are able to pick up the grammar after several such exposures. In
particular subjects are able to correctly infer information about novel stimuli
(Reber, 1989). However, they may not be able to explicitly articulate the complex
rules that they have learned.
Seger (1994) argued that implicit learning satisfies several criteria: (i) subjects
are not conscious of what they have learned and as mentioned above cannot
articulate it; (ii) the information learned is complex in the sense that it is not simply
based on correlations or counts of frequencies; (iii) the learning gained is not based
on hypothesis testing or explicitly trying to find patterns, but essentially people
acquire information incidentally through other cognitive processes than those that
might be employed through explicit, deliberate and directed learning. Since people
with amnesia apparently do as well on implicit learning as others, the neural basis
of such learning is quite different from that involved in tasks based on episodic
memory. A meta analysis of implicit learning in amnesic patients was carried out by
Kessels and Haan (2003). The neural basis of implicit learning is reviewed in
(Reber, 2013). Implicit learning is typically robust in the sense that the learning
does not fade over time, for example Agus, Thorpe, and Pressnitzer (2010) showed
how newly acquired sound patterns would be retained for weeks. A meta-analysis
has also been shown that people with autism spectrum disorders do well on implicit
learning (Foti, De Crescenzo, Vivanti, Menghini, & Vicari, 2015).
Apart from the obvious example of language, implicit learning is important for
such skills as surgery (Masters, Lo, Maxwell, & Patil, 2008), where was found that
learning by observation and without explicit verbal instruction produced results that
were particularly useful in the multi-tasking environment of a surgical operation. In
a similar vein Vine, Masters, McGrath, Bright, and Wilson (2012) showed that gaze
strategies of experts could be learned implicitly in the training of laparoscopic
skills, similar to a result that had earlier been shown using VR (Wilson et al., 2011).
Bailenson et al. (2008a) showed how implicit learning of motor tasks (Tai Chi)
exploiting VR could be accomplished. This also involved observation—of a
self-avatar seen from third person perspective and in a mirror, of recorded move-
ments of themselves and a teacher. The affordances offered through stereoscopy
outweighed viewing the same movements on a video with respect to ultimate
performance. Also using VR (Bell & Weinstein, 2011) showed how people with
psychiatric disability would improve their job interview skills by taking part in a
simulated job interview. Pan, Gillies, Barker, Clark, and Slater (2012) report an
experiment where males who are socially anxious about meeting women learn to
reduce their anxiety after a conversation with a virtual woman, that intersperses
mundane conversation (e.g., what work do you do?) with more personally alarming
discussion (e.g., do you have a girl friend?), even though there was no explicit
attempt at all to influence the participants in this direction. It as if the participants
incidentally learned through the mundane conversation that there was nothing to
fear from such an interaction, and then were able to carry this learning over to the
more personal aspects of the conversation at a lesser level of anxiety.
26 M. Slater

In all of the above examples the information to which people were exposed was
directly related to what was being implicitly learned: to implicitly learn a grammar
subjects were exposed to sentences, to implicitly learn Tai Chi or surgery subjects
observed examples. In the next Section we move on to a quite different form of
implicit learning—where the stimuli are not related to what is being learned except
in a quite indirect manner.

2.3 Implicit Change Through Virtual Body Ownership

2.3.1 The Proteus Effect

In Sect. 2.1.3 we introduced the concept of virtual body ownership, the perceptual
illusion that a virtual body coincident in space with the person’s real body will be
perceived as their own body. Here we examine the consequences of such trans-
formed body ownership.
The first work on these lines was by Yee and Bailenson (2007) who introduced
what they termed the ‘Proteus Effect’. (In modern parlance we might refer to the
god Proteus as a ‘shape shifter’.) Their observation was that the appearance and
actions of a person’s digital self-representation in both online non-immersive
environments and in VR affects their behavior. For example, they showed that
people in a collaborative VR moved closer or not to virtual representations of others
depending on whether the face of their own virtual body was judged more or less
attractive than their real one. People with taller virtual bodies were more aggressive
in a negotiation task than people with shorter bodies. A similar result has been
reported by Freeman et al. (2013) in a study of paranoia—that people seeing the
virtual world from a taller perspective (without actual embodiment in a virtual
body) were more confident in being with others than those with a point of view that
was shorter.
Groom, Bailenson, and Nass (2009) used the Proteus Effect to examine racial
bias. In the context of a simulated job interview they embodied White people in a
Black virtual body that they saw reflected in a virtual mirror for 1 min in a HMD,
with visuomotor feedback restricted to head movements. A racial Implicit
Association Test (IAT) (Greenwald, McGhee, & Schwartz, 1998) showed that there
was greater bias in favor of White after the embodiment.
The Proteus Effect is explained by Self Perception Theory (Bem, 1972) where it
is argued that people will infer the attitudes of others from inferring their behavior
in a situation, and also apply the same to themselves—i.e., infer their own attitudes
by inferring this from their own behavior. Yee and Bailenson (2007) also argue that
there is a stereotyping effect, that people behave in a situation according to how
others would expect a person with such a body to behave. These theories might
explain the racial bias results of Groom et al. (2009) since participants were placed
in a social situation (a job interview) where racial bias is known to operate.
2 Implicit Learning Through Embodiment in Immersive Virtual Reality 27

However, it would not explain results where there is no social context and there are
no behavioral demands on participants—they simply observe their own virtual body
from first person perspective or in a mirror. Most importantly they cannot explain
changes that occur as a result of embodiment that would not be expected to be
associated with transformed body ownership, or which are not under the control of
participants. For example, it has been observed that the RHI results in a cooling of
the associated real hand (Moseley et al., 2008), and Salomon, Lim, Pfeiffer, Gassert,
and Blanke (2013) showed that this applied to the whole virtual body.

2.3.2 The Multisensory Framework

In the multisensory framework people see their virtual body from first person
perspective usually with another type of sensory input consistent with the virtual
body being their own. This may be visuotactile stimulation, where objects seen to
touch the virtual body trigger corresponding feelings of touch on the real body (as
in the RHI), or visuomotor synchrony, where through real-time motion capture the
virtual body is programmed to move synchronously and in correspondence with
real body movements, or both (Kokkinara & Slater, 2014).
Theoretical underpinnings of body ownership have been formulated by Blanke,
Slater, and Serino (2015). This includes (i) multisensory integration of proprio-
ception, (ii) top-down body-related visual information in peripersonal space,
(ii) embodiment in the sense we have described above. These requirements are
clearly satisfied with virtual embodiment that includes first person perspective and
visuomotor synchrony. It was further argued in (Banakou & Slater, 2014), that
since whenever in our whole lives we have looked down towards ourselves we have
seen our body. Moreover, in normal healthy conditions whenever we move our
limbs we see them move. There is therefore, in the context of virtual embodiment,
overwhelming evidence that when there is embodiment with first person perspec-
tive and visuomotor synchrony that the simplest perceptual hypothesis for the brain
to adopt is that the virtual body is our own body, irrespective of how much it looks
like our body or not. As has been empirically found in the Proteus Effect sometimes
this change of body carries with it other changes at the attitudinal, behavioral,
physiological, and cognitive levels.
In a study of racial bias Peck, Seinfeld, Aglioti, and Slater (2013) embodied
light-skinned females (15 per group) in either a dark-skinned virtual body, a
light-skinned one, a purple one, or no body. The body in all cases was seen directly
from first person perspective and in a virtual mirror, and with visuomotor syn-
chrony. Those in the ‘no body’ group saw a reflection at the geometrically correct
place in the mirror, but with visuomotor asynchrony. Prior to the experiment the
racial IAT was administered, and again after the experiment. The period of
embodiment was about 12 min, during which time participants were only required
to move and look towards their body, and 12 virtual characters, half of them Black
and half of them White walked past. It was found that there was a decrease in IAT
28 M. Slater

indicating a reduction in implicit racial bias only for those who had been in a
light-skinned body. That a few minutes embodied in a Black virtual body could
reduce implicit racial bias seemed unlikely, however, using only a black rubber arm
in the RHI, Maister, Sebanz, Knoblich, and Tsakiris (2013) found a similar result.
In a subsequent study Banakou, PD, and Slater (2016) again embodied 90 White
female participants in a White or Black body. Each had 1, 2 or 3 exposures with
visuomotor synchrony following the movements of a virtual Tai Chi teacher.
However, the IAT was administered first one week before the initial exposure, and
one week after the final exposure. It was again found that those in the Black virtual
body showed a clear reduction in implicit bias, independently of the number of
exposures, whereas this did not occur for those in the White virtual body. This
indicates that the reduction in racial bias may be sustained, and that a single
exposure is sufficient for this.
The above relates only to implicit attitudes, however, changes have been found
due to embodiment in other domains. Kilteni, Bergstrom, and Slater (2013)
embodied 36 people in one of two different bodies: 18 in a body resembling Jimi
Hendrix (dark skinned, casually dressed) and the rest in a formally dressed
light-skinned body. Their task was to play hand drums. The virtual hand drums
were registered in space with real hand-drums. There was visuomotor synchrony
since participants would see their virtual hands moving with their real hand
movements, both directly and in a mirror, and visuotactile synchrony since as their
hand was seen to touch the virtual drums they would feel this because their real
hand would simultaneously strike the real drum. Through real-time motion capture
the extent to which participants moved their upper body was recorded (they were
seated with their legs holding the drums in place). It was found that those in the
more casual, dark-skinned body exhibited much greater body movement than those
in the formal light-skinned body. Participants had no idea about the purpose of the
experiment, since it was between-groups and therefore they could not know about
the other type of body used.
The drumming example is concerned with behavioral change. However, strong
perceptual changes can also be induced with transformed body ownership. van der
Hoort, Guterstam, and Ehrsson (2011), using the technique of streaming video data
to an HMD, embodied people in a Barbie doll body or a giant manikin body. This
was used together with synchronous visuotactile stimulation. They observed that in
the first case subjects overestimated object sizes and in the second case underes-
timated. Banakou, Groten, and Slater (2013) took this one step further using virtual
embodiment to show that the form of the body and not just the size influences this
perceptual result. Adult participants were embodied either the body of a child of
about 5 years old or in an adult shaped body but shrunk down to the same size as
the child. Visuomotor synchrony was used in addition to seeing the body from first
person perspective and in a mirror. The same result was found, that on the average
participants in these small bodies overestimated object sizes. However, the degree
of overestimation of those in the child body was around double that of those in the
shrunk down adult body. Moreover using an IAT that tested between
self-attribution of child-like or adult-like attributes, those in the child body
2 Implicit Learning Through Embodiment in Immersive Virtual Reality 29

condition self attributed more towards the child-like than the adult-like. In another
condition that used visuomotor asynchrony all these differences disappeared.
Our next example illustrates a cognitive change. Participants were asked to
explain a pressing personal problem to a character on the other side of a virtual
room (Osimo, Pizarro, Spanlang, & Slater, 2015). They were embodied in a virtual
body that was a very close likeness of their real body. The virtual person to whom
they explained their problem (referred to as the Counselor) was either a virtual
rendition of Dr. Sigmund Freud, or another copy of themselves. After explaining
their problem they were shifted to the body of the Counselor and saw and heard
their initial incarnation explaining the problem. Then as the Counselor they could
reply back to themselves offering advise as to how to resolve the problem, after
which they were re-embodied back in their original body as themselves, and saw
and heard (with a disguised voice) a replay of the advice of the Counselor (in fact
their own advice). They could keep swapping back and forth between
self-embodiment or embodiment as the Counselor hence maintaining a conversa-
tion. Therefore this setup was an objectification of ‘talking to oneself’ except that
the ‘one’ spoken to was represented as a virtual person. In each embodiment the
participant would see their virtual body directly from first person perspective and in
a mirror. The conditions of the experiment were that the Counselor was a copy of
themselves or of Freud, and that as Counselor they experienced visuomotor
asynchrony or synchrony. It was found that although in all conditions there was an
improvement in the rating of their mood in relation to the person problem, the best
improvement was when they were embodied as Freud with visuomotor synchrony.
This produced a strong level of body ownership and agency with respect to the
Freud body. In other words being embodied as Freud gave them access to different
mental resources, allowing them to better able move towards a solution to their
problem, than being embodied as another copy of themselves.
In the next section we consider these findings in the context of implicit learning.

2.3.3 Embodiment and Implicit Learning

All of the above examples show how transformed body ownership can lead to
changes. Although this was not the goal of those studies, for the purposes of
exposition we can reformulate them as ‘learning’ studies. For example, to learn how
to show less racial bias in an IAT, to learn how to resolve a personal problem
through talking to different representations of yourself, to learn how to play the
drums with greater vigor.
In Sect. 2.2.2 we saw that there are a number of criteria that implicit learning has
to satisfy: what is learned is non-conscious, it is complex, it is not as a result of
hypothesis testing, and it is not based on episodic memory. If we take the racial bias
as an example, there is no conscious attempt in the procedure to influence implicit
racial bias. In fact participants in these between-group experimental designs only
experience one body type. What is being learned is complex—an IAT is a complex
30 M. Slater

response-time based procedure. There is clearly no hypothesis testing or deliberate

attempt to learn something based on episodic memory. People simply have an
experience.
In these cases the effects are implicit, but the information that generates the
implicit changes is seemingly quite different from implicit learning. For the latter the
data for learning is directly and closely related to what is to be learned—e.g.,
sentences generated by a specific grammar. In the case of body ownership the
information is only indirectly related. In the self-talk example—the self-conversation
with Freud—no coaching whatsoever was given about how to address personal
problems in a more productive way.
Our hypothesis is that simply having experiences from another perspective
embodied in another body results in implicit change. When the body is related to an
‘issue’ (e.g., racism) then the change will be within that domain, even though there
is no explicit information whatsoever about racial bias in the experience. This also
conforms to the view of Gallagher (2005) who discusses the relationship between
body and mind. When the type of body conforms to what is to be learned then this
will aid learning over and above what can be learned through explicit learning.

2.4 Summary

This paper has reviewed the major concepts of virtual reality, in particular presence
(the illusion of being in the virtual place, and the illusion that events there are really
happening) and also how virtual reality can be used to produce virtual body own-
ership—a virtual body coincident in space with the real body and seen from first
person perspective can generate the illusion that it is the person’s body. We have
reviewed how presence has been used in education, and also discussed the notion of
implicit learning. Our discussion then led to the observation that virtual body
ownership can lead to implicit changes, so that, for example, when a person
embodies a body of different race their implicit bias against people of that race may
decrease. This is similar to what happens in implicit learning, even though the
approaches are quite different. We then reached the conclusion that learning may be
enhanced when participants are embodied in a virtual body that is appropriate for
that type of body. For example, someone learning how to be an orchestra conductor
might best be embodied in the body resembling, for example, Leonard Bernstein;
learning opera Luciano Pavarotti or Maria Callas; or learning ballet Natalia Osipova.
This remains an interesting hypothesis to test with controlled experimental studies.

Acknowledgements Some of the work described in this article was the product of various funded
projects: Ser Einstein: la influencia de internalizar un cuerpo virtual en la inteligencia, Ministerio
de Economía, Industria, y Competitividad, Spain (PSI2014-56301-R); Virtual Embodiment and
Robotic Re-Embodiment (VERE) European Union FET (#257695); Transcending Reality—
Activating Virtual Environment Responses through Sensory Enrichment (TRAVERSE) European
Research Council (ERC) Advanced Grant (#227985); BEAMING, Augmented Multi-Modal
Naturally-Networked Gatherings, FP7 EU collaborative project (#248620).
2 Implicit Learning Through Embodiment in Immersive Virtual Reality 31

References

Agus, T. R., Thorpe, S. J., & Pressnitzer, D. (2010). Rapid formation of robust auditory memories:
insights from noise. Neuron, 66(4), 610–618.
Bailenson, J., Patel, K., Nielsen, A., Bajscy, R., Jung, S.-H., & Kurillo, G. (2008a). The effect of
interactivity on learning physical actions in virtual reality. Media Psychology, 11(3), 354–376.
Bailenson, J., Yee, N., Blascovich, J., Beall, A. C., Lundblad, N., & Jin, M. (2008b). The use of
immersive virtual reality in the learning sciences: Digital transformations of teachers, students,
and social context. The Journal of the Learning Sciences, 17(1), 102–141.
Banakou, D., Groten, R., & Slater, M. (2013). Illusory ownership of a virtual child body causes
overestimation of object sizes and implicit attitude changes. PNAS, 110, 12846–12851. doi:10.
1073/pnas.1306779110.
Banakou, D., PD, H., & Slater, M. (2016). Virtual embodiment of White People in a black virtual
body leads to a sustained reduction in their implicit racial bias. Frontiers in Human
Neuroscience, 10, 601. doi:10.3389/fnhum.2016.00601.
Banakou, D., & Slater, M. (2014). Body ownership causes illusory self-attribution of speaking and
influences subsequent real speaking. PNAS, 111(49), 17678–17683. doi:10.1073/pnas.
1414936111.
Bell, M. D., & Weinstein, A. (2011). Simulated job interview skill training for people with
psychiatric disability: Feasibility and tolerability of virtual reality training. Schizophrenia
Bulletin, 37(suppl 2), S91–S97.
Bem, D. J. (1972). Self-Perception Theory. Advances in Experimental Social Psychology, 6, 1–62.
Blanke, O., Slater, M., & Serino, A. (2015). Behavioral, neural, and computational principles of
bodily self-consciousness. Neuron, 88(1), 145–166.
Botvinick, M., & Cohen, J. (1998). Rubber hands ‘feel’ touch that eyes see. Nature, 391(6669),
756. doi:10.1038/35784.
Cargill, R. R. (2008). The Qumran digital model: An argument for archaeological reconstruction
in virtual reality. ProQuest.
Dede, C., Salzman, M., Loftin, R. B., & Ash, K. (1997). Using virtual reality technology to convey
abstract scientific concepts. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=
10.1.1.136.4289&rep=rep1&type=pdf.
Foti, F., De Crescenzo, F., Vivanti, G., Menghini, D., & Vicari, S. (2015). Implicit learning in
individuals with autism spectrum disorders: A meta-analysis. Psychological Medicine, 45(05),
897–910.
Fowler, C. (2015). Virtual reality and learning: Where is the pedagogy? British Journal of
Educational Technology, 46(2), 412–422.
Freeman, D., Evans, N., Lister, R., Antley, A., Dunn, G., & Slater, M. (2013). Height, social
comparison, and paranoia: An immersive virtual reality experimental study. Psychiatry
Research, 213(3), 348–352. doi:10.1016/j.psychres.2013.12.014.
Freina, L., & Ott, M. (2015). A literature review on immersive virtual reality in education: State of
the Art and perspectives. In Proceedings of eLearning and Software for Education (eLSE)
(Bucharest, Romania, April 23–24, 2015).
Gallagher, S. (2005). How the body shapes the mind. Cambridge: Cambridge University Press.
Greenwald, A. G., McGhee, D. E., & Schwartz, J. L. K. (1998). Measuring individual differences
in implicit cognition: The implicit association test. Journal of Personality and Social
Psychology, 74, 1464.
Groom, V., Bailenson, J. N., & Nass, C. (2009). The influence of racial embodiment on racial bias
in immersive virtual environments. Social Influence, 4, 231–248.
Hwang, W.-Y., & Hu, S.-S. (2013). Analysis of peer learning behaviors using multiple
representations in virtual reality and their impacts on geometry problem solving. Computers &
Education, 62, 308–319.
32 M. Slater

Kaufmann, H., Schmalstieg, D., & Wagner, M. (2000). Construct3D: A virtual reality application
for mathematics and geometry education. Education and information technologies, 5(4), 263–
276.
Kessels, R. P., & Haan, E. H. (2003). Implicit learning in memory rehabilitation: A meta-analysis
on errorless learning and vanishing cues methods. Journal of Clinical and Experimental
Neuropsychology, 25(6), 805–814.
Kilteni, K., Bergstrom, I., & Slater, M. (2013). Drumming in immersive virtual reality: The body
shapes the way we play. Transactions on Visualization and Computer Graphics, 19, 597–605.
doi:10.1109/TVCG.2013.29.
Kokkinara, E., & Slater, M. (2014). Measuring the effects through time of the influence of
visuomotor and visuotactile synchronous stimulation on a virtual body ownership illusion.
Perception, 43(1), 43–58. doi:10.1068/p7545.
Lanier, J. (2006). Homuncular flexibility. Edge: The World Question Center. Accessed November,
26, 2012.
Lin, H., Chen, M., Lu, G., Zhu, Q., Gong, J., You, X., et al. (2013). Virtual Geographic
Environments (VGEs): A new generation of geographic analysis tool. Earth-Science Reviews,
126, 74–84.
Maister, L., Sebanz, N., Knoblich, G., & Tsakiris, M. (2013). Experiencing ownership over a
dark-skinned body reduces implicit racial bias. Cognition, 128, 170–178. doi:10.1016/j.
cognition.2013.04.002.
Masters, R., Lo, C., Maxwell, J., & Patil, N. (2008). Implicit motor learning in surgery:
Implications for multi-tasking. Surgery, 143(1), 140–145.
Moseley, G. L., Olthof, N., Venema, A., Don, S., Wijers, M., Gallace, A., et al. (2008).
Psychologically induced cooling of a specific body part caused by the illusory ownership of an
artificial counterpart. Proceedings of the National Academy of Sciences of the United States of
America, 105, 13169–13173. doi:10.1073/pnas.0803768105.
Müns, A., Meixensberger, J., & Lindner, D. (2014). Evaluation of a novel phantom-based
neurosurgical training system. Surgical Neurology International, 5, 173.
O’Regan, J. K., & Noë, A. (2001). A sensorimotor account of vision and visual consciousness.
Behavioral and Brain Sciences, 24, 939–1031.
Osimo, S. A., Pizarro, R., Spanlang, B., & Slater, M. (2015). Conversations between self and self
as Sigmund Freud—A virtual body ownership paradigm for self counselling. Scientific
Reports, 5, 13899. doi:10.1038/srep13899.
Pan, X., Gillies, M., Barker, C., Clark, D. M., & Slater, M. (2012). Socially anxious and confident
men interact with a forward virtual woman: An experimental study. PLoS ONE, 7, e32931.
Peck, T. C., Seinfeld, S., Aglioti, S. M., & Slater, M. (2013). Putting yourself in the skin of a black
avatar reduces implicit racial bias. Consciousness and Cognition, 22, 779–787. doi:10.1016/j.
concog.2013.04.016.
Petkova, V. I., & Ehrsson, H. H. (2008). If I Were You: Perceptual illusion of body swapping.
PLoS ONE, 3, e3832. doi:10.1371/journal.pone.0003832.
Reber, A. S. (1967). Implicit learning of artificial grammars. Journal of Verbal Learning and
Verbal Behavior, 6(6), 855–863.
Reber, A. S. (1989). Implicit learning and tacit knowledge. Journal of Experimental Psychology:
General, 118(3), 219.
Reber, P. J. (2013). The neural basis of implicit learning and memory: A review of
neuropsychological and neuroimaging research. Neuropsychologia, 51(10), 2026–2042.
Roussou, M. (2009). A VR playground for learning abstract mathematics concepts. Computer
Graphics and Applications, IEEE, 29(1), 82–85.
Salomon, R., Lim, M., Pfeiffer, C., Gassert, R., & Blanke, O. (2013). Full body illusion is
associated with widespread skin temperature reduction. Frontiers in Behavioral Neuroscience,
(submitted).
2 Implicit Learning Through Embodiment in Immersive Virtual Reality 33

Sanchez-Vives, M. V., & Slater, M. (2005). From presence to consciousness through virtual
reality. Nature Reviews Neuroscience, 6, 332–339.
Seger, C. A. (1994). Implicit learning. Psychological Bulletin, 115(2), 163.
Slater, M. (2009). Place Illusion and Plausibility can lead to realistic behaviour in immersive
virtual environments. Philosophical Transactions of the Royal Society of London, 364, 3549–
3557.
Slater, M., Perez-Marcos, D., Ehrsson, H. H., & Sanchez-Vives, M. (2008). Towards a digital
body: The virtual arm illusion. Frontiers in Human Neuroscience, 2. doi:10.3389/neuro.09.
006.2008.
Slater, M., & Sanchez-Vives, M. V. (2016). Enhancing our lives with immersive virtual reality.
Forthcoming.
Slater, M., Spanlang, B., Sanchez-Vives, M. V., & Blanke, O. (2010). First person experience of
body transfer in virtual reality. PLoS ONE, 5(5), e10564–e10564. doi:10.1371/journal.pone.
0010564.
van der Hoort, B., Guterstam, A., & Ehrsson, H. H. (2011). Being Barbie: The size of one’s own
body determines the perceived size of the world. PLoS ONE, 6, e20195.
Vine, S. J., Masters, R. S., McGrath, J. S., Bright, E., & Wilson, M. R. (2012). Cheating
experience: Guiding novices to adopt the gaze strategies of experts expedites the learning of
technical laparoscopic skills. Surgery, 152(1), 32–40.
Wilson, M. R., Vine, S. J., Bright, E., Masters, R. S., Defriend, D., & McGrath, J. S. (2011). Gaze
training enhances laparoscopic technical skill acquisition and multi-tasking performance: A
randomized, controlled study. Surgical Endoscopy, 25(12), 3731–3739.
Yee, N., & Bailenson, J. N. (2007). The Proteus effect: The effect of transformed
self-representation on behavior. Human Communication Research, 33, 271–290.

Author Biography

Mel Slater DSc is an ICREA Research Professor at the University of Barcelona in the Faculty of
Psychology. He has been Professor of Virtual Environments at University College London since
1997 in the Department of Computer Science. He has been involved in research in virtual reality
since the early 1990s, and has been first supervisor of 36 PhDs in graphics and virtual reality since
1989. In 2005 he was awarded the Virtual Reality Career Award by IEEE Virtual Reality ‘In
Recognition of Seminal Achievements in Engineering Virtual Reality.’ He has been involved in
and led several international projects in this field. He held a European Research Council grant
TRAVERSE. He is Field Editor of Frontiers in Robotics and AI, and Chief Editor of the Virtual
Environments section. He has contributed to the scientific study of virtual reality and to technical
development of this field. He has aimed to bring the use of virtual reality as a tool in scientific
research in these areas to the highest level—for example, with publications in PNAS, Neuron,
Trends in Cognitive Science, Philosophical Transactions of the Royal Society, Nature Reviews
Neuroscience, and ACM Transactions on Graphics (SIGGRAPH).
Chapter 3
Authenticity in Immersive Design
for Education

Jeffrey Jacobson

Abstract Authenticity, is a concept found in both media design and educational

design, usually as a quality needed for success. Here, we develop a theory of
authenticity for educational experiences with immersive media (VR, MR, MUVEs,
etc.) to help educators and authors in this new field. In our framework, authenticity
refers to the relationship between a truth and its representation, guided by a pur-
pose. By truth, we refer to a fact, concept, or procedure, about something in the
world or in the body of human knowledge, something we want to learn. To scaffold
the learning process, students require a representation of the thing. It may be a
written article (for concepts), an image (e.g., a photograph), or maybe an exemplar
(an idealized example of a category). A representation or an experience is said to be
authentic, when it successfully captures the fundamental truth of what we are
learning. The immersive media have unique capabilities and just in the last few
years have become available to the public on a large scale. Our theory is not a
comprehensive style guide, but a practical way to look at one key dimension of
good educational design.

Keywords Authenticity
Elegance
Pedagogy
Design
Immersive
VR

3.1 Introduction

In this chapter, we develop a theory of authenticity as an essential quality of edu-

cational experiences with immersive media (VR, MR, MUVEs, etc. see Chap. 1).
We produced it to be an analytical tool for educators and designers, to help them
build effective experiences for their students. It is not a comprehensive style guide
for authoring, but illustrates a crucial dimension of good design. It will help with
the lack of guiding theory in education in immersive media (Fowler, 2015).

J. Jacobson (&)
CEO, EnterpriseVR, 333 Lamartine St., Jamaica Plain, MA 02130, USA
e-mail: [email protected]

© Springer Nature Singapore Pte Ltd. 2017 35

D. Liu et al. (eds.), Virtual, Augmented, and Mixed Realities in Education,
Smart Computing and Intelligence, DOI 10.1007/978-981-10-5490-7_3
36 J. Jacobson

In our framework, authenticity refers to the relationship between a truth and its
representation, guided by a purpose. By truth, we refer to a fact, concept, or
procedure, about something in the world or in the body of human knowledge,
something we want to learn. To scaffold the learning process, students require a
representation of the thing. It may be a written article (for concepts), an image (e.g.,
a photograph), or maybe an exemplar (an idealized example of a category).
A representation or an experience is said to be authentic, when it successfully
captures the fundamental truth of what we are learning. For example:
Suppose we create an animated 3D model of a bird design to help us to teach the flight
dynamics of its species. In that case, the representation does not need to be highly detailed
—a simple, flexible, 3D model will do, as long as its proportions are right (Fig. 3.1).
However, the movement of that model must very accurately depict the movements of a real
bird of that type. On the other hand, if the goal is to learn exactly what that bird looks like
(feathers, beak, skin, etc.) then the representation must have a lot of physical detail, but its
motion would not be relevant. It may not need to move at all (Fig. 3.2).

Importantly, authenticity is not realism. Usually, a representation is said to be

more realistic the better it mimics the real thing in every way. Sometimes, that is
what we need for authenticity, but more often, we do not need it.
In our discussion, we will call this the principle of truth (Sect. 3.3.1), that the
relevant facts or ideas are represented in the virtual environment depicted by the
immersive medium. However, we must understand the purpose of the learning
experience to know which truth we are trying to represent (Sect. 3.3.1). Further, we
strive for the right level of detail (Sect. 3.3.2), meaning that the representation
shows what it needs to show, but without extraneous detail. Finally, the design
should be elegant (Sect. 3.3.3), meaning that the different representations in the
design fit together and function well.

Fig. 3.1 Artful model of a

bird, low resolution, but well
proportioned. Suitable for
animation to demonstrate
movement. http://clipart-
library.com/clipart/
XrcjKGXTR.htm
3 Authenticity in Immersive Design for Education 37

Fig. 3.2 High resolution

drawing of a bird, suitable for
study of detail. http://clipart-
library.com/clipart/
ATbr7BLRc.htm

But first, we will look at existing theories of learning in educational media,

because the purpose comes from the learning objective and the pedagogies that help
us get there. Specifically, immersive media is well suited to constructivist
approaches, where students build their own learning experience within the learning
environment as guided by the teacher (Duffy & Jonassen, 2013). Also, Winn’s
(2003) framework for learning in VR is an excellent tool to focus that approach. We
will use both to guide our discussion.
Versions of the concept of authenticity is also found in writings throughout
media design (Kronqvist, Jokinen, & Rousi, 2016), and in education (Strobel,
Wang, Weber, & Dyehouse, 2013) (Herrington & Parker, 2013). We will explore
those theories, describe our own frame with examples, and finally list the topics
where immersive media are most likely to be useful (Sect. 3.6).

3.2 Existing Concepts of Authenticity and Learning

Good design in educational media a broad topic, and achieving high quality is
usually more art than science. This leads authors to employ (and create) frameworks
of design advice, to point out the key issues and provide practical ways to develop
solutions. These guides are usually specialized by topic, media type, and purpose,
although some can be quite broad. Others have a particular theme or theoretical
approach. In this section we will look at some of the existing theories of authen-
ticity and learning.
38 J. Jacobson

Kronqvist et al. (2016) define authenticity as a function of the degree of pres-

ence, affordance, and control the environment provides to the user. William Winn
(2003) describes learning in VR as an adaptive response to a challenging task or
environment. The learner is embedded in the virtual environment while s/he is
embodied there. Both approaches apply reasonably well to the other immersive
media. We also reference constructivism, which posits that the learner constructs his
or her own learning experience, within the learning environment. Constructivism is
the most widely cited pedagogical approach in the historical literature on educa-
tional VR (Duffy & Jonassen, 2013).
Then, we look at two theories of authenticity in education, one by Strobel et al.
(2013) and another by Herrington and Parker (2013). Both frameworks emphasize
the search for a way to convey the fundamental or foundational truth of the topic to
the student in way that is respectful of the material. They also contain much
practical advice, specific to their fields. We are fortunate to have their work,
because the idea of authenticity is widely cited in the education literatures, but often
without adequate definition or theoretical foundation (Strobel et al., 2013).
Readers who wish to go deeper should look at Barab, Squire and Dueber (2000)
and their work on authenticity as an emergent property of good practice in training.
Donnelly (2014) provides a practical design guide for authenticity in architectural
design.

3.2.1 Kronqvist’s Authenticity in Virtual Environments

Kronvqvist et al. (2016) created a metric for measuring authenticity in virtual

environments, when accessed through a VR-like interfaces (e.g. HMD, or big
screen). They developed a post-test questionnaire to measure it, across several types
of interfaces. Their goal is to give VR researchers a way to verify that their virtual
environments are sound.
They define an affordance to be a quality of the virtual environment (VE), which
is the degree to which representations of (objects and phenomena) in the VE
provide the same affordances as the real thing. It allows user to perform their
learning tasks much as they would in real life. This implies that we must know what
the purpose of the simulation is, in order to define which affordances should be
captured. (See Slater’s Plausibility Illusion, Chap. 2). That makes Kronqvist’s af-
fordance similar to our own definition of authenticity. However, his definition of
authenticity includes the sense of presence in the user (equivalent to Slater’s Place
Illusion, Chap. 2), but our definition does not.
3 Authenticity in Immersive Design for Education 39

3.2.2 Winn’s Framework for Learning in Sensorially

Immersive VR

Winn’s approach (Winn, 2003) states how authentic learning can work in an
immersive environment, even though he did not use the word, authenticity. While
his focus was on VR, the principles apply well to the other immersive media. He
proposes that optimal learning requires (quote):
Embeddedness: The mental and physical tasks a person performs cannot be defined
without some reference to the environment. This does not imply that the environment
defines cognition, but that some reference to the environment is required to describe it.
Embodiment: The learner/actor’s physical body is an essential part of the process,
because everything we perceive must be mediated through our limited senses. We directly
use our bodies to accomplish most tasks and the brain must be regarded as an organ of the
body.
Dynamic Adaptation: In a changing environment, we must continually adapt to the
changing circumstances. It is also true that the environment changes in response to the
person’s actions. In this way the individual and his or her environment are evolving
together, responding to each other.

Embeddedness would certainly apply to many or most immersive learning

environments, especially those with high authenticity. The embodiment clause
requires an engaging physical environment, a good mixed reality, or a fully
immersive virtual reality. In Slater’s terms (Chap. 2) embeddedness requires what
he calls place illusion, and dynamic adaptation requires what he calls plausibility
illusion, as well as an instructional strategy. Slater’s chapter also explores
embodiment.
However, Winn’s framework is only partial, with respect to the overall enter-
prise of educating and learning. For example, it does not address assessment, which
should be hidden within the learning activities and the environment itself (Shute
et al., Chap. 5; Dede, 2012) to avoid distraction from the learning task.

3.2.3 Constructivism in VR for Education

Ideally, learning as an active process, which the student constructs within the
learning environment, to achieve the required goals, and complete the learning task,
all under the teacher’s supervision. This is the constructivist approach to learning
theory, which is widely cited in the literature on educational immersive media.
Describing it further is beyond the scope of this chapter, but we recommend (Duffy
& Jonassen, 2013) for a good description. Also, Gardner and Sheaffer (Chap. 9)
and Shute et al. (Chap. 5) discuss constructivism in their chapters.
40 J. Jacobson

3.2.4 Strobel’s Framework for Authenticity in Engineering

Education

Strobel et al. (2013),1 distills the idea of authenticity found in the literature on
education for engineering. The consensus, there, is to bring the learner closer to the
realities of the workplace, with principles, summarized here:
Context Authenticity: Context resembles real-world context [….]
Task Authenticity: Activities of students resemble real-world activities [….]
Impact Authenticity: Student work is utilized in out-of school situations [….]

Srobel proposes two more dimensions that come from the Applicative or
Sociocultural Perspective:
Personal Authenticity: projects are close to students’ own life. (i.e. life-stories of their
neighborhood, biodiversity in the forest nearby)
Value Authenticity: personal questions get answered or projects satisfy personal or
community needs.

They recommend that learning environments should have the above character-
istics, and others not included in our summary.

3.2.5 Herrington’s Framework for Authenticity

Herrington, Reeves and Oliver (2009, p. 17) propose a framework of authentic

e-learning, summarized, here:
• Provide authentic contexts that reflect the way the knowledge will be used in
real life.
• Provide authentic activities.
• Provide access to expert performances and the modeling of processes.
• Provide multiple roles and perspectives.
• Support collaborative construction of knowledge.
• Promote reflection to enable abstractions to be formed.
• Promote articulation to enable tacit knowledge to be made explicit.
• Provide coaching and scaffolding by the teacher at critical times.
• Provide for authentic assessment of learning within the tasks.
They recommend that learning environments have the above characteristics, and
others not included in our summary.

1
Strobel relies heavily on theories by Anderson, Billet, and Buxon, and cites them properly. We do
not have the space to repeat those citations, here.
3 Authenticity in Immersive Design for Education 41

3.3 Our Framework for Authenticity in Educational

Immersive Media

In this chapter, we develop our theory of authenticity, as a way to think about the
design of immersive educational experiences. It is centered on the idea that for a
simulation or representation to be authentic, it must capture the basic truth of what it
represents. However, you also have to know what the instructional goal of the
experience is, before you can decide what truths you want to represent.
Herrington et al. (2009), Herrington and Parker (2013), Strobel et al. (2013),
Kronqvist et al. (2016) each place the same requirement on their definitions of
authenticity. However, Kronqvist’s definition of authenticity is broad, including
presence for authenticity in immersive media, which we do not. Strobel’s
authenticity includes design advice for education in engineering, and Herrington’s
authenticity does the same for e-learning.
Our theory of authenticity is meant to be a helpful tool for designers and users of
immersive educational experiences, but not a comprehensive guiding theory.
Instead, we propose authenticity as one dimension of good design, one tool among
several. The key components are:
• Purpose
• Truth
• Elegance
• Continuity
We will describe these, in turn, and how they work together. Then, we will
discuss where authenticity resides and how to measure it.

3.3.1 Purpose and Truth

As with so many endeavors, it begins with the purpose, our learning goals, which
drive everything else in the design. Generally, we want successful transfer, where
the student learns something from the immersive media that s/he successfully
applies elsewhere (Grotzer et al., 2015). The most straightforward examples are
simulations, like this one:
A good way to begin learning how to drive a car is to use a driving simulator. There, the
student pilots a virtual automobile through increasingly challenging lessons. The virtual
car must behave like a real one, and have realistic controls, to support the training.
However, neither the virtual car nor the virtual environments has to look good. Even
relatively crude models will do, as long as they are correctly proportioned and readable
(Fig. 3.3).

This illustrates how purpose defines the next component of our framework,
which we call the truth. That is the relationship between the real thing and its
42 J. Jacobson

Fig. 3.3 Driving simulator (Teen Driver Source, Children’s Hospital of Philadelphia). http://
www.teendriversource.org/images/drivingsimulator.jpg

simulation in the virtual environment. We want the simulation to capture or rep-

resent some fundamental truth about the real thing.
In our driving simulator example, the need to learn how to drive dictates which aspects of
the car we want to simulate. If, instead, we wanted to teach the student how to decorate a
car, we wouldn’t care much about how the simulation behaves, only how it looks.

This applies to experiences, not just objects. For example:

Suppose the student is using a VR simulation to overcome a fear of public speaking. S/he
sees herself in an auditorium room filled with automated human figures (bots). At the
simplest level, she speaks for a set period of time, giving a short speech, while the virtual
audience either sits in silence or responds in a positive or negative way. After each interval,
the student provides feedback to the software indicating her level of discomfort and her
desire to continue. Based on the curriculum built into the software, the student is pro-
gressively challenged with an increasingly difficult audience (Pertaub, Slater, & Barker,
2002) Fig. 3.4.
In this case, the bots only need to be recognizably human, capable of expressing emotion
through simple facial expressions or even body language, alone. This requires that their
motions be authentic in terms of what they convey, but the actual appearance of their bodies
can be quite simple. In the physical world, puppeteers use this to great effect (Engler &
Fijan, 1997).
3 Authenticity in Immersive Design for Education 43

Fig. 3.4 Two audience reaction scenarios presented by a VR trainer for public speaking (Pertaub
et al., 2002). https://publicspeaking.tech/wp-content/uploads/2016/11/Pertuab_etal_PublicSpeaking
Anxiety.jpg

The learning experience is effective if it induces the required feelings of dis-

comfort, and provides the student with the opportunity to overcome it. It is
authentic if it is well focused on the lesson, and only the lesson, in an elegant way.
The truth in the above example is that public speaking won’t hurt you, the
representation is the audience and its reactions, and the purpose is to overcome
one’s fear of speaking.
See Kraemer (Chap. 4) for a deeper exploration of how students can learn from
virtual social immersion with sophisticated bots and avatars. In Sect. 3.4, we will
discuss where authenticity resides, and how to measure it, but first, we will com-
plete the description of our framework.
44 J. Jacobson

3.3.2 Level of Detail

In every media, the author chooses which level of detail is most appropriate for his
or her purpose. S/he may present a large number of details in order to convey
complex information, create a mood, build an argument, or for other reasons. For
example, a scientific visualization may require complex forms to represent the data
fairly. On the other hand, certain traditional Japanese paintings convey a great deal
of information with just a few clean lines. Also, a high level of detail is not the same
thing as realism. Many representations that are not realistic have a lot of detail in
them, while some realistic depictions don’t need much detail.
Usually, an element will belong in the design, because:
• It conveys something about the topic to be learned. (For example, if the topic is
chemistry, we might see models of molecules.)
• It makes the experience more aesthetically pleasing, even if the author is not
explicitly making art.
• It helps the experience function.

For example, a student might be using a mixed reality program to examine (representations
of) different species of frog. The command s/he uses to switch from one representation to
another doesn’t represent anything, but it is essential for the process.

Here is an example that illustrates some of these themes:

The Virtual Ancient Egyptian temple (Gillam & Jacobson, 2015, Chaps. 3 & 7) (Fig. 3.5)
was built to be a symbolic representation of a late period cult temple. Our virtual temple had
the minimum architectural features required to properly house a representative sample of
Egyptian ceremonies and public events. The temple also contains hieroglyphics and art-
work on the walls, arranged in the correct order, saying the right things, to typify an ancient
Egyptian temple.
The virtual temple (minimally) embodies the key characteristics of late period Egyptian cult
temples, but not a particular one. It is an exemplar, an idealized model used to describe a
category of things (Barsalou, 1992). Had we faithfully simulated one particular, real,
temple, its idiosyncratic features would have misled our readers into thinking they were
universal. Instead, a less realistic, idealized, temple is paradoxically a better exemplar.
However, we did add some naturalistic lighting and material effects, just to make the space
more comfortable and readable. We also (unrealistically) enlarged the hieroglyphics to
make them readable in the computer displays available at the time.

Level of detail can refer to action and motion, as well as the appearance of
things. Japanese anime’ often has only the most important elements in the scene
moving, an example of minimalism in movement. In immersive media, especially
VR, one must be careful to present enough motion in the virtual environment so it
feels true to the things it depicts. But do not use so much motion that the user
become disoriented, distracted, or annoyed.
3 Authenticity in Immersive Design for Education 45

Fig. 3.5 The Hypostyle Hall of the Virtual Egyptian Temple (Gillam & Jacobson, 2015). http://
publicvr.org/images/HypostylePromoShot.jpg

3.3.3 Elegance

When the learner’s experience is elegant, we mean that is achieves its purpose
artfully, with the right level of detail. The experience is well integrated with its
environment, well-coordinated with other instructional materials, internally con-
sistent, sensitive to its audience, and consistent with it purpose. The narrative design
must fit well within in the larger curriculum. (Dede (Chap. 8) and Schneider
(Chap. 12) both discuss what stages of the student’s learning process are more
likely to benefit from immersive media and those that are not.)
For example, the authenticity of any historical or archaeological digital simulation partly
depends on how well it fits with other materials and media that describe the same thing.

In immersive media, the user’s physical or pseudo-physical location in the

virtual environment is a critical part of the narrative and usually is the foundation of
how they interact with it. (See Winn’s embeddedness, above).
46 J. Jacobson

In the mixed reality that Gardner and Sheaffer describe in Chap. 9, some of the students are
in a physical space, while others are in a MUVE, but the two spaces are connected through
a kind of portal wall. This allows the students “in” the MUVE to participate in the
classroom activities sufficiently, so they apparently learn just as well as the students who
are there, physically.

An elegant design should also exhibit continuity in its use of detail and other
factors. We do not expect a written document to have too many different fonts. We
do not expect a movie to speed up or slow down during the presentation, nor do we
expect the picture quality to change abruptly. That is why most works of most
media usually have a similar level of detail throughout; but there can be exceptions,
if skillfully handled. For example, there may be some moment or object in the
representation that is rendered at a deliberately higher or lower level of detail,
specifically to focus attention, create a mood, or convey information.
Two excellent examples of elegance in authentic design are the mixed realities of
Johnson-Glenberg (Chap. 11) and Schneider (Chap. 12). In both cases, they add just
enough virtualized information for a profound impact, exploiting the advantages of mixed
reality (MR). They take it a step further by designing unique and tangible interfaces that
embody the information the student must learn and the action s/he must take to do so.

Elegance is not minimalism. The idea behind minimalism is to use as few

elements as absolutely possible to achieve the purpose in the simplest way. This is a
valid design choice, but an elegant design does not have to be so strict. It may
contain more elements than necessary for aesthetic or functional reasons.
Suppose you want to show medical students what it is like to be in a hospital operating
room during a surgery. An example of the type is shown in Fig. 3.6 (Kapralos, Moussa, &
Dubrowski, 2014).

Fig. 3.6 VR simulation of a hospital operating room with user interaction (Kapralos et al., 2014)
3 Authenticity in Immersive Design for Education 47

Suppose further that the student could take on the role of a nurse or doctor. If the goal of the
simulation is to teach operating room procedure, then it must accurately show those pro-
cedures. That would require only a moderate level of detail in the models, the virtual
people, and how they move.
However, additional details might create a more comfortable experience and help the user
focus. For example, the simulated people, the patient, the doctor, and staff, should all look a
little different, which is more information for the system to implement. But if everyone in
the room were identical, it would be distractingly strange.

Another aspect of elegance, is that the student should have a seamless and
comfortable experience. Ideally, nothing should distract them, like motion sickness,
which happens in VR, if it is not handled properly. See Richards (Chap. 6). Also,
Shute et al.’s stealth assessment methods (Chap. 5) would be very useful, because
they afford a way to test the student’s knowledge, but without interrupting their
experience in the virtual environment—an elegant solution.

3.4 Where Does Authenticity Reside and How Do We

Measure It?

Authenticity resides in both the system (the virtual environment and its interface)
and within the mind of the user. Simply put, if the builder of the virtual environ-
ment has created a representation that expresses the learning objective (a truth)
successfully, then the student will perceive that truth and be able to interact with the
representation productively.
Ideally, the ultimate test would be whether the student could demonstrate
knowledge transfer, from the learning experience to a genuine situation in real life.
However, the history of educational innovation is littered with examples of students
learning despite technology or regardless of it. Additional measures are needed to
determine the value and authenticity of the virtual environment.
Fortunately, there are many test instruments available for educational media and
immersive media that should be directly useful. But we cannot propose a generic
measure for authenticity, based on our theory alone. To evaluate some learning
experience for authenticity, we’d also have to know it’s base media (VR, MR, or
MUVE), it’s topic, and much more about its structure. What we can do is discuss
basic approaches.
One way to measure authenticity would be to embed measures integral to the
virtual environment and the experience, itself. We would be interested in whether
the student perceives the truths we want the representations to convey. Shute’s
stealth assessment techniques could be quite useful here (Chap. 5). Slater (Chap. 2)
does something similar in his tests for presence.
One could also measure how authentic the immersive learning environment is,
outside of its educational mission, using something like the instrument in Kronqvist
et al. (2016). However, he also measures presence, which our definition of
48 J. Jacobson

authenticity does not include. We treat presence (Slater, Chap. 2) as a different

dimension of immersive design, which exists alongside authenticity.
Finally, one could examine an educational immersive experience, and see how
well each element works and how well they all work together. Hopefully, they will
perform their key functions and represent the key facts in an elegant way, so the
student can focus on the learning task. In the case of virtual objects, we expect them
to behave like the real thing. Kronqvist et al. (2016) call these affordances.

3.5 Handling Uncertainty

Immersive media always use 3D models for objects and (in VR and MUVEs) the
environment itself. That creates difficulty for the author, when s/he needs to express
uncertainty or competing ideas about what things should look like. People tend
believe what they see, so they often look uncritically at immersive representations.
Usually, the author and the user, both want as much detail as possible in the
simulation, but to depict what is known, one often must complete the picture with
elements are partly speculation. Otherwise, the virtual environment won’t be use-
able. Examples:
Imagine a simulation of a crucial historical event, like the battle of Gettysburg in the US
Civil war (Fig. 3.7). We know the basic clothing the soldiers wore, their weapons, the
terrain, and much about the battle itself. But there are many gaps in our understanding.
Exactly, how many soldiers were here or there? How much of the fight was in the town, and
how much was outside of it?
If the simulation is to show individual soldiers, at any level of detail, the author must show
them moving about the battlefield. Without any other knowledge, the viewer cannot tell
how much of that comes from established historical fact verses the artist’s need to complete
the environment. But without a complete environment, the immersive medium can’t
function.
Another example: what happens when one attempts to simulate a building that is known to
be a fully enclosed space, but the character and placement of an entire wall is not known?
The author has no choice but to put something in there, which amounts to an educated
guess, because leaving a gap would be worse.

A corollary problem is how to handle competing theories of how something

should be represented.
For example, an archaeologist may discover evidence of what appears to be a single
building. While assembling the known elements, s/he may discover that some of the
archaeological evidence conflicts with other clues. Or the total evidence does not make
sense when used together in one building. A possible explanation is that the observed
evidence comes from more than one building. They may have existed in the same location,
but at different time periods.
One strategy for the instructor would be to represent each competing theory its own 3D
model, and let the students explore them. Sometimes, the competing models can be overlaid
to occupy the same space, for easier comparison.
3 Authenticity in Immersive Design for Education 49

Fig. 3.7 Simulation of the battle of Gettysburg. Not from an immersive application, but
illustrative. http://www.scourgeofwar.com/index.shtml

There are many visual techniques an immersive media designer could use to
indicate uncertainty in parts of the virtual environment. S/he could indicate
uncertain elements with a particular color, or lack of color, or lack of detail. S/he
could use labels or symbols to explain the prominence of each virtual object. Many
creative solutions are possible, but they’ll have to be handled skillfully to preserve
the system’s elegance.

3.6 When to Use Immersive Media in Education

Like any other toolset, immersive media is good for teaching some topics and less
effective for others. That begs the question: when is immersive media useful in
industry and/or society? The gross outlines are fairly straightforward, from 60 years
of experimentation with VR and other immersive media. However, the details are
only just now being worked out, as the medium is adopted by society, generally.
While the fundamentals of the technology have not changed, new applications are
invented every day on a massive scale, because of a drastic drop in cost in just the
last few years. (See Richards, Chap. 6). This will lead to both a steady evolution of
the technology and occasional breakthroughs.
50 J. Jacobson

We list many good uses of immersive media, below, although it could never be
an exhaustive accounting. Pantelidis (2010) also provides a good summary.

3.6.1 Training in Real Life Tasks

Many activities in real life do not employ immersive media, but immersive media
could be used for training in those pursuits. These are usually activities that require
procedural knowledge. Major examples are:
• Simulators of aircraft and other big dangerous machines. In fact, it was
the military Aerospace industry that invented modern VR and they have a long
track record of Success in training Airline pilots and military aviators.
• Military training has a long and productive history with VR and more recently
MR.
• Medical training, through simulation, has been a huge area of research and
development for years. Entirely physical props, mixed realities, and full VR are
all being studied.
• Basically, learning how to do anything dangerous like firefighting, first
responder to disasters, and So on.
Training is the oldest and most studied use for Virtual Reality, and the literature
on it is vast.

3.6.2 Learning Topics Usually Taught in School

Many pursuits require knowledge about things, usually facts and concepts, such as
history or physics. This is called declarative knowledge, which encompasses most
of what one learns in school. Here are a few topics that require learning a large
amount of declarative knowledge:
• Cultural history and heritage, where ancient monuments, events, and even whole
societies can be simulated, interactively (Gillam & Jacobson, 2015).
Bandt points out that aboriginal peoples Australia identified certain rock for-
mations as having spirits and magical properties. Most of those formations have
unique acoustical properties, so an immersive acoustical simulation of such as
site would be informative. See http://www.sounddesign.unimelb.edu.au/site/
NationPaper/NationPaper.html
• Astronomy, where students can explore the solar system, or reason about
observable movements of objects in the sky (Vosniadou & Brewer, 1992).
• Physics, where students can see and interact with a variety of simulations (Shute
& Ventura, 2013).
3 Authenticity in Immersive Design for Education 51

• Simulations of social situations, investigations, laboratories, and a much else

using MUVEs (Dede, Chap. 8), (Kraemer, Chap. 4), and (Gardner & Sheaffer,
Chap. 9).
Of the immersive media, MUVEs are most widely used and studied in education,
and probably have the most uses. See Klopfer (Chap. 10) and Dede (Chap. 8).

3.6.3 Clinical Psychology

Years of productive research have gone into using VR for diagnosis and treatment
of things like PTSD in adults, ADHD in children, and treatment of phobias.
For example, most PTSD and phobia treatments center on gradual exposure to what the
user fears, while keeping him/her physically safe. That helps them overcome their aversion,
and work through their cognitive dissonance around similar situations (Fig. 3.8) (Rizzo,
2016).

A great deal of literature is available; we recommend a search on the work of Dr.

Albert Rizzo (2016).

Fig. 3.8 Gradual exposure therapy to cure post-traumatic stress disorder (PTSD) (Rizzo, 2016).
https://c1.staticflickr.com/8/7609/16983316895_5e8decce03_b.jpg
52 J. Jacobson

3.6.4 Training for Direct Use in Industry

There are many industries where immersive media are directly useful, and students
will need training to use them well.
• When Mixed Reality reveals something not otherwise seen:
– Internal anatomy during surgery or diagnosis.
– Subsystems in existing buildings.
– Internals of complex machinery during maintenance.
– Threat factors in dangerous situations. For example Augmented Reality
glasses could be used to see areas of dangerous radiation inside a nuclear
power plant.
• When virtual reality can be used for collaborative or remote work:
– Users can collaborate at a distance as they engineer some complex object via
an immersive MUVE.
– Collaborators coming inside a digital dome or CAVE for a design review of
some artifact. This is more useful for structures one wants to be inside, such
as a building (Jacobson 2011, 2013).
– Orthoscopic surgery offers a kind of VR as the interface for the surgeon
trainee.
– An operator can use a VR-like interface to control a robot carrying out any
number of types of missions.
• Building community and other social uses:
– Entertainment, such as World of Warcraft and similar games with millions of
users and complex social dynamics.
– Remote meetings and socializing, as with Second Life, AltspaceVR and VR
Chat.
– Immersive news reporting can give the audience a much more powerful view
of some faraway situation. Immersive film is being used to great effect here.
• Other uses:
– First responders, security people and the military can use VR to learn the
layout of a terrain before going there, physically.

3.6.5 When Not to Use Immersive Media

Usually, one should not use immersive media when some other media will do just as
well or better. For example when one needs to understand the exterior of an object, a
3-D model visible on a computer monitor is adequate. Or when the information is
abstract, such as budgets, data, music, and poetry. One could use abstract visual-
ization to make three-dimensional visualizations, but then you have the problem of
3 Authenticity in Immersive Design for Education 53

elements in the graph covering up (occluding) each other from the user's point of
view. It should only be done for a specific reason that arises from the data itself. Last
but not least, the expense and barriers to access to VR and MR remain significant for
most k-12 schools, although that is changing (Richards, Chap. 6).

3.7 Conclusion

We live in an exciting and productive time. The mass availability of decent quality
immersive media tools has led to an explosion of creativity across many industries,
education not least (Richards, Chap. 1). The social learning process will require a
great deal of trial and error, just as it does with individuals. And with each
experiment, it is important that practitioners in education learn the right lessons. For
some direct design advice, hard won through experience, see Johnson-Glenberg
(Chap. 11) and Dede et al. (Chap. 8).
Guiding theory has an important role to play. Experimental science can provide
the foundation for these theories, but practical advice must also be developed from
experience. In this chapter, we developed a theory of authenticity in educational
immersive media, as a means to understand this key feature of good design. It is not
a comprehensive framework, but a dimension of design that touches all aspects of
an immersive education. In future articles, we will look at practical applications.

References

Barab, S. A., Squire, K. D., & Dueber, W. (2000). A co-evolutionary model for supporting the
emergence of authenticity. Educational Technology Research and Development, 48(2), 37–62.
Barsalou, L. W. (1992). Cognitive Psychology. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
0-8058-0691-1 (hard) 0-8058-0691-0 (pbk) of research on educational communications and
technology (pp. 723–734). New York: Springer.
Dede, C. (2012, May). Interweaving assessments into immersive authentic simulations: Design
strategies for diagnostic and instructional insights. In Invitational Research Symposium on
Technology Enhanced Assessments.
Donnelly, P. (2014). Let’s get real: Authenticity in design. WorkDesign Magazine (Oct 10th,
2104). https://workdesign.com/2014/10/lets-get-real-authenticity-design/.
Duffy, T. M., & Jonassen, D. H. (Eds.). (2013). Constructivism and the technology of instruction:
A conversation. London: Routledge.
Engler, L., & Fijan, C. (1997). Making puppets come alive: How to learn and teach hand
puppetry. Courier Corporation.
Fowler, C. (2015). Virtual reality and learning: Where is the pedagogy? British Journal of
Educational Technology, 46(2), 412–422.
Gillam, R., & Jacobson, J. (Eds.). (2015). The Egyptian Oracle Project: Ancient ceremony in
augmented reality. London: Bloomsbury Publishing.
Grotzer, T. A., Powell, M. M., Derbiszewska, K. M., Courter, C. J., Kamarainen, A. M., Metcalf,
S. J., et al. (2015). Turning transfer inside out: The affordances of virtual worlds and mobile
54 J. Jacobson

devices in real world contexts for teaching about causality across time and distance in
ecosystems. Technology, Knowledge and Learning, 20(1), 43–69.
Herrington, J., Reeves, T. C., & Oliver, R. (2009). A practical guide to authentic e-learning.
London: Routledge.
Herrington, J., & Parker, J. (2013). Emerging technologies as cognitive tools for authentic
learning. British Journal of Educational Technology, 44(4), 607–615.
Jacobson, J. (2011). Digital dome versus desktop display in an educational game: Gates of Horus.
International Journal of Gaming and Computer-Mediated Simulations (IJGCMS), Special
issue on educational applications, Spring, IGI Global.
Jacobson, J. (2013). Digital dome versus desktop display; Learning outcome assessments by
domain experts. International Journal of Virtual and Personal Learning Environments, Fall,
IGI Global.
Kapralos, B., Moussa, F., & Dubrowski, A. (2014). An overview of virtual simulation and serious
gaming for surgical education and training. In Technologies of inclusive well-being (pp. 289–
306). Berlin: Springer.
Kronqvist, A., Jokinen, J., & Rousi, R. (2016). Evaluating the authenticity of virtual environments.
Advances in Human-Computer Interaction, 2016, 3.
Pantelidis, V. S. (2010). Reasons to use virtual reality in education and training courses and a
model to determine when to use virtual reality. Themes in Science and Technology Education,
2(1–2), 59–70.
Pertaub, D. P., Slater, M., & Barker, C. (2002). An experiment on public speaking anxiety in
response to three different types of virtual audience presence-teleoperators and virtual.
Environments, 11(1), 68–78.
Rizzo, A. (2016). BRAVEMIND: Advancing the Virtual Iraq/Afghanistan PTSD Exposure
Therapy for MST. University of Southern California Los Angeles. http://www.dtic.mil/docs/
citations/ADA636987.
Shute, V., & Ventura, M. (2013). Stealth assessment: Measuring and supporting learning in video
games. Cambridge: MIT Press.
Strobel, J., Wang, J., Weber, N. R., & Dyehouse, M. (2013). The role of authenticity in
design-based learning environments: The case of engineering education. Computers &
Education, 64, 143–152.
Vosniadou, S., & Brewer, W. F. (1992). Mental models of the earth: A study of conceptual change
in childhood. Cognitive Psychology, 24(4), 535–585.

Author Biography

Dr. Jacobson has investigated fully immersive virtual reality (VR) as a learning medium for two
decades, developing the technology and conducting experimental research. His early technical
work in affordable free software is widely cited in the literature. His experimental trials on VR
verses desktop displays were one of the few successful media comparison studies ever conducted.
His later work (NSF and NEH funded) is highly regarded among scholars of cultural history and
heritage. He has given hundreds of talks and demonstrations at top universities, academic
conferences, and industrial conventions. Today, Dr. Jacobson is a co-founder and leader
BostonVR, the fifth largest VR meet-up group in the world. He is currently consulting with
educators and professionals in several industries, including university graduate studies, architec-
tural design, large-vehicle piloting, and virtual reality display design. All this work is described in
his many scientific publications and two books. Dr. Jacobson has served as a project reviewer for
the National Science Foundation, NOAA, NASA, and many academic publishers. He is currently
the CEO of EnterpriseVR (http://enterprisevr.com), his consulting firm.
Chapter 4
The Immersive Power of Social Interaction
Using New Media and Technology to Foster
Learning by Means of Social Immersion

Nicole C. Krämer

Abstract The chapter reviews new technologies and their impact on learning and
students’ motivation. The main argument is that in order to achieve immersion,
social interactions should be fostered. Therefore, three technologies are discussed
which either inherently draw on social interactions (pedagogical agents, trans-
formed social interaction) or can be enriched by including collaborative learning
elements (augmented reality). For each of the three realms, a short overview on the
state of current developments as well as on empirical studies and results is given.
Also, it is discussed to what extent they built on social interaction, how this might
be extended and whether beneficial outcomes can be expected from this.

Keywords Immersion Pedagogical agents Augmented reality



Transformed social interaction Collaborative learning Social interaction

4.1 Introduction

Recent years have seen the development of new and affordable Virtual Reality
technology that is widely disseminated and especially made available for enter-
tainment and leisure time use. For example, the WII uses video analysis to capture
people’s behavior and transfer it to virtual, animated worlds, Oculus rift allows to
be immersed in 3D worlds and augmented reality games like Pokemon Go are
played on every street. While the entertainment industry has grasped the potential
and optimized the technology for leisure time use, applications targeted at educa-
tional use still have to catch up. It is, however, highly plausible to assume that these
interactive technologies will also be beneficial in education. This is not only due to
the fact that people experience fun when using these technologies which might lead

N.C. Krämer (&)

Department for Computer Science and Applied Cognitive Science, University
Duisburg-Essen, Forsthausweg 2, 47057 Duisburg, Germany
e-mail: [email protected]
URL: https://www.uni-due.de/sozialpsychologie/index_en.php

© Springer Nature Singapore Pte Ltd. 2017 55

D. Liu et al. (eds.), Virtual, Augmented, and Mixed Realities in Education,
Smart Computing and Intelligence, DOI 10.1007/978-981-10-5490-7_4
56 N.C. Krämer

to increased learning motivation, but they might have inherent benefits for learning
applications. It has, for example, been described that augmented reality technolo-
gies which provide location-based virtual overlays in the real world enable learning
at the exact right time and place (Bower, Howe, McCredie, Robinson, & Grover,
2014; Yuen, Yaoyuneyong, & Johnson, 2011). However, it is certainly not suffi-
cient to simply bring these technologies to classrooms and other learning situations
and to expect that they will (a) widely be used because of their popularity in other
realms and (b) automatically lead to beneficial results. In parallel to their
employment, the technologies need to be adapted to the educational goal and to be
orchestrated in order to best enhance learning outcomes as well as motivation.
Therefore, the developments need to be informed by relevant theories of learning so
that they cannot only be immersive but also effectively shape the learning process
(e.g., guided social constructivism, situated learning).
The focus within the present chapter will be on social aspects of learning and the
question whether new immersive VR technologies can be used to foster learning via
optimizing social interactions between teacher and learner or between learners in
collaborative learning environments. In general, it aptly has been stressed that
learning and education are inherently social: Research on how people learn suggests
that learning and cognition are complex social phenomena distributed across mind,
activity, space, and time (Dunleavy, Dede & Mitchell, 2009).
It will be suggested that social interaction can provide a “new” form of immersion
that is slightly different from the forms that have been distinguished (actional
immersion as being immersed in a task, narrative immersion which is achieved by
(fictional) narratives that lead to powerful semantic associations and sensory
immersion which can be experienced when using immersive displays, Dede, 2009).
Immersion via social interaction will be experienced when teacher and learner or
learners amongst themselves engage in exchange or collaboration on the learning
contents. Against the background of these considerations, this chapter will focus on
those aspects of using virtual reality technology which either foster social interaction
among humans or inherently entail social interaction because one of the entities
included in the setting is artificially built to provide a social learning situation.
When focusing on social aspects of instructional communication in immersive
media, it is important to provide theoretical models that can help systematize
research in this area. Several researchers have suggested appropriate theoretical
frameworks (Kim & Baylor, 2006; Veletsianos & Russell, 2014). First and foremost,
Bandura’s (2001) social cognitive theory has been named as suitable background:
Within all kinds of immersive media, other students or autonomous teacher agents
can be seen as fostering learning by serving as a model for specific behavior—
especially when the other shares some similarities with the learner. Also, Kim and
Baylor (2006) refer to distributed cognition (Salomon, 2001): based on the
assumption that cognition is distributed among individuals, tools, and artifacts, an
interaction with other learners in an immersive environment or an autonomous
agent’s communicative abilities can be used as scaffolding mechanisms by asking
questions or giving hints (Veletsianos & Russell, 2014). When enhanced by adding a
social dimension, also cognitive theory of multimedia learning might be seen as a
4 The Immersive Power of Social Interaction 57

suitable framework (Mayer & DaPra, 2012). Also, Vygotsky’s zone of proximal
development (Vygotsky, 1978) is often referred to as he already claimed that adding
a social dimension to the learning situation is important. The understanding that
learning is a social situation helps to clarify that not only the cognitive impact has to
be appropriate but also the social interaction has to work in order to make instruc-
tional communication beneficial: Here, the establishment of common ground (Clark,
1996), for example between instructor and student, but also between collaborating
students, can be mentioned as decisive and can be seen as a prerequisite for
instructional communication. Here, the student must be able to understand the
instruction correctly and—probably more importantly—the teacher has to be able to
recognize whether shared meaning or common ground have been achieved. Also,
peers learning together need to be supported by technology in a way that they can be
sure that their hints, explanations or questions were understood by the fellow student
trying to benefit from the explanation or trying to clarify a problem or its solution.
Klopfer’s chapter provides examples of this. The need to analyze and solve these
issues by developing appropriate media environments has long been recognized in
the field of computer-supported-collaborative-learning (CSCL, and similarly in
Computer-Supported-Cooperative-Work, CSCW, and human-computer-interaction,
HCI, research). Here, a rich body of research laid the groundwork for understanding
and supporting learning related social interaction in mediated environments. One
important notion suggested by this research realm is the insight that mediated
communication is not necessarily deficit-laden but that technologies always also
entail specific benefits which might ease, support or enhance communication.
Gardner and Scheaffer’s chapter provides examples of this.
In the following, three lines of research which are each connected to specific
technology, will be presented. First, building on CSCL, transformed social inter-
action, a specific form of immersive virtual reality applications will be discussed
that enrich communication by unique opportunities provided by immersive sys-
tems. Here, the fabrics of social interaction are manipulated while people are
interacting within immersive virtual worlds, for example, by altering smiling or
gaze behavior. Then, the state of the art with regard to pedagogical agents will be
presented and discussed with a special focus on social instead of cognitive aspects
and effects. Finally, augmented reality is targeted which does not only allow for
situated learning but is also well suited to enable social interaction that is fueled by
narratives connected to the displayed virtual entities.

4.2 Technologies Fostering Social Immersion

4.2.1 Transformed Social Interaction

The notion that new technology bears unique opportunities for human-human social
interaction in learning realms is especially true for “Transformed Social Interaction”
(Bailenson, 2006). Here, social interaction is conducted within an immersive virtual
58 N.C. Krämer

reality and is amended in ways that would not be possible when interacting
face-to-face. Immersive virtual reality is realized by all interactants wearing motion
capture devices such as optical, magnetic, or mechanical sensors or (more recently)
facing video analyzing technology (such as included in the WII) that track aspects
of their behavior (e.g., walking, head movement, gestures, gaze, facial expressions).
The tracked behavior is then broadcasted to the other interactants by means of
virtual reality, for example, an avatar, that is a virtual person that is animated based
on the tracked person’s movements (and vice versa). However, unlike in direct
face-to-face situations the behavior does not have to be transferred naturalistically,
i.e., in the exact way it is shown but can be altered by algorithms (Blascovich &
Bailenson, 2012). By means of immersive displays every user is placed in an
immersive virtual world in which the environment and other users’ behavior are
rendered and displayed stereoscopically by means of a device such as an HMD or
Oculus Rift. Thereby, individuals from different (potentially remote) places can join
the same virtual world. As mentioned above, their interaction behavior can either be
transferred naturalistically, i.e. in the way it is produced by the individuals, or can
be altered by algorithms. This is possible because collaborative virtual environ-
ments render the world separately for each user simultaneously, so that the inter-
action appears differently for each user and potentially different from what
happened in the real world (Bailenson, 2006). Here, three dimensions are distin-
guished which are meant to classify each currently conceivable alteration of reality:
situational context/social environment, sensory abilities, and self-representation.
With regard to situational context/social environment people and things can be
represented at different locations and different time-settings. Therefore, the spatial
or temporal structure of a conversation is transformed. In a learning environment
this entails, for instance, that every learner can sit right in front of the virtual
blackboard (while the other students—who perceive themselves to be in front—are
perceived as sitting behind the learner). The ability to alter time can enable a student
to “pause” or “rewind” during a conversation (which might increase comprehen-
sion). While the alteration of time does not necessarily increase immersion in the
situation, the possibility to place learners in different environments and in different
locations within a specific environment can foster immersion.
Sensory abilities describes the possibility to augment the sensoric potential by
adding information that humans usually cannot perceive or derive consciously.
Dede’s chapter provides illustrations of this. Here, the system would provide
aggregated information about other participants (e.g., how often did he/she gaze at
the blackboard; how often did he/she smile or frown?). Bailenson (2006) suggests
to realize this by providing “invisible” consultants. This algorithm-based, real-time
summary information about the attentions and movements of other interactants is
automatically collected by the virtual environment technologies. Teachers in a
distant learning scenario can thereby either control whether learner seem to be
attentive or can control their own behavior, when, e.g., receiving information about
whether they spread their attention evenly. Apart from altering social interaction,
there are many other forms of transforming sensory abilities, for example, shrinking
4 The Immersive Power of Social Interaction 59

to the microscopic level to observe things invisible at the macroscopic level,

or seeing wavelengths of light invisible to human eyesight (e.g., infrared) (Dede,
2009).
The probably most important and most analyzed opportunity, however, is self
representation, meaning the potential that appearance and/or behavior of the user is
manipulated. Slater’s chapter discusses several examples of this. The virtual rep-
resentation of a person can be taller, shorter, more attractive, than in reality or can
be altered with regard to sex, age, or race. Also, the nonverbal behavior can be
altered and, for example, more or less smiling can be shown. With regard to the
latter, recent results show that a conversation indeed is perceived as more positive
and that interactants feel more positive affect when the smiling behavior is
enhanced (Oh, Bailenson, Krämer, & Li, see Fig. 4.1). This effect could also be
capitalized on in learning situations by either in general having the teacher display
more smiling or having the teacher show enhanced smiling only to those students
who react favorably to this and show a stern face to others who might be more
impressed by this (Bailenson, 2006). The impact of gaze has been shown impres-
sively in studies of the so-called non zero sum gaze paradigm. While in face-to-face
interactions, a person can either look at one conversation partner OR the other, in a
virtual environment it is possible to look at both at the same time (since the virtual
environment and all avatars are individually rendered for each individual user and
therefore one person can be rendered differently for each other interactant).
Empirical studies have indeed demonstrated that it lead to positive effects when
gaze was augmented and a person looked at two others for 100% of the time each
(Beall, Bailenson, Loomis, Blascovich & Rex, 2003). People being gazed at 100%
of time returned gaze more often and were more persuaded by the person with
augmented gaze (see Fig. 4.2).
With regard to the manipulation of one’s avatar’s appearance two strands of
research can be distinguished: In identity capture studies (Bailenson, Garland,
Iyengar, & Yee, 2004) it was demonstrated that virtual interaction partners that by

Fig. 4.1 Avatar with normal smile and enhanced smile (Oh et al., 2016a)
60 N.C. Krämer

Fig. 4.2 Non zero sum gaze

(Beall et al., 2003)

morphing techniques include aspects of the specific subjects’ facial structure were
perceived as more favorable. While this can easily be explained by findings that
people show affinity towards something that bears resemblance to themselves it
suggests intriguing possibilities for rendering student-teacher interactions smoother
(learner and teacher perceiving each other to be similar to themselves). Second, the
proteus effect (Yee & Bailenson, 2007) describes the possibility to alter the par-
ticipant’s appearance in a way that he/she perceive themselves (by the help of a
virtual mirror) as member of a different gender, age or race and by this literally step
into the shoes of another person. Numerous studies have shown that people alter
their attitudes and/or their behavior in accordance with the appearance they assume.
For example, people who embodied an elderly person reduce ageism more com-
pared to those who engaged in mere perspective taking (Oh, Bailenson, Weisz, &
Zaki, 2016b; see also the chapter by Mel Slater in this book).
To conclude, with regard to the social aspects of learning, all of these different
forms of transformed social interaction are particularly interesting as they enable
researchers to take apart the fabric of social interaction. By this, scholars can (a) use
TSI in order to analyze and understand the mechanisms of social interaction and
(b) can utilize the technological possibilities directly to improve learning situations
when they are conducted within immersive virtual environments.

4.2.2 Animated Pedagogical Agents

Besides transformed social interaction which targets human-human-interaction in

immersive environments, other technology tries to incorporate autonomously acting
entities in immersive settings that are based on artificial intelligence. These
so-called pedagogical agents autonomously interact with and support the learner as
tutor or peer. Even if so far no respective applications exist, these agents are
conceivable to enhance social learning in Virtual Reality and Augmented Reality
environments (for example in EcoMOBILE, http://ecomobile.gse.harvard.edu).
4 The Immersive Power of Social Interaction 61

Pedagogical agents have long been suggested as emerging technology in order to

enhance students’ learning and motivation. Future systems are expected to enhance
e-learning programs for individual learning with a human-like motivator or to
support teachers in the classroom by attending to small groups of students. As one
major advantage of these embodied automatic tutors, their advocates claim that
pedagogical agents can, or will be able in the future, communicate via verbal and
nonverbal means, thus facilitating and personalizing the interaction with an
e-learning program (see Fig. 4.3). Furthermore, increased motivation is expected:
Baylor and Ryu (2003) suggest that the key advantage is that human-likeness
creates more positive learning experiences and provides a strong motivating effect.
In line with these assumptions, research on pedagogical agents suggests that even
the mere social presence of an embodied autonomous tutor might be capable of
fostering interest, attention, and subsequently learning.
Several overviews of pedagogical agent research (Baylor, 2001; Heidig &
Clarebout, 2011; Moreno, 2004) outline various developments, theoretical
assumptions on effects and results of evaluation studies. With regard to develop-
ments, systems from both cognitive and educational psychology and from the field
of computer science have been presented. For example, based on discourse analyses
of face-to-face tutoring lessons, Graesser, Wiemer-Hastings, Wiemer-Hastings,
Kreuz, and The Tutoring Research Group (1999) constructed the AutoTutor as a
dialog partner, which by asking questions and giving feedback helps the student to
actively construct subjective explanations and engage in deep reasoning. Here, the
agent utilizes a hint–prompt–elaboration circle until the learner utters the correct
answer. A talking head is used in order to ground the conversation between the
tutor and the learner by means of nonverbal feedback cues (nodding or shaking the
head, facial expressions). Similarly, also Baylor and colleagues provide a sophis-
ticated learning program that is amended by simple embodiment techniques and
basic nonverbal capabilities (Baylor & Ryu, 2003). Here, the focus is clearly on
social interaction and social support for the learner (Kim & Baylor, 2016). One of

Fig. 4.3 Virtual agent

62 N.C. Krämer

the oldest, but nevertheless ground-breaking systems has been presented by Rickel
and Johnson (2000) who locate their agents in a virtual learning environment.
Agent Steve leads learners through a US navy ship and is capable of reacting to
changes in the virtual environment as well as to learners’ behavior. Based on
information on the environment and the learner, he asks the student appropriate
questions or gives explanations. The best known pedagogical agents are probably
the systems by Lester et al. (2000) who developed various agents that are supposed
to motivate children to learn within desktop-based learning environments. Some of
these agents are able to use gestures and movements to highlight objects, but are
also capable of displaying a wide array of emotions.

4.2.2.1 Results on the Effects of Pedagogical Agents

Several reviews and meta-analyses on pedagogical agents report that there is

empirical support for the notion that pedagogical agents motivate the learner and
lead to increased learning (Baylor 2001; Moreno 2004; Schroeder, Adesope, &
Gilbert, 2013). However, results are not consistent and mostly show small effect
sizes. For example, Graesser, Jackson, and McDaniel (2007) concluded that
AutoTutor improves learning by nearly one letter grade compared with reading a
textbook for an equivalent amount of time or in comparison with a pretest.
However, Rajan et al. (2001) demonstrated that it is first and foremost the voice that
is responsible for these effects. Baylor and colleagues provided more evidence for
an impact of pedagogical agents on learners’ subjective experiences, but not on
their performance and learning outcome (Baylor & Ryu, 2003). Moreno (2003)
further summarized that—in line with results that especially the voice is decisive—
there is no evidence for the social cue hypothesis as it has not been shown that the
mere presence of social aspects such as a human-like body leads to distinct effects.
However, the cognitive guiding functions provided by vocalizations and a pro-
gram’s didactic concept did prove to be influential. Moreno concluded that the main
strength of pedagogical agents resides in the specific instructional method
embedded in the agent rather than in the visual presence of the agent itself. Also,
recent research (Carlotto & Jaques, 2016) as well as a recent meta-analysis
(Schroeder & Adesope, 2014) has supported the notion that voice is more important
than nonverbal expressiveness.
However, these results have to be considered with caution given the fact that the
systems that had been evaluated did not (yet) include very sophisticated nonverbal
behavior. Nonverbal behavior as it is used in face-to-face interaction includes facial
displays and all kinds of kinesics in the sense of body movement. It needs to be
acknowledged that nonverbal behavior is very complex: The dynamics of the
movements are important, very subtle movements have distinct effects (e.g., head
movements such as a head tilt) and the effects are context-dependent (e.g., a smile
leads to a different effect when accompanied by a head tilt). This complexity,
however, is mostly not considered and implemented in pedagogical agents. For
example, as Graesser himself acknowledges, the nonverbal implementations as well
4 The Immersive Power of Social Interaction 63

as their theoretical foundations for AutoTutor are rather shallow and fall short of the
sophisticated dialogue model, while Rickel and Johnson (2000) focused more on
multimodal input (e.g., tracking the student’s behavior) than multimodal output
(e.g., deictic gestures and further nonverbal behaviors). Also, Lester’s agents do not
show gestures and movements which represent human nonverbal behavior—
especially as the agent is only partly anthropomorphic. So far, only very few
pedagogical agent systems (even more recent ones) have achieved realistic and
sufficiently subtle nonverbal behavior in order to administer a fair test. And indeed,
when employing technology that provides realistic, dynamic nonverbal behavior,
results show that nonverbal rapport behavior leads to an increase in effort and
performance (Krämer et al., 2016). Therefore, the conclusion that embodiment and
nonverbal behavior is less decisive compared to voice is premature. On the other
hand, there are studies which have demonstrated considerable effects of nonverbal
behavior even though the cues displayed were very basic (eye gaze of a comic-style
agent, realized by eye-direction of the eyeball only, Lee, Kanakogi & Hiraki, 2015).
Similarly, Mayer and DaPra (2012) present evidence for the “embodiment effect” in
the sense of the question whether nonverbal behavior will yield better learning
results. They demonstrate both increased values regarding a learning transfer test
and more positive ratings of the social attributes of the agent. They explain the
result with social agency theory (Mayer, 2005): social cues from a social agent
prime a social stance in learners that causes them to work harder by activating deep
cognitive processing in order to make sense of the presented material.

4.2.2.2 The Importance of Social Processes When Building

Pedagogical Agents

Recent developments increasingly take the fact that learning is a social process into
account. While there are still studies focusing on cognitive and metacognitive
aspects, more and more priority is set on the social relationship with the user.
Instead of merely providing expert guidance, the agents are considered as tools to
support learners by social and affective capabilities (Kim & Baylor, 2016; Krämer
& Bente, 2010; Veletsianos & Russell, 2014). Kim and Baylor (2006) argue that
learning environments should provide situated social interaction since it is well
documented that the cognitive functioning of learners is framed by social contexts
and that teaching and learning are highly social activities.
Situated social interaction, as Kim and Baylor argue, can be realized by peda-
gogical agents that simulate human instructional roles such as teacher or peer.
Similarly, Graesser (2006) states that social psychological aspects have to be
considered in pedagogical agent research since cognitive representations might be
social. He specifies conditions under which more or less social context has to be
provided and concludes that the social context of knowledge construction is par-
ticularly important when knowledge is vague, open-ended, underspecified, and
fragmentary. As a consequence, especially when building computer systems that
can conduct effective conversation, the system has to possess basic social abilities
64 N.C. Krämer

in terms of having internal representations of the knowledge, beliefs, goals,

intentions, and values of the human user (see Krämer 2008, who argues in favor of
implementing a theory of mind).
Some systems already explicitly consider social aspects: For example, in several
developments agents take the role of a (fellow) student instead that of a teacher and
foster learning by the fact that the student has to engage in explaining the learning
matter to the (artificial) peer (see, for example, the teachable agents paradigm,
Schwartz, Blair, Biswas, & Leelawong 2007). According to Veletsianos and
Russell (2014) these agents are expected to lower learner anxiety (by seeming less
threatening than instructors), act as role models and “teach” via giving the students
the chance to detect mistakes that the agent makes during the learning process.
Another possibility to engage the learner socially has been presented as social
instructional dialog (Ogan, Aleven, Jones, & Kim, 2011) by which the agent tries to
improve learner-agent interpersonal relations. Agents who used conversational
strategies assumed to produce positive interpersonal effects, lead to positive effects
on learners’ entitativity (feeling of working together in a team), shared perspective,
and trust with the agent.
Therefore, with a view to social psychological aspects the distinction between
different roles of the agent as either a mentor, a peer or a protégé is important.
Against the background that by means of pedagogical agents developers attempt to
replicate useful human characteristics in the virtual world, future research needs to
gain insights on what the desirable and beneficial attributes for each role in
face-to-face interaction are. These attributes can be transferred to the virtual agents
in order to test whether these attributes also have beneficial effects in VR.
At least with regard to the agent’s role as mentor, future research can build on
Klauer’s (1985) taxonomy of teaching functions that have been proposed by Heidig
and Clarebout (2011) as useful for the realm of pedagogical agents: motivation,
information, information processing, storing and retrieving, transfer of information,
monitoring and directing.
What is additionally important to note, is that social effects of agents are to be
expected anyway: early evaluation studies of conventional computers characterized
by human-like attributes (Reeves & Nass, 1996) as well as with embodied con-
versational agents provided evidence that machines and agents are readily perceived
as social entities: even minimal cues and similarity with humans suffice to lead
users to show behavior that would be expected in human–human interaction. It is
therefore plausible to assume that learners will also interact socially with peda-
gogical agents. This will also enhance chances that social mechanisms of learning
will also transfer to the virtual world.
Given the open questions, e.g. with regard to the effects of the agents’ nonverbal
behavior, future research is needed. This is best conducted against the background
of a coherent theoretical framework that is able to explain the mechanisms. There
have been suggestions for frameworks for future studies (Heidig & Clarebout,
2011) as well as suggestions on the necessities for methodology. With regard to the
latter, it will be important to test agents in naturalistic settings and in longterm
studies (Veletsianos & Russell, 2014; Krämer & Bente, 2010).
4 The Immersive Power of Social Interaction 65

4.2.3 Augmented Reality for Learning Interaction

Augmented realities also have enormous potential for learning environments and
can—depending on how they are shaped—also especially foster beneficial social
interaction. Recently, several researchers proposed to employ augmented reality
technology in learning settings.
Augmented reality is characterized as enhancement of the real world by
computer-generated content which is tied to specific locations or activities (see e.g.,
Yuen et al., 2011). The digital content is laid over the vision of the real world. By
means of tablets or smartphones (and therefore available to a large percentage of
people), text, audio, images, 3D models or video can be mixed into the users’
perceptions of the real world. A large number of Augmented Reality applications
rely on GPS technology implemented in the devices and offer location-based ser-
vices. The specific location can be enriched with relevant narrative, navigation,
and/or academic information. Bower et al. (2014) conclude that by this kind of
location-based procedure, information for students can be provided at the exact time
and place of need. By this, cognitive overload can be reduced as students experi-
ence “perfectly situated scaffolding”. Schneider’s chapter provides examples of
this.
Yuen et al. (2011) enthuse over the emerging possibilities for education and
suggest that the dream of ubiquitous learning now becomes reality and will enable
learners to access a wide range of location-specific information. By this, the users’
perceptions and knowledge is enhanced.
Although the technology is still young, already several overviews on research in
educational context have been presented (Dunleavy & Dede, 2014). Wu, Lee,
Chang, and Liang (2013) in a literature review for the years 2000–2012 find 54
papers on AR in education. Yuen et al. (2011) and Wu et al. (2013) summarize the
benefits that Augmented Reality can have for teaching and learning environments:
Among other aspects such as fostering student motivation, creativity and imagi-
nation or experiencing phenomena not observable in the real world, the enhance-
ment of collaboration between students and instructors as well as among students is
named. Wu et al. (2013) classify instructional approaches with augmented reality in
three major categories: engaging learners into roles, focusing on interactions with
locations and emphasizing tasks. With a view to social immersion especially the
category role is important. Here, participatory simulations, role playing and jigsaw
approach can be distinguished. All of these foster interaction and collaboration
among students, but all employ different mechanisms for engaging users in a joint
learning task: For example, in participatory simulations students jointly simulate a
phenomenon (e.g. the spreading of a virus) by means of beaming information from
handheld device to handheld device (Klopfer, Yoon, & Rivas, 2004). Roleplay can
be helpful in comprehending complex mechanisms such as, for example, the
socially situated nature of scientific investigations by means of socially immersing
students by assigning roles of scientists, environmental investigators and activists
(Squire & Klopfer, 2007). The name jigsaw approach stems from the fact that
66 N.C. Krämer

different students receive different parts of the “puzzle” in the sense that this
approach fosters collaboration by assigning different roles and different knowledge
to students who need to solve a problem together (Dunleavy et al., 2009; see Wu
et al., 2013).
Overall, especially game-based learning can be helpful in engaging students
socially and immersively. Yuen et al. (2011) report that in 2000 Bruce Thomas
created the first outdoor AR game “AR Quake” (Thomas, Close, Donoghue,
Squires, Bondi, & Piekarski, 2001) with many more to follow (von der Pütten et al.,
2012).
A game that combines augmented reality, social interaction and education is the
AR game Alien Contact (Dunleavy et al., 2009). Students play different roles in a
team and have to form hypotheses by collecting evidence and complete tasks from
maths, science, and language skills. Qualitative evaluations with middle school and
high school students demonstrate that the game is perceived as highly engaging.
The authors attribute this engagement to the innovative tools whose attractiveness
for learning, however, will decline over time. Therefore, they recommend to
identify curricular-specific and technology-specific characteristics which might be
combined with sound pedagogy in order to keep students engaged over time
(Dunleavy et al., 2009).
For the future, further developments such as augmented reality holographic
projection can be expected (Yuen et al., 2011). Thereby, video conference-like
meetings and an effortless communication over a distance can be realized. This
might improve early attempts to bring together people and their virtual represen-
tations. For example, education-related events in Second live virtual worlds could
be replaced by holographic projection. This could be useful especially in different
forms of online courses and distance learning such as Massive Open Online
Courses, in which collaborative endeavours in small groups can be supported by
holographic representations. As Huang, Rauch, and Liaw (2010) describe, VR
technologies that support immersive learning are expected to bear great potential for
social scaffolding in collaborative learning.

4.3 Conclusion

Since learning situations and instructional communication are inseparably inter-

woven with social interaction (Dunleavy et al., 2009; Rummel & Krämer, 2010) it
is important to also cater for social needs when employing immersive technologies
in learning settings. With regard to pedagogical agents this will entail that
researchers have to contribute to building the necessary capacities, by, for example,
developing nonverbal behavior that is suited to foster learning. As a means to this
end, transformed social interaction research will also be helpful in identifying
which social signals are most beneficial with regard to supporting learning.
Since TSI allows to manipulate teachers’ and learners’ behavior in a virtual envi-
ronment it is not only helpful in order to actually implement suitable behavior
4 The Immersive Power of Social Interaction 67

(for example to render a teacher who does not smile friendlier towards students who
need this), but also in order to serve as a research tool helping to understand the
fabrics of social interaction. With regard to augmented reality it will be important to
not only consider cognitive aspects of learning but to also take social aspects into
account by, for example, developing narrative structures that will invite social
interactions between learners or between learners and teachers as this will not only
deepen the experience but also the learning outcomes.
For all forms of immersive social learning experiences it will further be
important to derive future developments from theoretical frameworks that have to
be refined for these contexts. Only then it can be achieved that the natural relation
between learning and social interaction is further optimized by means of technology
that increases social immersion with a view to better learning outcomes.

References

Bailenson, J. (2006). Transformed social interaction in collaborative virtual environments.

In P. Messaris & L. Humphreys (Eds.), Digital media: Transformations in human
communication (pp. 255–264). New York: Peter Lang.
Bailenson, J. N., Garland, P., Iyengar, S., & Yee, N. (2004). Transformed facial similarity as a
political cue: A preliminary investigation. Political Psychology, 27(3), 373–386.
Bandura, A. (2001). Social cognitive theory of mass communication. Media Psychology, 3(3),
265–299.
Baylor, A. L. (2001). Permutations of control: Cognitive considerations for agent-based learning
environments. Journal of Interactive Learning Research, 12(4), 403–425.
Baylor, A. L., & Ryu, J. (2003). The effects of image and animation in enhancing pedagogical
agent persona. Journal of Educational Computing Research, 28(4), 373–394. doi:10.2190/
V0WQ-NWGN-JB54-FAT4.
Beall, A.C., Bailenson, J. N., Loomis, J., Blascovich, J., & Rex, C. (2003). Non-zero-sum mutual
gaze in immersive virtual environments. In Proceedings of HCI 2003.
Blascovich, J., & Bailenson, J. N. (2012). Infinite reality. The hidden blueprint of our virtual lives.
New York: Harper Collins.
Bower, M., Howe, C., McCredie, N., Robinson, A., & Grover, D. (2014). Augmented reality in
education—cases, places and potentials. Educational Media International, 51(1), 1–15. doi:10.
1080/09523987.2014.889400.
Carlotto, T., & Jaques, P. A. (2016). The effects of animated pedagogical agents in an
English-as-a-foreign-language learning environment. International Journal of
Human-Computer Studies, 95, 15–26. doi:10.1016/j.ijhcs.2016.06.001.
Clark, H. H. (1996). Using language. Cambridge: Cambridge University Press.
Dede, C. (2009). Immersive interfaces for engagement and learning. Science, 323(5910), 66–69.
doi:10.1126/science.1167311.
Dunleavy M., & Dede, C. (2014). Augmented reality teaching and learning. Handbook of research
on educational communications and technology (pp. 735–745). New York: Springer.
Dunleavy, M., Dede, C., & Mitchell, R. (2009). Affordances and limitations of immersive
participatory augmented simulations for teaching and learning. Journal of Science Education
and Technology, 18, 7–22. doi:10.1007/s10956-008-9119-1.
68 N.C. Krämer

Graesser, A. C. (2006). Views from a cognitive scientist: Cognitive representations underlying

discourse are sometimes social. Discourse Studies, 8, 59–66. doi:10.1177/1461445606059555.
Graesser, A. C., Jackson, G. T., & McDaniel, B. (2007). AutoTutor holds conversations with
learners that are responsive to their cognitive and emotional states. Educational Technology,
47, 19–22.
Graesser, A. C., Wiemer-Hastings, K., Wiemer-Hastings, P., Kreuz, R., & the Tutoring Research
Group. (1999). AutoTutor: A simulation of a human tutor. Journal of Cognitive Systems
Research, 1, 35–51.
Heidig, S., & Clarebout, G. (2011). Do pedagogical agents make a difference to student motivation
and learning? Educational Research Review, 6, 27–54.
Huang, H., Rauch, U., & Liaw, S. (2010). Investigating learners’ attitudes toward virtual reality
learning environments: Based on a constructivist approach. Computers & Education, 55(3),
1171–1182.
Kim, Y., & Baylor, A. L. (2006). A social–cognitive framework for pedagogical agents as learning
companions. Educational Technology Research and Development, 54(6), 569–590. doi:10.
1007/s11423-006-0637-3.
Kim, Y., & Baylor, A. L. (2016). Research-based design of pedagogical agent roles: A review,
progress, and recommendations. International Journal of Artificial Intelligence in Education,
26(1), 160–169. doi:10.1007/s40593-015-0055-y.
Klauer, K. J. (1985). Framework for a theory of teaching. Teaching and Teacher Education, 1, 5–
17.
Klopfer, E., Yoon, S., & Rivas, L. (2004). Comparative analysis of palm and wearable computers
for participatory simulations. Journal of Computer Assisted learning, 20(5), 347–359. doi:10.
1111/j.1365-2729.2004.00094.x.
Krämer, N. C. (2008). Theory of Mind as a theoretical prerequisite to model communication with
virtual humans. In I. Wachsmuth & G. Knoblich (Eds.), Modeling communication with robots
and virtual humans (pp. 222–240). Berlin: Springer.
Krämer, N. C., & Bente, G. (2010). Personalizing e-Learning. The social effects of pedagogical
agents. Educational Psychological Review, 22, 71–87. doi:10.1007/s10648-010-9123-x.
Krämer, N. C., Karacora, B., Lucas, G., Dehghani, M., Rüther, G., & Gratch, J. (2016). Closing
the gender gap in STEM with friendly male instructors? On the effects of rapport behavior and
gender of a virtual agent in an instructional interaction. Computers & Education, 99, 1–13.
doi:10.1016/j.compedu.2016.04.002.
Lee, H., Kanakogi, Y., & Hiraki, K. (2015). Building a responsive teacher: How temporal
contingency of gaze interaction influences word learning with virtual tutors. Royal Society of
Open Science, 2, 140361. doi:10.1098/rsos.140361.
Lester, J. C., Towns, S. G., Callaway, C. B., Voerman, J. L., & FitzGerald, P. J. (2000). Deictic
and emotive communication in animated pedagogical agents. In J. Cassell, J. Sullivan, S.
Prevost, & E. Churchill (Eds.), Embodied conversational agents (pp. 123–154). Boston: MIT
Press.
Mayer, R. E. (2005). Principles of multimedia learning based on social cues: Personalization,
voice, and image principles. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia
learning (pp. 201–212). New York, NY: Cambridge University Press.
Mayer, R. E., & DaPra, C. S. (2012). An embodiment effect in computer-based learning with
animated pedagogical agents. Journal of Experimental Psychology, 18(3), 239–252. http://
psycnet.apa.org/doi/10.1037/a0028616.
Moreno, R. (2003). The role of software agents in multimedia learning environments: When do
they help students reduce cognitive load? Paper presented at the European Association for
Research on Learning and Instruction Annual Conference, Padova, Italy.
Moreno, R. (2004). Animated pedagogical agents in educational technology. Educational
Technology, 44(6), 23–30.
Ogan, A., Aleven, V., Jones, C., & Kim, J. (2011, June). Persistent effects of social instructional
dialog in a virtual learning environment. Paper presented at the International Conference on
Artificial Intelligence in Education, Auckland, New Zealand.
4 The Immersive Power of Social Interaction 69

Oh, S. Y., Bailenson, J., Kramer, N., Li, B. (2016a). Let the avatar brighten your smile: Effects of
enhancing facial expressions in virtual environments. PloS ONE, 11(9), doi:10.1371/journal.
pone.0161794.
Oh, S. Y., Bailenson, J., Weisz, E., & Zaki, J. (2016b). Virtually old: Embodied perspective taking
and the reduction of ageism under threat. Computers in Human Behavior, 60, 398–410.
Rajan, S., Craig, S. D., Gholson, B., Person, N. K., Graesser, A. C., & TRG. (2001). AutoTutor:
Incorporating backchannel feedback and other human-like conversational behaviors into an
intelligent tutoring system. International Journal of Speech Technology, 4, 117–126.
Reeves, B., & Nass, C. I. (1996). The media equation: How people treat computers, television, and
new media like real people and places. New York: Cambridge University Press.
Rickel, J., & Johnson, W. L. (2000). Task oriented collaboration with embodied agents in virtual
worlds. In J. Cassell, J. Sullivan, S. Prevost, & E. Churchill (Eds.), Embodied conversational
agents (pp. 95–122). Cambridge: MIT Press.
Rummel, N., & Krämer, N. (2010). Computer-supported instructional communication: A
multidisciplinary account of relevant factors. Educational Psychology Review, 22, 1–7.
doi:10.1007/s10648-010-9122-y.
Salomon, G. (2001). Distributed cognition: Psychological and educational considerations. New
York: Cambridge University Press.
Schroeder, N. L., & Adesope, O. O. (2014). A systematic review of pedagogical agents’ persona,
motivation, and cognitive load implications for learners. Journal of Research on Technology in
Education, 46(3), 229–251. doi:10.1080/15391523.2014.888265.
Schroeder, N. L., Adesope, O. O., & Gilbert, R. B. (2013). How effective are pedagogical agents
for learning? A meta-analytic review. Journal of Educational Computing Research, 46(1), 1–
39. doi:10.2190/EC.49.1.a.
Schwartz, D., Blair, K. P., Biswas, G., & Leelawong, K. (2007). Animations of thought:
Interactivity in the teachable agent paradigm. In R. Lowe & W. Schnotz (Eds.), Learning with
animation: Research and implications for design (pp. 114–140). Cambridge (UK): Cambridge
University Press.
Squire, K., & Klopfer, E. (2007). Augmented reality simulations on handheld computers. Journal
of the Learning Sciences, 16(3), 371–413. doi:10.1080/10508400701413435.
Thomas, B., Close, B., Donoghue, J., Squires, J., Bondi, P. D., & Piekarski, W. (2001). First
person indoor/outdoor augmented reality application: ARquake. Personal and Ubiquitous
Computing, 6, 75–86.
Veletsianos, G., & Russell, G. (2014). Pedagogical Agents. In M. Spector, D. Merrill, J. Elen, &
M. J. Bishop (Eds.), Handbook of research on educational communications and technology
(4th ed., pp. 759–769). New York: Springer Academic.
von der Pütten, A. M., Klatt, J., Broeke, S., McCall, R., Krämer, N. C., & Wetzel, R. (2012).
Subjective and behavioral presence measurement and interactivity in the collaborative
augmented reality game TimeWarp. Interacting with Computers, 24(4), 317–325. doi:10.1016/
j.intcom.2012.03.004.
Vygotsky, L. S. (1978). Mind in society. Cambridge, MA: Harvard University Press.
Wu, H.-K., Lee, S. W.-Y., Chang, H.-Y., & Liang, J.-C. (2013). Current status, opportunities and
challenges of augmented reality in education. Computers & Education, 62, 41–49. doi:10.
1016/j.compedu.2012.10.024.
Yee, N., & Bailenson, J. (2007). The Proteus Effect. The effect of transformed self-representation
on behavior. Human Communication Research, 33(3), 271–290. doi:10.1111/j.1468-2958.
2007.00299.x.
Yuen, S. C.-Y., Yaoyuneyong, G., & Johnson, E. (2011). Augmented reality: An overview and
five directions for AR in education. Journal of Educational Technology Development and
Exchange, 4(1), 119–140.
70 N.C. Krämer

Author Biography

Nicole C. Krämer is professor for Social Psychology—Media and Communication at the

University Duisburg-Essen. She has a background in social and media psychology. She finished
her Ph.D. in 2001 with a thesis on socio-emotional effects of nonverbal behavior and computer
animation as a method in communication research. She worked as a visiting researcher and visiting
lecturer at the University of Cambridge in the academic year 2002/2003. In 2006 she received the
venia legendi for psychology with the habilitation thesis on “Social effects of embodied
conversational agents”. Her research interests include human-computer-interaction, social
psychological aspects of web 2.0, nonverbal behaviour and computer supported instructional
communication.
Chapter 5
Assessment for Learning in Immersive
Environments

Valerie Shute, Seyedahmad Rahimi and Benjamin Emihovich

Abstract Immersive Environments (IEs) hold many promises for learning. They
represent an active approach to learning and are intended to facilitate better, deeper
learning of competencies relevant for success in today’s complex, interconnected
world. To harness the power of these environments for educational purposes (i.e., to
support learning), we need valid assessments of the targeted competencies. In this
chapter we focus on how to design and develop such valid assessments, particularly
those providing an ongoing, unobtrusive collection and analysis of data as students
interact within IEs. The accumulated evidence on learning thus provides increas-
ingly reliable and valid inferences about what students know and can do across
multiple contexts. This type of assessment is called “stealth assessment” and is
applied toward the real-time measurement and support of learning in IEs—of
cognitive and non-cognitive variables. The steps toward building a stealth assess-
ment in an IE are presented through a worked example in this chapter, and we
conclude with a discussion about future stealth assessment research, to move this
work into classrooms for adaptivity and personalization.

Keywords Augmented reality Diagnostic assessment
Immersive environments



Stealth assessment Digital games Virtual reality

V. Shute (&)
S. Rahimi
B. Emihovich

Educational Psychology and Learning Technologies, Florida State University, Stone
Building, 1114 West Call Street, 3064453, Tallahassee, FL 32306-4453, USA
e-mail: [email protected]
URL: http://myweb.fsu.edu/vshute/
URL: http://sahmadrahimi.wixsite.com/mysite
S. Rahimi
e-mail: [email protected]
URL: http://sahmadrahimi.wixsite.com/mysite
B. Emihovich
e-mail: [email protected]

© Springer Nature Singapore Pte Ltd. 2017 71

D. Liu et al. (eds.), Virtual, Augmented, and Mixed Realities in Education,
Smart Computing and Intelligence, DOI 10.1007/978-981-10-5490-7_5
72 V. Shute et al.

5.1 Introduction

In this chapter, we examine immersive learning environments (e.g., virtual reality,

augmented reality, and digital games) and techniques toward the measurement and
support of knowledge and skills therein. Our premise is that immersive environ-
ments (IEs) represent an active approach to learning and should thus facilitate
better, deeper learning of competencies relevant for success in today’s increasingly
complex world. Such environments also permit the application and practice of
competencies in relatively safe and authentic spaces. Moreover, well-designed IEs
that incorporate theoretically-grounded learning principles (authentic problem
solving, rules/constraints, challenge, control, ongoing feedback, and sensory
stimulation—see Shute, Ventura, Kim, & Wang, 2014) can be intrinsically moti-
vating and therefore engaging; and student engagement is a key component of
learning (Dede, 2009).
The IEs on which we focus are based on learning through experiencing, and
understood through the theoretical lenses of constructivism (Piaget, 1973) and
situated learning (Lave & Wenger, 1991). These theories emphasize active learners
who construct meaning (Vygotsky, 1978). Constructivism states that effective
learning environments are interactive places where learners achieve learning goals
by collaborating with tools, information resources, and with others. Situated
learning views cognition as being nestled within the activity, context, and culture in
which it is developed. The learner is active in the learning process, where “doing” is
more important than listening, and the learner determines the pace of learning.
Constructivism and situated learning are not, however, solely cognitive in nature
as affect and cognition are complementary processes within all forms of learning.
For example, cognitive complexity theory predicts that well-designed IEs facilitate
learning by simultaneously engaging students’ affective and cognitive processes
(Tennyson & Jorczak, 2008). Affective processes are dependent on how environ-
mental stimuli engage the student.
Similarly, flow theory (Csikszentmihalyi, 1990) argues that flow—a positive
experience associated with immersive environments—is an optimal learning state
induced by intrinsic motivation, well-defined goals, appropriate levels of challenge,
and clear and consistent feedback. Attaining a state of flow involves motivation,
effort, and sustained attention thus there is a convergence between the core elements
of well-designed IEs and the characteristics of productive learning (Shute et al.,
2014).
The purpose of this chapter is to describe how to design and develop valid
assessments to support learning in immersive environments, particularly in
well-designed digital games. The basic idea is that learners’ interactions within such
environments generate large amounts of data—cognitive and non-cognitive—
which may be captured in log files and analyzed to yield cumulative estimates of
current states of targeted competencies (Shute, Leighton, Wang, & Chu, 2016a).
The results of the ongoing analyses can be used as the basis for feedback and other
types of learning support, such as adapting the environment to fit learners’ needs.
5 Assessment for Learning in Immersive Environments 73

In the following sections of this chapter, we review the relevant literature on IEs
and their effects on learning, examine the role of diagnostic assessment in
immersive learning environments by introducing stealth assessment, provide an
example of stealth assessment within a well-designed game, and discuss next steps
in this research. Our overarching thesis is that: (a) learning is at its best when it is
active, goal-oriented, contextualized, and motivating; and (b) learning environ-
ments should thus be interactive, provide ongoing feedback, capture and hold
attention, and have appropriate and adaptive levels of challenge. Advances in
technology, the learning sciences, and measurement techniques help to support
these features through the design of IEs with deeply embedded assessment of
targeted competencies.

5.2 How Does Immersion Improve Learning?

In this chapter, immersion refers to the subjective impression one experiences when
interacting with a realistic, digitally-enhanced environment (Dede, 2009).
Immersion may be experienced within contexts such as: (1) Virtual Reality (VR),
where learners wear VR gear and go into an immersive computer-generated world
with the illusion of “being there” or having a sense of presence, with immediate
adjustments of the environment according to the learner’s head or body move-
ments; (2) Multi-User Virtual Environment (MUVE), where learners can enter a 3D
virtual world with their digital avatars and virtually interact with other people (Hew
& Cheung, 2010); and (3) Mixed Reality (MR) or Augmented Reality (AR), that
combines digital information (e.g., images, videos, 3D objects, and audio layers)
with real-world settings, and allows users to interact in real-time within a rich
immersive experience (Barfield, 2015). Well-designed digital games can provide
immersive experiences in any of these three types of environment.
Interactions within an immersive environment produce a suspension of disbelief
for learners (i.e., sacrificing realism and logic for the sake of enjoyment) that can be
further enhanced when the immersive environment incorporates design strategies
that emphasize actional, symbolic, and sensory elements (Dede, 2009). One clear
benefit of immersive environments is that they allow participants to safely engage
in actions that might be considered too risky or difficult in natural environments
(actional immersion). For example, training medical students on triage processes is
difficult due to the constraints in which activities undertaken during training reflect
the natural world conditions where triage is needed, such as a natural disaster or a
plane crash. Replicating the realism and extent of injuries along with patient
deterioration using natural world training is both expensive and incompatible for an
individual learning experience. Given the natural world restrictions of triage
training, researchers designed, built, and tested an immersive game to support
learning about how to conduct a triage sieve, as taught in a Major Incident Medical
Management and Support Course (MIMMS) in the United Kingdom.
74 V. Shute et al.

The game, Triage Trainer (Knight et al., 2010), was evaluated relative to its
effectiveness, compared to traditional learning methods (i.e., card sorting exercises).
A total of 91 participants (i.e., 44 in the card-sorting group and 47 in the Triage
Trainer group) were tested on their ability to correctly prioritize each casualty
(tagging accuracy) as well as follow the procedure correctly (step accuracy).
According to Knight et al. (2010), participants using Triage Trainer performed
significantly better than the card-sorting group for tagging accuracy (v2(5) = 13.14,
p < 0.05) (i.e., 72% compared to 55%, respectively). In addition, the step accuracy
results indicated four times as many participants in the Triage Trainer group (28%)
correctly triaged all eight of the casualties compared to the card-sorting group (7%),
and significantly more participants in the Triage Trainer group scored the maxi-
mum compared to the card-sorting group (v2(1) = 5.45, p < 0.05).
In addition to cognitive effects, well-designed digital games that fully immerse
learners in environments often elicit affective reactions (e.g., excitement, boredom,
confusion, frustration) that differentially influence learning, such as the develop-
ment of problem-solving skills and spatial abilities (e.g., Shute, Ventura, & Ke,
2015). Furthermore, there are several conditions of gameplay that one can expe-
rience in well-designed digital games (e.g., identity formation, persistent problem
solving, practice, and interaction) that impact motivation, which in turn promotes
engagement and meaningful learning (Clark, Tanner-Smith, & Killingsworth,
2014).
Consider the game World of Warcraft (WoW). This is a good example of a fully
immersive digital game in which the learning takes place in a goal-driven problem
space where players negotiate different contexts (i.e., levels, scenarios, interactions)
solving assorted problems with their avatars (Gee, 2008). Playing WoW success-
fully requires various competencies (e.g., problem-solving skills and collaboration)
as well as planning and executing strategies synchronously to accomplish goals. As
players traverse each level in WoW, it is natural to reflect on and process gameplay
choices, which helps to promote a more motivating gameplay/learning experience.
Players additionally enjoy customizing different skills and abilities for their avatars
because different combinations of abilities can lead to improved gameplay per-
formance, which results in greater rewards earned.
An example customization by game players includes design modifications to the
game that build models to be used for: (1) in-game performance improvement, and
(2) addressing a naturally occurring and frustrating in-game problem—i.e., dealing
with freeloaders. Thus, to improve in-game avatar performance, WoW players
created an add-on modification called Skada Damage Meter, which displays how
well each person in a group is performing based on feedback that is given to players
as a percentage of damage or healing done per avatar. Skada Damage Meter dis-
plays a chart with various metrics such as overall damage done, damage per minute,
overall healing done, and healing per minute. These metrics enable group leaders to
identify which players are underperforming based on their avatar role (i.e., damage
absorber, damage dealer, and healer). Developing this modification illustrates how
players were sufficiently motivated to solve a WoW problem, which has a
5 Assessment for Learning in Immersive Environments 75

real-world parallel in workplace environments (i.e., individuals who attempt benefit

from the success of others by trying to obscure their incompetent skills).
In addition to promoting problem-solving skills through gameplay, immersive
games can serve as learning vehicles to support the development of knowledge and
skills across various domains including: inquiry-based science learning with Quest
Atlantis (Barab, Thomas, Dodge, Carteaux, & Tuzun, 2005) and River City
(Ketelhut, 2007), spatial skills with Portal 2 (Shute et al., 2015), and computational
problem-solving (Liu, Cheng, & Huang, 2011) with TrainB&P (Train: Build and
Program it). Immersion fosters learning by enabling multiple frames of reference
and situated learning experiences (Dede, 2009). These multiple frames of reference
provide different benefits for immersive learning. For instance, egocentric frames of
reference support immersion and motivation through embodied learning, while
exocentric frames of reference support abstract symbolic insights when one is
further from the context of the environment.
Immersive environments also enhance a contextualized understanding of
instructional content for learners in ways that are often decontextualized in formal
learning settings. As mentioned earlier, these environments support meaningful
learning experiences that are grounded by situated learning and constructivism
learning theories. Situated learning is an active process that can generate excitement
and curiosity in the learner to acquire knowledge by constructing meaning through
specific problem-solving scenarios (Barab et al., 2005). Situated learning can also
involve the adoption of multiple roles and perspectives and receiving guidance from
expert modeling (Bransford, Brown, & Cocking, 2000). Through immersive
interactions and gameplay, novice players can develop their skills by observing,
communicating and interacting with other expert players; essentially emulating how
junior scholars learn from their advisors in academic environments. In addition,
players acquire in-game terminology through interactivity and communication with
experts and novices, and language acquisition is an essential element to scaffolding.
Finally, immersive gameplay or other in situ interactions enable learners to
traverse the zone of proximal development (ZPD; Vygotsky, 1978), which refers to
the distance between what a learner can do with support by collaborating with peers
or through guided instruction, and what they can do without support. The acqui-
sition of knowledge begins with interaction, followed by the acquisition of lan-
guage which provides meaning and intent so that behaviors can be better
understood. Towards that end, well-designed IEs consist of rules, goals, feedback,
skill mastery, and interactivity. To achieve quantifiable outcomes, players must
acquire knowledge, skills, and other abilities. Immersive environments like digital
games also promote play which is integral for human development, and is vital to
assimilating and accommodating new information by interacting with a fluid
environment (Shute et al., 2015). Well-designed IEs promote learning by requiring
learners to apply various competencies (i.e., creativity, rule application, persistence)
to solve novel problems thereby providing meaningful assessment environments for
learners during gameplay. So how can these evolving competencies be accurately
measured and thereby used as the basis for good evidence-based learning support?
76 V. Shute et al.

5.3 Assessment in Immersive Environments

Assessment of student learning in IEs should not be measured using traditional

summative tests (Shute, Leighton et al., 2016a). Such standardized tests provide a
very narrow snapshot of student learning. Moreover, traditional assessments cannot
provide immediate feedback to support learning or adapt the environment to
learners’ needs. Therefore, it is not surprising that the question of how to administer
responsive, comprehensive, and balanced assessments within IEs is an emergent
and complex question. Immersive environments provide novel learning opportu-
nities that demand new assessment methodologies.
Shute (2011) used the term “stealth assessment” to refer to evidence-based,
ongoing, and unobtrusive assessments, embedded within IEs (e.g., digital games,
virtual reality, augmented reality). Stealth assessments capture, measure, and support
the development of learners’ targeted competencies in IEs which serve as vehicles
for learning. Stealth assessment can be used to adapt the environment to accom-
modate learners’ current levels/needs, as well as to provide appropriate feedback and
other types of learning support (Shute, Ke, & Wang, 2017). According to
Csikszentmihalyi (1990), such personalized support permits learners to maintain the
state of flow (note: adaptivity is further discussed in the next section).
As a learner interacts with the IE (e.g., an augmented reality activity or video
game), stealth assessment analyzes specific actions and interactions via data that are
captured in the log file to estimate the learner’s competency states in terms of
evidence-based claims. Stealth assessment creates a student model and continu-
ously updates it as the person interacts with the IE. Information from the student
model, then, can be used as the basis for which to provide relevant feedback and/or
adapt the IE to suit the learner’s needs. In the process, this creates a personalized
learning/playing experience.
Stealth assessment employs a principled assessment design framework called
evidence-centered design (ECD; Mislevy, Steinberg, & Almond, 2003). ECD
involves the development of conceptual and computational models (e.g., the com-
petency, evidence, and task models) that work together to accomplish valid
assessment (see Fig. 5.1).

Assessment Design

Diagnostic inferences

Fig. 5.1 Simplified ECD adapted from Mislevy et al. (2003)

5 Assessment for Learning in Immersive Environments 77

The first model in ECD framework is the competency model, which explicitly
specifies the knowledge, skills, and other attributes (collectively referred to as
“competencies” in this chapter) to be measured by the assessment. This is intended
to facilitate the operationalization of the construct with all of its associated facets and
observable behaviors. The second model is the evidence model, which specifies the
assignment of scores to the observable behaviors (i.e., the learner’s performance),
such as whether dichotomous (i.e., an item response or activity is assigned a value of
1 if correct, otherwise a 0) or polytomous (i.e., an item response or activity is
assigned values other than just 0 or 1 to show increasing performance quality)
scoring will be used, and how the scores will be accumulated. Finally, the third
model is the task model, which outlines the types of tasks, including all features,
requiring development to elicit the competencies of interest from the learner.
Stealth assessment’s evidence-based models work together to accomplish
ongoing analyses of all gameplay/interaction data. This provides more valid and
reliable assessment results compared to traditional summative tests. Shute et al.
(2017) delineate the steps for creating a stealth assessment in an IE:
1. Develop the competency model (CM) of targeted knowledge, skills, or other
attributes based on comprehensive literature and expert reviews
2. Determine the IE (e.g., a game or other immersive media applications) into
which the stealth assessment will be embedded
3. Create a full list of relevant actions/indicators that serve as evidence to inform
the CM and its facets
4. Create new tasks in the IE, if necessary
5. Create a matrix to link actions/indicators to relevant facets of target competencies
6. Determine how to score indicators by classifying them into discrete categories
for the “scoring rules” part of the evidence model (EM)
7. Establish statistical relationships between each indicator and associated levels
of CM variables using, for example, Bayesian Networks (BNs) (EM)
8. Pilot test the BNs and modify parameters
9. Validate the stealth assessment with external measures
10. Use the assessment estimates to provide feedback and targeted learning sup-
ports in the IE.
We now examine a worked example of a stealth assessment of problem-solving
skills that was developed and used within a modified version of a popular
immersive 2-dimensional game based on the steps described above.

5.4 An Illustration of Stealth Assessment in a Game

Environment

To make the process of creating a stealth assessment come alive, we present an

example in which a problem-solving stealth assessment was developed and built
into a game called “Use Your Brainz” (UYB; a modified version of the game Plants
78 V. Shute et al.

vs. Zombies 2; Shute, Wang, Greiff, Zhao, & Moore, 2016b). In the game, players
position a variety of special plants on their lawn to prevent zombies from reaching
their house. Each of the plants has different attributes. For example, some plants
(offensive ones) attack zombies directly, while other plants (defensive ones) slow
down zombies to give the player more time to attack the zombies. A few plants
generate “sun,” an in-game resource needed to utilize more plants. The challenge of
the game comes from determining which plants to use and where to position them
on the battlefield to defeat all the zombies in each level of the game.
To create a stealth assessment measuring problem-solving skills, Shute and
colleagues first developed a competency model of problem solving based on an
extensive literature review (step 1). The operationalized problem-solving CM
included four main facets: (a) analyze givens and constraints, (b) plan a solution
pathway, (c) use tools effectively/efficiently when solving the problem, and
(d) monitor and evaluate progress. In parallel with developing the problem-solving
CM, Shute and her team selected an appropriate IE (the UYB game) in which to
embed the stealth assessment (step 2). They selected this game for several reasons.
First, UYB requires ongoing problem-solving skills (like chess). Second, although
it is a 2D game, it can provide an immersive experience in that its engaging
environment requires players to continuously apply the various in-game rules to
solve challenging problems. Third, this work was part of a joint project with
GlassLab (see https://www.glasslabgames.org/), and Glasslab had access to the
game’s source code which allowed the researchers to modify the data to be captured
in the log files and embed the stealth assessment models directly into the game.
After finalizing the problem-solving competency model, Shute and her team
identified dozens of observable in-game indicators (after repeatedly playing the game
and watching expert solutions on YouTube). The indicators are used as evidence to
update the problem-solving CM (step 3; in this example step 4 was not needed). For
example, the research team determined that planting three or more sun-producing
plants (which provide the currency to use other plants) before the first wave of zombies
arrive is an indicator of the “analyze givens and constraints” facet and shows that the
player understands time and resource constraints. Table 5.1 includes some examples
of problem-solving indicators in UYB.
Table 5.1 Example indicators for problem solving (from Shute, Wang, et al., 2016b)
Facets Example indicators
Analyze givens and • Plants >3 Sunflowers before the second wave of zombies
constraints arrives
• Selects plants off the conveyor belt before it becomes full
Plan a solution pathway • Places sun producers in the back, offensive plants in the
middle, and defensive plants up front/right
• Plants Twin Sunflowers or uses plant food on (Twin)
Sunflowers in levels that require the production of X amount
of sun
Use tools and resources • Uses plant food when there are >5 zombies in the yard or
effectively/efficiently zombies are getting close to the house (within 2 squares)
• Damages >3 zombies when firing a Coconut Cannon
Monitor and evaluate • Shovels Sunflowers in the back and replaces them with
progress offensive plants when the ratio of zombies to plants exceeds
2:1
5 Assessment for Learning in Immersive Environments 79

The next task in the UYB project was to create a Q-matrix (Almond, 2010) with
the four problem-solving facets in columns and all of the relevant indicators listed
in rows (step 5; where the crossed cells contain the value of “1” if the indicator is
related to the facet and a “0” if they’re unrelated). Afterwards, they determined the
scoring rules (step 6). This entails deciding about how to score the indicators by
classifying them into discrete categories (e.g., yes/no, high/medium/low relative to
the quality of the actions). For example, if a player planted six sunflowers before the
second wave of zombies, the action will be automatically recorded as “yes” pro-
viding positive evidence of the first facet “analyze givens and constraints.”
After categorizing all indicators, Shute and her team connected each indicator to
the related CM variable(s) and established a statistical relationship between them
(step 7). They used Bayesian Networks to create the statistical relationships,
accumulate the incoming gameplay data, and update the beliefs in the competency
model (note: they created one BN for each level, 43 BNs in total). Why were BNs
used over other techniques? De Klerk, Veldkamp, and Eggen (2015) conducted a
systematic literature review on various analytical approaches used in
simulation-based and game-based assessments to analyze performance data (i.e., the
data generated by learners’ interaction with the IE). The most prevalent examples of
such analytic tools include Bayesian Networks (BNs), Exploratory and
Confirmatory Factor Analysis, Item Response Theory, Multidimensional Item
Response Theory, Cluster Analysis, Artificial Neural Networks, and Educational
Data Mining. Overall, BNs were the most used analytical and data modeling
framework to analyze learners’ performance data in game-based and
simulation-based assessment. Moreover, there are several advantages to using BNs
as a data modeling framework in IEs such as: (1) BNs provide an easy-to-view
graphical representation of the competency model (direct and indirect relationships
among variables) for clear operationalization; (2) BNs can “learn” from data as
they’re probability models (thus make probabilistic predictions)—the degree to
which observed data meet expectations of the model can help improve the original
model as more data become available; (3) Updating BNs is immediate (as perfor-
mance data come from the IE) compared to other analytical approaches (like IRT),
so they provide real-time diagnosis—overall and at sub-score levels; and
(4) Enhancements to BN software permit large and flexible networks with as many
variables as wanted (Almond et al., 2015). Moreover, by using only discrete
variables, BNs can be scored very quickly, making them suited for embedded
scoring engines.
Consider indicator #37 in Fig. 5.2 (use of iceberg lettuce in UYB). This indi-
cator is connected to the “tool use” facet, and a player has just performed some
action in the game which was judged as “poor” (e.g., placed an iceberg lettuce
proximal to a fire-breathing plant, thus cancelling out the “freezing” effect of the
lettuce). The real-time estimate that the learner is low on the “tool use” facet is
p = 0.61 (for more details see Shute, Wang, et al., 2016b).
When establishing the BNs for UYB, the game experts and psychometricians in
the team initially set the probabilities of the various states, per competency model
variable (i.e., the prior probabilities in BNs). However, after pilot testing the BNs,
80 V. Shute et al.

Fig. 5.2 An example of a BN with data for indicator #37 entered (poor use of iceberg lettuce)

data were used to modify the BN parameters (difficulty and discrimination)

accordingly (step 8). Finally, to validate the stealth assessment using external
measures (step 9), Shute and colleagues used two external measures of problem
solving skill: (a) Raven’s Progressive Matrices (Raven, 1941) which examines
inductive ability (i.e., rule identification) based on given information; and
(b) MicroDYN (Wustenberg, Greiff, & Funke, 2012), a simulation which measures
problem solving skills based on acquiring and applying existing information (i.e.,
rule application).
Validation study participants were 7th grade students (n = 55) from a middle
school in suburban Illinois. They students played the game for 3 h (1 h per day
across three consecutive days). The results showed that the students’ scores from
the two external tests significantly correlated with the in-game stealth assessment
estimates [Raven’s (r = 0.40, p < 0.01) and MicroDYN (r = 0.41, p < 0.01)].
Therefore, the stealth assessment embedded in the UYB game appears to be valid.
Other studies have been conducted using stealth assessment to measure various
competencies, e.g., physics understanding (Shute et al., 2013) and persistence
(Ventura & Shute, 2013). The overall findings from these studies also show sig-
nificant correlations between external and the in-game estimates. Finally, it’s
important to note that this assessment approach, while illustrated in a 2D envi-
ronment, can also be used in 3D games and environments (e.g., Portal 2 research,
see Shute et al., 2015). We now discuss the next logical steps to take—making IEs
adaptive based on assessment data.
5 Assessment for Learning in Immersive Environments 81

5.5 Next Steps

After creating and embedding a stealth assessment into an IE and testing its psy-
chometric properties (i.e., reliability, validity, and fairness), the next step is to
provide adaptive or personalized learning supports (e.g., appropriate feedback and
challenges) based on current estimates of competency states (Shute et al., 2017).
This type of adaptation (i.e., micro-adaptation; see Kickmeier-Rust & Albert, 2010)
keeps learners motivated to progress throughout the game/IE, engenders a state of
flow, and aligns with their ZPD.
As mentioned earlier, Csikszentmihalyi (1990) asserted that when learners are
fully engaged in tasks that are neither too difficult nor too easy, they enter the state
of flow in which they learn best. Similarly, Vygotsky (1978) believed that the best
learning experience happens when learners receive learning materials just beyond
their current knowledge or skill level. Research has shown that adaptive learning
activities generally yield better learning outcomes than non-adaptive activities (e.g.,
Kanar & Bell, 2013). We suspect that similar learning outcomes can be achieved
via adaptive IEs. Moreover, learning/playing in an adaptive IE can facilitate
learners’ self-efficacy (Bandura, 1994) and self-determination (Ryan & Deci, 2000)
because learners establish new beliefs about their personal capabilities when they
progressively tackle challenges that are tailored to their current ability levels. In
other words, the more learners overcome appropriately-challenging tasks, the more
efficacious they feel in the IE in which they interact. The gratifying experience of
efficacy makes the learners intrinsically motivated to continue facing new chal-
lenges (Klimmt, Hartmann, & Schramm, 2006).
To enhance learning—both processes and outcomes—learners’ state of flow
would be maintained by adjusting tasks/activities in the IE coupled with ongoing
targeted feedback. In theory, this would motivate them to persist and enhance their
self-efficacy (e.g., Van Oostendorp, van der Spek, & Linssen, 2013). To accomplish
this goal, accurate, ongoing, and unobtrusive measurements of learners’ current
competency states (relative to cognitive, non-cognitive, and even affective vari-
ables) are needed to continuously adapt the IE to the learners’ needs and capabilities
in real-time. Research is needed on how to best prioritize the skill or affective state
most in need of support.
One way to accomplish adaptation in an IE is via a task selection algorithm. For
instance, Shute, Hansen, & Almond (2008) developed an adaptive algorithm that
tends to select tasks for which the student has an approximately 50–50 chance of
solving correctly. These tasks are likely to reside within the student’s zone of
proximal development (Vygotsky, 1978) and hence may be good candidates for
promoting learning, particularly if accompanied by feedback. In contrast,
non-adaptive (e.g., linear) IEs/games may present fixed sequences of activities or
tasks, often arrayed from easy-to-difficult. This may lead to predictable and
impersonal learning/gameplay experiences (Lopes & Bidarra, 2011) and perhaps
boredom. Creating adaptive IEs empowered by stealth assessment is currently
under development and we expect to see positive results on students’ learning.
82 V. Shute et al.

5.6 Conclusions

Immersive technologies are now available to use in formal education settings as

they become more affordable. Historically, VR has been used in military training
for many years, MUVEs have been around for more than fifteen years, and AR has
been used in museums, factories, medical arenas, and the military since the early
1990s. Nonetheless, their use in public educational settings has not been feasible
due to the cost and availability of the technologies, until recently. Currently,
low-cost VR experiences are possible with products like Google Cardboard which
only costs $15 and a smart phone (Brown & Green, 2016). Furthermore, according
to a recent report (Adams Becker, Freeman, Giesinger Hall, Cummins, & Yuhnke,
2016), large investments are being made in the immersive media industry, and it is
expected that the education sector will benefit from these investments within the
next two to three years. In another report, Goldman Sachs predicted that the
immersive media industry has the potential of being an $80-billion market by 2025
(Bellini et al., 2016).
Because of these trends, many companies (e.g., Facebook, Samsung, Google,
and HTC) have entered the race for developing content with advanced technologies
to make the immersive media experience possible for all (Brown & Green, 2016).
Furthermore, industry leaders recognize the potential benefits of immersive
well-designed games just as learning theorists posit that gameplay experiences in
immersive environments can substantially improve learners’ problem solving skills
through multiple interactions with novel problem solving scenarios (e.g., Van Eck
& Hung, 2010). However, there are still barriers to adopting IEs in formal education
settings—mainly related to getting the assessment part right.
Our broad vision relating to assessment for learning involves the ongoing col-
lection of data as students interact within various IEs during and after regular school
hours. When these various data streams coalesce, the accumulated information can
potentially provide increasingly reliable and valid evidence about what students
know and can do across multiple contexts. To accomplish this goal, we need
high-quality, ongoing, unobtrusive assessments embedded in various IEs that can
be aggregated to inform a student’s evolving competency levels (at various grain
sizes) and aggregated across students to inform higher-level decisions (e.g., from
student to class to school to district to state, to country).
The primary goal of this idea is to improve learning, particularly learning pro-
cesses and outcomes necessary for students to succeed in the twenty first century,
such as persistence, creativity, problem solving skill, critical thinking, and other
constructs. Current approaches to assessment/testing are typically disconnected
from learning processes. With innovative assessment technologies like stealth
assessment, teachers do not need to disrupt the normal instructional process at
various times during the year to administer external tests to students. Instead,
assessment should be continuous and invisible to students, supporting real-time,
just-in-time instruction and other types of learning support in all types of IEs.
5 Assessment for Learning in Immersive Environments 83

For this vision of assessment—as ubiquitous, unobtrusive, engaging, and valid

—to gain traction, there are a several hurdles to overcome. Several immediate
concerns are presented here (for more details on challenges and future research, see
Shute, Leighton, et al., 2016a).
1. Ensuring the quality of assessments. U.S. schools are under local control, thus
students in a given state could engage in thousands of IEs during their educa-
tional tenure. Teachers, publishers, researchers, and others will be developing
IEs, but with no standards in place, they will inevitably differ in curricular
coverage, difficulty of the material, scenarios and formats used, and many other
ways that will affect the adequacy of the IE, tasks, and inferences on knowledge
and skill acquisition that can justifiably be made from successfully completing
the IEs. More research is needed to figure out how to equate IEs or create
common measurements from diverse environments. Towards that end, there
must be common models employed across different activities, curricula, and
contexts. Moreover, it is important to determine how to interpret evidence where
the activities may be the same but the contexts in which students are working
differ (e.g., working alone vs. working with another student).
2. Making sense of different learning progressions. IEs can provide a greater
variety of learning situations than traditional face-to-face classroom settings,
thus evidence for assessing and tracking learning progressions becomes more
complex rather than general across individual students. As a result, we need to
be able to model learning progressions in multiple aspects of student growth and
experiences, which can be applied across different learning activities and con-
texts. Moreover, there is not just one correct order of progression as learning in
IEs involves many interactions between individual students and situations,
which may be too complex for most measurement theories that assume linearity
and independence. So theories of learning progressions in IEs need to be
actively researched and validated to realize their potential.
3. Privacy/Security. This issue relates to the accumulation of student data from
disparate sources. However, information about individual students may be at
risk of being shared far more broadly than is justifiable. And because of the
often high-stakes consequences associated with tests, many parents and other
stakeholders fear that the data collected could later be used against the students.
Despite these obstacles, constructing the envisioned ubiquitous and unobtrusive
assessments within IEs across multiple learner dimensions, with data accessible by
diverse stakeholders, could yield various educational benefits. First, the time spent
administering tests, handling make-up exams, and going over test responses is not
very conducive to learning. Given the importance of time on task as a predictor of
learning, reallocating those test-preparation chores into meaningful pre-instructional
activities that are more engaging for learners can benefit almost all students.
Second, by having assessments that are continuous and ubiquitous, students are no
longer able to “cram” for an exam. Although cramming can provide good
short-term recall, it is a poor route to long-term retention and transfer of learning.
84 V. Shute et al.

Standard assessment practices in school can lead to assessing students in a manner

that conflicts with their long-term success. With a continuous assessment model in
place, the best way for students to perform well is to engage with the content,
interact with peers, and communicate ideas. The third direct benefit is that this shift
in assessment mirrors the national shift toward evaluating students on acquired
competencies. With increasing numbers of educators growing wary of traditional,
high-stakes tests for students, ensuring students have acquired the “essential” skills
needed to succeed in twenty first century workplace environments are consistent
with the innovative type of assessment outlined in this chapter.
There is a need for innovative assessments given (a) the urgency for supporting
new twenty first century skills, and (b) the increased availability of immersive
technologies, both of which make it easy to capture the results of routine student
work—in class, at home, or any place with available broadband access. One pos-
sibility is for twenty first century assessments to be so well integrated into students’
day-to-day lives that they are unaware of its existence. This represents quite a
contrast to our current testing contexts. However, while the benefits of using a
seamless-and-ubiquitous model to run a business have been clear for more than four
decades (e.g., using barcodes), applying this metaphor to education may require
modifications given the desired outcome is knowledge rather than financial capital.
For instance, there are certain risks to consider: students may come to feel like they
are constantly being evaluated which could negatively affect their learning by
causing unwanted stress. Another risk of a continuous assessment approach in
education could result in teaching and learning turning into ways to “game the
system” depending on how it is implemented and communicated. But the afore-
mentioned hurdles and risks, being anticipated and researched in advance, can help
to shape the vision for a richer, deeper, more authentic assessment (to support
learning) of students in the future.

Acknowledgements This research did not receive any specific grant from funding agencies in the
public, commercial, or not-for-profit sectors, however we do acknowledge the intellectual support
by Chris Dede and John Richards, as well as various reviewers that we received while writing this
chapter.

References

Adams Becker, S., Freeman, A., Giesinger Hall, C., Cummins, M., & Yuhnke, B. (2016). Horizon
Report: 2016 K-12 Edition. Austin, TX: New Media Consortium.
Almond, R. G. (2010). Using evidence centered design to think about assessments. In V. J. Shute
& B. G. Becker (Eds.), Innovative assessment for the 21st century: Supporting educational
needs (pp. 75–100). New York: Springer.
Almond, R. G., Mislevy, R. J., Steinberg, L., Yan, D., & Williamson, D. (2015). Bayesian
networks in educational assessment. New York: Springer.
Bandura, A. (1994). Self-efficacy: Toward a unifying theory of behavioral change. Psychological
Review, 84(2), 191–215.
5 Assessment for Learning in Immersive Environments 85

Barab, S., Thomas, M., Dodge, T., Carteaux, R., & Tuzun, H. (2005). Making learning fun: Quest
Atlantis: A game without guns. Educational Technology Research and Development, 53(1),
86–107.
Barfield, W. (2015). Fundamentals of wearable computers and augmented reality (2nd ed.). New
York: CRC Press.
Bellini, H., Chen, W., Sugiyama, M., Shin, M., Alam, S., & Takayama, D. (2016). Profiles in
innovation: Virtual & augmented reality. Goldman Sachs group, inc.
Bransford, J., Brown, A., & Cocking, R. (2000). Brain, mind, experience, and school.
Washington, DC: National Academies Press.
Brown, A., & Green, T. (2016). Virtual reality: Low-cost tools and resources for the classroom.
TechTrends, 60(5), 517–519. doi:10.1007/s11528-016-0102-z.
Clark, D., Tanner-Smith, E., & Killingsworth, S. (2014). Digital games, design, and learning: A
systematic review and meta-analysis. Review of Educational Research, 86(1), 79–122. doi:10.
3102/0034654315582065.
Csikszentmihalyi, M. (1990). Flow: The psychology of optimal experience. New York: Harper and
Row.
Dede, C. (2009). Immersive interfaces for engagement and learning. Science, 323(5910), 66–69.
doi:10.1126/science.1167311.
De Klerk, S., Eggen, T. J. H. M., & Veldkamp, B. P. (2015). Psychometric analysis of the
performance data of simulation-based assessment: A systematic review and a Bayesian
network example. Computers & Education, 85, 23–34.
Gee, J. P. (2008). Learning and games. In K. Salen (Ed.), The ecology of games: Connecting
youth, games, and learning (pp. 21–40). Cambridge, MA: The MIT Press. doi:10.1162/dmal.
9780262693646.021.
Hew, K. F., & Cheung, W. S. (2010). Use of three-dimensional (3-D) immersive virtual worlds in
K-12 and higher education settings: A review of the research. British Journal of Educational
Technology, 41(1), 33–55.
Kanar, A. M., & Bell, B. S. (2013). Guiding learners through technology-based instruction: The
effects of adaptive guidance design and individual differences on learning over time. Journal of
Educational Psychology, 105(4), 1067.
Ketelhut, D. J. (2007). The impact of student self-efficacy on scientific inquiry skills: An
exploratory investigation in river city, a multi-user virtual environment. Journal of Science
Education and Technology, 16(1), 99–111.
Kickmeier-Rust, M. D., & Albert, D. (2010). Micro-adaptivity: Protecting immersion in
didactically adaptive digital educational games. Journal of Computer Assisted learning, 26
(2), 95–105.
Klimmt, C., Hartmann, T., & Schramm, H. (2006). Effectance, self-efficacy, and the motivation to
play video games. In J. Bryant & P. Vorderer (Eds.), Psychology of entertainment (pp. 291–
313). Mahwah, NJ: Erlbaum.
Knight, J. F., Carley, S., Tregunna, B., Jarvis, S., Smithies, R., de Freitas, S., et al. (2010). Serious
gaming technology in major incident triage training: A pragmatic controlled trial.
Resuscitation, 81(9), 1175–1179.
Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge,
MA: University of Cambridge Press.
Liu, C., Cheng, Y., & Huang, C. (2011). The effect of simulation games on the learning of
computational problem solving. Computers & Education, 57(3), 1907–1918.
Lopes, R., & Bidarra, R. (2011). Adaptivity challenges in games and simulations: A survey. IEEE
Transactions on Computational Intelligence and AI in Games, 3(2), 85–99.
Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2003). Focus article: On the structure of
educational assessments. Measurement: Interdisciplinary Research and Perspectives, 1(1),
3–62.
Piaget, J. (1973). To understand is to invent: The future of education. New York: Grossman.
Raven, J. C. (1941). Standardization of progressive matrices, 1938. British Journal of Medical
Psychology, 19(1), 137–150.
86 V. Shute et al.

Ryan, R. M., & Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic
motivation, social development, and well-being. American Psychologist, 55(1), 68–78.
Shute, V. J. (2011). Stealth assessment in computer-based games to support learning. Computer
Games and Instruction, 55(2), 503–524.
Shute, V. J., Hansen, E. G., & Almond, R. G. (2008). You can’t fatten a hog by weighing It–Or
can you? Evaluating an assessment for learning system called ACED. International Journal of
Artificial Intelligence in Education, 18(4), 289–316.
Shute, V., Ke, F., & Wang, L. (2017). Assessment and adaptation in games. In P. Wouters &
H. van Oostendorp (Eds.), Techniques to facilitate learning and motivation of serious games
(pp. 59–78). New York, NY: Springer.
Shute, V. J., Leighton, J. P., Jang, E. E., & Chu, M. (2016a). Advances in the science of
assessment. Educational Assessment, 21(1), 1–27.
Shute, V. J., Ventura, M., & Ke, F. (2015). The power of play: The effects of portal 2 and lumosity
on cognitive and noncognitive skills. Computers & Education, 80, 58–67.
Shute, V. J., Ventura, M., & Kim, Y. J. (2013). Assessment and learning of qualitative physics in
newton’s playground. The Journal of Educational Research, 106(6), 423–430.
Shute, V., Ventura, M., Kim, Y., & Wang, L. (2014). Video games and learning. In W. G. Tierney,
Z. B. Corwin, T. Fullerton, & G. Ragusa (Eds.), Postsecondary play: The role of games and
social media in higher education (pp. 217–235). Baltimore, MD: John Hopkins Press.
Shute, V. J., Wang, L., Greiff, S., Zhao, W., & Moore, G. (2016b). Measuring problem solving
skills via stealth assessment in an engaging video game. Computers in Human Behavior, 63,
106–117.
Tennyson, R. D., & Jorczak, R. L. (2008). A conceptual framework for the empirical study of
instructional games. In H. F. O’Neill & R. S. Perez (Eds.), Computer games and team and
individual learning (pp. 3–20). Boston: Elsevier.
Van Eck, R., & Hung, W. (2010). A taxonomy and framework for designing educational games to
promote problem solving. In paper presentation at the videogame cultures & the future of
interactive entertainment annual conference of the inter-disciplinary. Net Group, 227–263.
Van Oostendorp, H., Van der Spek, E., & Linssen, J. (2013). Adapting the complexity level of a
serious game to the proficiency of players. EAI Endorsed Transactions on Serious Games,
1(2), 8–15.
Ventura, M., & Shute, V. (2013). The validity of a game-based assessment of persistence.
Computers in Human Behavior, 29(6), 2568–2572.
Vygotsky, L. (1978). Mind in society: The development of higher psychological processes.
Cambridge, MA: Harvard University Press.
Wüstenberg, S., Greiff, S., & Funke, J. (2012). Complex problem solving—More than reasoning?
Intelligence, 40(1), 1–14.

Author Biographies

Valerie Shute is the Mack & Effie Campbell Tyner Endowed Professor in Education in the
Department of Educational Psychology and Learning Systems at Florida State University. Before
coming to FSU in 2007, she was a principal research scientist at Educational Testing Service
where she was involved with basic and applied research projects related to assessment, cognitive
diagnosis, and learning from advanced instructional systems. Her general research interests hover
around the design, development, and evaluation of advanced systems to support learning–
particularly related to twenty first century competencies. Her current research involves using
games with stealth assessment to support learning—of cognitive and noncognitive knowledge,
skills, and dispositions. Her research has resulted in numerous grants, journal articles, books,
chapters in edited books, a patent, and a couple of recent books (e.g., Shute & Ventura, 2013,
5 Assessment for Learning in Immersive Environments 87

Measuring and supporting learning in games: Stealth assessment, The MIT Press; and Shute &
Becker, 2010, Innovative assessment for the twenty first century: Supporting educational needs,
Springer-Verlag). She is also the co-founder of www.empiricalgames.org.

Seyedahmad Rahimi is a Doctoral Candidate at Florida State University in the Instructional

Systems & Learning Technologies program. He is also getting his second MS degree in
Measurement & Statistics at Florida State University. He holds a BS degree in Computer Software
Engineering from Islamic Azad University in Iran as well as an MS degree in e-learning
technologies from Multimedia University in Malaysia. His Master’s thesis was about the
perception of college-level students on immersive learning environments like Second Life. He is
currently working with Dr. Valerie Shute as one of her Research Assistants on an NSF grant
examinining learning supports and adaptivity in games. His general research interests include
game-based stealth assessment and adaptivity in learning games.

Benjamin Emihovich is a Doctoral Candidate in the Instructional Systems & Learning

Technologies program at Florida State University. He holds a M.Ed. from the University of Florida
and a B.A. in Psychology and Social Behavior from the University of California, Irvine. He is
interested in exploring how well-designed video games can be used to improve a wide range of
knowledge, skills, and abilities referred to as game-based learning (GBL). His dissertation research
measures the impact of video gameplay on undergraduates’ problem-solving skills. Video games
have broad appeal with plenty of research opportunities available to meet the demands of a diverse
learner population and those at-risk of failing.
Chapter 6
Infrastructures for Immersive Media
in the Classroom

John Richards

Abstract VR, AR, and MR are becoming ubiquitous in consumer gaming, military
applications, and office environments. These successes are driving emerging efforts
to integrate these immersive media into the K-12 classroom. In this chapter, first we
summarize the distribution and availability of the infrastructure needed for using
VR and MR in the schools. Using immersive media requires a technology infras-
tructure consisting of dependable high-speed Internet connectivity to the classroom,
a ratio of at least one-to-one computer to student, an interactive white board, and
curriculum materials that can be monitored and controlled by the teacher. This
infrastructure is quickly becoming a reality. However, a larger and more complex
barrier remains: integrating the new technologies with existing classroom systems
and with existing and emerging pedagogical practice. I argue that the Digital
Teaching Platform serves as a model for classroom practice. The evolving nature of
digital curricula, formative assessment, and classroom practice impact how teachers
will be able to integrate these new technologies. Finally, I examine how immersive
media such as virtual reality, augmented reality, mixed reality, and multi-user
virtual reality can work as supplemental digital materials for instruction and
assessment. In particular, I focus on the sensory comfort and fidelity of interaction
as these issues impact the viability of these technologies in the classroom.

Keywords Digital teaching platforms Distributed digital curriculum


Formative assessment Real-time classroom management VR, AR and MR

in the classroom

J. Richards (&)
Harvard University, 13 Appian Way, Cambridge, MA 02138, USA
e-mail: [email protected]
URL: http://www.cs4ed.com
J. Richards
Consulting Services for Education, Inc., 22 Floral St., Newton, MA 02461, USA

© Springer Nature Singapore Pte Ltd. 2017 89

D. Liu et al. (eds.), Virtual, Augmented, and Mixed Realities in Education,
Smart Computing and Intelligence, DOI 10.1007/978-981-10-5490-7_6
90 J. Richards

6.1 Introduction

The other chapters in this volume focus on “learning” in a virtual, augmented, or

mixed environment. These chapters lay a research foundation for bringing the
immersive1 technologies into the classroom. However, if these new technologies
are to be more than a fad or distraction, they must be integrated into a compre-
hensive “hybrid” curriculum that includes digital, print, and hands-on exploration.
In this chapter we consider the use of digital infrastructure as the fundamental
carrier of such a core curriculum, even as it is being invented.2
The challenge is how immersive media will fit in the institutional K-12 market.
First, we look at the state of school technology to assess its support for a curriculum
and a classroom that are digital at their core. Second, we introduce the notion of a
Digital Teaching Platform as a framework for understanding the dynamics of
introducing VR/AR/MR into a teacher-led classroom where the digital environment
is the major carrier of curriculum content and provides the functions of a primary
instructional environment. Finally, we look the issues of sensory comfort and the
potential for the fidelity of interaction in VR, MR, and AR. The state of the art is
moving quickly, but issues of nausea stand in the way of full acceptance in an
institutional environment.

6.2 Classroom Technology Infrastructure

In this section, we summarize the existing distribution and availability of band-

width, connectivity, equipment, and management systems required for supporting
any digital curriculum in the classroom. Administrators want access to data and
analysis, teachers want dependable class sessions, and students want universal
access for lessons, homework, and practice.

6.2.1 The VR/AR/MR Industry

According to BCC Research, the global VR/AR market for Education was $548.8M
in 2014, $1025.40M in 2015 and is expected to grow to almost $16B by 2020

1
As mentioned in the Introduction, we use the term “immersive media” to refer to virtual reality,
augmented reality, mixed reality, and MUVEs.
2
We have restricted this analysis to the K-12 world. In higher education, as in the consumer
market, there is ample bandwidth, students have multiple devices, and digital infrastructure is
readily available in the learning environment Moreover, unlike K-12 education, higher education
offers many interesting options for individual exploration and independent extension of the
curriculum.
6 Infrastructures for Immersive Media in the Classroom 91

Table 6.1 Global market for virtual and augmented reality by sector (BCC, 2016, pp. 7–8)
($ Millions)
Sector 2014 2015 2020 CAGR% 2015–2020
Gaming 1404.50 2521.40 32,030.40 66.3
Military 1051.00 1839.90 20,808.90 62.4
Healthcare 636.7 1150.20 15,115.70 67.4
Education 548.8 1025.40 15,591.60 72.3
Industrial training 496.8 901.8 12,315.70 68.7
Others 362.2 661.3 9337.70 69.8
Total 4500.00 8100.00 105,200.00 67.0

(BCC, 2016, p. 7). Success at the extremes—in consumer gaming and military
applications—is pushing innovation, investment, and lower prices for VR/AR/MR.
Interestingly, the Education market is consistently is larger than the Industrial
Training market (see Table 6.1).
Immersive media require dependable high-speed Internet connectivity, at least
one-to-one device to user ratios, interactive displays, and digital materials that can
be monitored and controlled locally and dynamically. Until recently, almost none of
this existed in K-12 education environments. However, what is becoming ubiqui-
tous in the consumer and office environments is rapidly emerging for K-12.

6.2.2 Classroom Infrastructure

In 2004, the World Summit on the Information Society (WSIS) established goals in
order to “enhance the development of a global information society” [ITU, 2014,
p. 1]. In particular, one of the four key goals was, “adapting primary and secondary
school curricula to meet the challenges of the information society” [ibid, p. 2]. This
was translated into the second of eleven targets: “Connect all secondary schools and
primary schools with information and communication technologies” (ITU) [ibid].
One of the indicators that is relevant to VR/AR is the proportion of schools with
Internet access.

6.2.2.1 Internet Access for Schools

The proportion of schools with Internet access is near 100% in Europe, North
America, and the OECD (ITU 2014, pp. 72–3). In addition, according to the U.S.
Department of Education, in the U.S. as of 2008 (the last year for which data was
collected) 98% of classrooms had Internet access (Digest, 2016, p. 231), that would
be effectively 100% by now.
92 J. Richards

6.2.2.2 Device to Student Ratios

The second WSIS indicator is the learners-to-computer ratio. Assuming the need for
at least one-to-one device per student ratio for VR/AR, this is critical for imple-
mentation. The learners-to-computer ratio is lowest in North America, Europe, and
other OECD countries, and highest in developing countries (ITU, 2014, p. 51) (see
Figs. 6.1, 6.2 and 6.3).

11

Fig. 6.1 Learner to computer ratio, Asia [ITU, p. 66]. Source UIS database, Partnership on
Measuring ICT for Development WSIS Targets Questionnaire, 2013

LCR

LCR in schools with CAI

Fig. 6.2 Learner-to-computer ratio, Africa and the Americas [ITU, p. 67]. Source UIS database,
Partnership on Measuring ICT for Development WSIS Targets Questionnaire, 2013
6 Infrastructures for Immersive Media in the Classroom 93

Fig. 6.3 Learner-to-computer ratio, Oceania and Europe [ITU, p. 67]. Source UIS database,
Partnership on Measuring ICT for Development WSIS Targets Questionnaire, 2013

It is important to note that these figures are from a 2014 report, and the data
comes from several years before that. There have been significant reductions in the
cost per device since the time of the actual survey. As a result we believe the ratio
of students to device are lower.

6.2.3 Interactive Displays

The shift to Chromebooks, tablets, and smartphones also has an immediate impact
on the classroom. comScore reports that, in the U.S., “total digital media usage”
nearly tripled between 2010 and 2015, with smartphone usage accounting for most
of this. In the two years between December 2013 and December 2015, the report
cites a usage increase of 78%. Tablets grew 30% over this two year period.
(comScore, 2016, p. 5). While this data is restricted to the U.S., in my judgment the
switch to smartphones and tablets is clearly a global phenomenon. According to
IDC, Samsung and Apple are the global leaders in the Smartphone market (IDC,
August 2015).
According to Market Data Retrieval, 69% of districts in the U.S say they have
“substantially implemented” interactive white boards, and 69% say they have
“substantially implemented” the standard projector (MDR, 2015, p. 65).
94 J. Richards

6.2.4 Digital Management Systems

As we have seen, the technological infrastructure for the print-to-digital curricular

transition is mostly in place: bandwidth, connectivity, access to devices from
computers to tablets to smart phones, and interactive whiteboards. In addition, there
has been a significant investment in enterprise management systems including
learning management systems (LMS), student information systems (SIS), and
(although lagging behind) data warehouse and analytics systems. This is certainly
enough to support the basic transition from print to digital curriculum; but, as we
shall see in the next section this transition requires much more.

6.3 Modelling the Digital Classroom

To better understand the dynamics of the device-intensive, networked, digital

curriculum classroom, we present a model for the role of digital media in the
classroom. The Digital Teaching Platform (DTP) is a model of school based
learning designed to bring interactive technology to teaching and learning in
classrooms in which each student and the teacher has a laptop, or some equivalent
computational device, connected to the network (see Dede & Richards, 2012).
The very personal and immersive technologies that enrich the student’s expe-
rience by their very nature separate the student from the classroom environment.
There is precedent for this in the financially successful, but pedagogically limited
“Integrated Learning Systems” (ILS’s) of the ‘80s and ‘90s. ILS’s provided
one-on-one tutoring. Students had their own computer with headphones. This
separation of the student meant that the ILS’s were relegated to separate computer
labs or to a couple of computers in the back of the classroom. In either case,
students were pulled out of the curriculum, and the lessons were claimed to be
“teacher-proof.” For many reasons this approach failed, but from our perspective
the most significant is that it ignored the central role of the teacher in the school. As
with all classroom media, the teacher needs to be engaged, to understand when to
use these, for which students, and under what circumstances.
For VR/AR/MR to succeed in the institutional setting it must be incorporated into
the school culture, and treated like any other supplemental curriculum component. It
must meet the demands of the school environment. In particular it must be:
• Aligned to curriculum standards;
• Compliant to technological standards;
• Integrated into the curriculum;
• Linked to formative assessment; and
• Supported by professional development.
6 Infrastructures for Immersive Media in the Classroom 95

6.3.1 Digital Teaching Platforms

DTP is a model of school-based learning experiences. The model provides a

framework to understand the dynamics of a teacher-led classroom where the digital
environment is the major carrier of curriculum content, and functions as the primary
instructional environment in today’s technology-intensive classrooms. There are
two feedback loops in the model. The main classroom loop displays the dynamic
between the teacher, students, and curriculum. This is a continuous feedback sys-
tem as the teacher manages the classroom, facilitates learning, and monitors student
progress. The inner, technology-intensive personalized practice loop is individu-
alized dynamically for each student (Fig. 6.4).
A full-fledged DTP addresses three major requirements of contemporary class-
rooms: First, a DTP models a completely realized, networked digital environment
that includes interactive interfaces for both teachers and students. Teachers use the
administrative tools of this digital environment to create lessons and assignments
for students and to manage and evaluate the work the students return. A DTP
includes specific tools for assessment: for creating tests, assigning them to students,
and reviewing the results. The teacher tools also provide timely reports on student
progress or their remedial needs. The administrative tools for students allow them to
complete assignments and assessments. More importantly, these tools allow for
both individual and group work. Some students can work independently on indi-
vidualized assignments, while others can work collaboratively on shared
assignments.
Second, a DTP provides the content of the curriculum and assessments for
teaching and learning in digital form. This content includes all of the information in
the curriculum, the instruction, the exercises, and the assessments. The content also
includes interactive elements, manipulative activities, special-purpose applications,
multimedia materials, and so on.

Fig. 6.4 The inner Assessment-Content Management loop provides personalized practice. The
outer curriculum loop provides space for VR/AR/MR applications
96 J. Richards

Third, a DTP supports real-time, teacher-directed interaction in the classroom. It

includes special tools for managing classroom activity; for monitoring progress on
assignments; for displaying student work, demonstrations and challenges on
interactive displays; for managing group discussions; and for coordinating all
large-group and small-group activities.
All of these capabilities of a DTP are designed to function effectively in the
give-and-take atmosphere of a regular classroom. The teacher can shift quickly
from large group demonstrations, to small group activities, to individualized
practice and assessment. Students move seamlessly from using their devices for
some of these activities to closing their computers and participating in discussions.
The teacher is fully in control of student activities by making assignments, men-
toring individuals, and leading discussions. In DTPs, the pedagogy of the cur-
riculum is designed using principles of guided social constructivism as a theory of
learning, and the system provides the support for a transformation of teaching and
learning.
The teacher, as facilitator, is a demanding role. We speak of this as under-
standing the choreography of the classroom—the movement of the teacher:
• From the desktop and interactive whiteboard;
• To the small groups collaboratively solving problems;
• To the half-class lesson;
• While managing individuals practicing their individualized problem set.
Being the “guide on the side” is a new dynamic for many teachers and the DTP
needs to provide scaffolding for the teacher in this role.
The DTP model incorporates a curriculum model, an assessment model, an
instructional model, a classroom management model, an infrastructure model, and a
policy model (at minimum). In the next two sections we look at the digital cur-
riculum, and the assessment models. When we present the classroom architecture,
we look at models for instruction, classroom management, and infrastructure.

6.3.2 Curriculum

The digital curriculum has the potential to provide the student, or students, with
exploratory environments that can challenge their curiosity and creativity. The
teacher can pose problems for the class, or a part of the class on the interactive
white board. These fit the classroom just as the manipulatives have, and the science
laboratory. This requires that the curriculum move far beyond the e-book page
turner. Simulations, time line tools, geographic information services, can bring the
world into the classroom to engage the student.
The traditional curriculum, as expressed in textbooks, is created once and pre-
served in the print copies. Traditionally, particularly in adoption states, warehouses
were required to store enough copies to cover seven years usage. The result is a
6 Infrastructures for Immersive Media in the Classroom 97

stagnant, fixed document that is not subject to change. Teachers did modify the
curriculum by adding supplemental materials, by omitting sections of the text, or by
changing the order.
In contrast, digital curricula are distributed to each device in the classroom. In so
doing, there is the possibility of personalization at several levels. The teacher can
modify the curriculum as was done with textbooks, but in addition the actual
materials could be selected for any individual student, or for a group of students.
This could be done by the teacher by modifying lessons, by the formative engine
adjusting to student performance, or, potentially, by the student seeking deeper
materials as part of a project, or more or less challenging materials in practice.
The ability to modify the curriculum applies not only to the depth that the main
curriculum provider supplies, but also to the inclusion of supplemental materials.
These are traditionally videos, media, interactive learning objects that promote
exploration and challenge the students. This is where AR/VR/MR can be integrated
into the existing curriculum.
The challenge is to allow the system to accept these additional materials. They
must be aligned to the curriculum and appropriate standards, metadata must be
commensurate with the system, and the teacher has to know when they would fit
with the lesson.

6.3.3 Assessment

The traditional Learning Management System (LMS) maintains a close connection

between content delivery and student assessment. The LMS model invoked a
constrained curriculum delivery with regular feedback from the assessments for
learning, as seen in Fig. 6.5.
The nature of the assessment in the traditional LMS is summative. At the end of
a curriculum unit there is a test with a mapping either back into the unit or on to the
next unit. The metaphor is that there is a “boxcar” of content, and the assessment
looks at the student as they enter and leave this boxcar. While this inner loop looks
the same in a DTP, the nature and purpose is quite different. First, the LMS content

Content Assessment Grade SIS

Fig. 6.5 Testing—curriculum practice loop

98 J. Richards

is the curriculum with the intent to provide instruction. In the DTP, the content is
clearly practice, and the purpose is to provide this practice in support of the
classroom instruction. The assessment is ongoing formative assessment that is
constantly interacting with the student’s performance.

6.3.4 Classroom Architecture

The basic architecture is constructed around a Digital Asset Manager that provides
access to tools, and a variety of libraries of lessons, projects, videos, graphics, and
simulations. All of this is integrated with meta-data to connect to standards and an
assessment engine. Multiple curricula can be constructed in this environment. The
classroom functionalities are controlled by a Student-Facing App and a
Teacher-Facing App (see Fig. 6.6).
The Student Facing App gives the student access to challenging, interesting, and
dynamic content. The VR experience is part of a full complement of media for the
student to explore and analyze. This is the heart of the student-centric pedagogy that
changes the dynamic of the classroom. In addition to interactive lessons, the student
will also have access to basic tools such as journaling, planning, access to feedback
from the teacher or other students, and assignments. The student needs a
workspace/lab to assemble assignments. This should allow the ability to take screen
snapshots of work and incorporate into their journals or assignments.
The Student-Facing App provides the student with:
• Lesson/Tools Interaction
• Content Sequencing
• Assignments
• Assessment

Fig. 6.6 Classroom side of the DTP architecture

6 Infrastructures for Immersive Media in the Classroom 99

• Progress Reporting
• Social Networking/Communication.
The Teacher-Facing App gives the teacher control over the classroom and the
ability to monitor, adapt, and modify the curriculum. The structure of lessons must
be completely transparent so that the teacher can modify parts of a lesson. As a part
of teachers’ lesson planning and classroom management they should have access to
a dashboard that provides information on any individual student as well as
cross-class information about specific topics. The teacher needs to be able to assign
different sets of materials to different students.
An important part of the teacher’s ability to modify lessons is a Lesson Planning
tool that provides a framework for planning curriculum (calendar, plans for a day
including sequencing activities, and differentiation of instruction and materials).
Thus, the Teacher-Facing App provides the teacher with:
• Real-Time Classroom Management
• Lesson Planning/Calendar
• Assessment Tools
• Reporting
• Social Networking/Communication
• Tools for Managing Students and Classes
• Dashboards and Tools to Review Student Progress, Work, and Performance.

6.4 Special Considerations for Using VR/AR/MR3

This section expands upon the concept of comfort (see Jacobson, Chap. 3), which is
important with any media. The user experience should be comfortable and efficient,
providing a seamless experience. This essential with immersive media, especially
VR and MR, because they depend on creating sensory effects that closely mimic or
parallel effects in the real world. Most of these effects are triggered by user actions,
especially when s/he looks in a particular direction, uses the controls to navigate
(in VR), or uses the controls to manipulate a virtual object.
There are literally decades of literature on best practices with VR engineering
and human factors (Kronqvist, Jokinen, & Rousi, 2016) (Oculus, 2016). Here, we
will present some of the most important factors for immersive media, especially
those that cut across the sub-media types.

3
The author wants to acknowledge Jeffrey Jacobson who contributed significantly to this section.
100 J. Richards

6.4.1 Field Of View (FOV) and Field Of Regard (FOR)

Humans have approximately a 180°–200° horizontal field of view, but only

approximately 30° in the center of that comprises our highly detailed central vision.
The rest is peripheral vision, which sees much less detail, is less sensitive to color,
but is more sensitive to motion.
The wider the FOV in the VR display, the more realistic and comfortable the
experience will be, if other aspects of the display are managed well. It allows the
user to spend less time looking around and to see more things in relation to each
other. Fortunately, today’s commodity head-mounted displays provide a 90° field
of view, which is acceptable. Their main strength is a fully 360° field of regard,
which lets the viewer look in any direction. The visual cortex does an excellent job
of effortlessly stitching together what the eye sees, as the user looks around, pro-
viding a panoramic sense of the virtual world.
Field of View is also important in AR applications, because it limits what you
can add to the physical world. When the user of a head-mounted AR display turns
their head past the FOV capability of the device, the AR objects disappear, which
usually breaks the illusion. Unfortunately, this is true of most MR capable headsets
available to the public.
Typically a cave or dome has a field of view at least 180°; this is often wider
horizontally and vertically in the case of a dome or a CAVE with the ceiling. They
rarely include a projection onto the floor, however, and the user always has a
number of stable objects around them to remind them that they are still in the
physical world.

6.4.2 Fast View Update for HMDs

As we discuss in the introductory chapter, a head mounted display (HMD) allows

the user to look in any direction and see one of three things: a completely virtual
reality (VR), a mixed reality (MR), or a panoramic film, also known as 360 film.
For this to work, the user’s view must update immediately, with no lag or jitter. The
image should be refreshed at 90 frames-per-second or better. Otherwise, it breaks
the illusion of being in the virtual space (VR or film), or the virtual objects appear to
float strangely (MR).
Unfortunately, designers tend to build too much into their VR environments or
their AR objects. The temptation to add more geometry, more textures, and more of
everything is nearly irresistible. The result is a poor frame rate, which is unac-
ceptable. One advantage of 360 films is that anything the camera can capture can be
represented, with no effect on how quickly the image refreshes.
Interestingly, even the worst dome theaters and CAVEs do not have this limi-
tation, because all the pixels in the panoramic view are there to be seen in the single
view. The view must only refresh at a minimum of 30 frames per second (fps), as
6 Infrastructures for Immersive Media in the Classroom 101

they would in a typical movie theater. The downside, of course, is that the view is
always less than 360, often much less. Similarly, when Mixed Reality (MR) is
implemented with projectors that add digital objects to the physical world, they
only need 30 fps to look stable, as with any other projection.

6.4.3 Motion Sickness in Virtual Reality

This has always been a major problem for VR applications, and to some extent
always will be. It arises from a basic sensory conflict, where some sense report to
the brain that the user is moving through the virtual environment while others report
that s/he is not moving at all. Cataloguing all the factors is beyond the scope of this
article, but the most important ways to reduce motion sickness are (1) fast view
update (2) otherwise good quality image (3) minimize use of continuous movement,
in favor of jump or teleport, (4) and much more. See Lawson (2014) and Davis,
Nesbitt, and Nalivaiko (2014) for more information.

6.4.4 The Registration Problem in Mixed Reality

The hardest thing in Mixed Reality (MR) is called the registration problem, which
refers to making the digital objects apparently stay in one place in relation to the
physical world. The device has to somehow be aware of the user’s location and
direction of gaze on the real world. Then it redraws the digital object(s) in the
correct physical location 90 times per second,4 regardless of how quickly the user is
looking around and/or moving.
Lower-end AR applications can use a simple camera, like the one on smart-
phone, but this is reliable only when it has a high-contrast symbol to lock onto. This
supports many useful applications, but is ultimately quite limiting. More advanced
tools analyze the geometry of the physical environment, either using the camera or
Lidar, infrared, or some other scanning technology. All of these solutions require a
great deal of processing power, which makes fast updates difficult. There has been a
great improvement recently, with displays such as the Microsoft HoloLens, the
Google Tango, and Meta glasses, with many others on the horizon. Nevertheless
registration remains a difficult problem and will be an obstacle for some time.

4
This standard is somewhat subjective. 90 frames per second is acceptable, today.
102 J. Richards

6.4.5 Fidelity of Interaction

If the user is expected to interact with the experience, then the application must be
extremely responsive, which is challenging in VR and MR. If the device is tracking
the user’s body, it must be accurate and precise. The interaction between the body
and the virtual objects must also be crisp and accurate. Even when the control
devices are “Low-end” or primitive, such as using an Xbox 360 controller to
navigate, that interaction must also be responsive and carefully tuned.
Interestingly, users “holding” a virtual tool will tolerate a great deal of indi-
rection between their hand and the business end of the tool. It can look completely
different from the tracked object they are holding, it can move at a different scale,
and it can be oddly shaped. Nevertheless, people are able to use the tool effectively,
as long as the interaction is crisp and they can see the results. This is because
hand-eye coordination and other capabilities that the brain employs for tool using
are very highly developed in humans.

6.5 Conclusions: The Impact and Viability of AR, VR,

and MR in the K-12 Classroom

Even though research has been conducted for over twenty-five years, we have not
seen large-scale successful applications of immersive learning technologies in the
classroom. The other chapters in this volume are documenting successful learning
episodes using these technologies. I have argued that the move from learning to
institutional acceptance requires infrastructure, which is becoming a reality, and a
model for the successful implementation of digital curriculum, a Digital Teaching
Platform.
The DTP classroom architecture provides the support for the teacher to expand
the curriculum. If the student spends time in a virtual world, that experience must be
coordinated with the rest of the instructional process. The VR experience provides
content and motivation, but the tools of the classroom promote reflection, and
communication. In this regard, VR functions very much like simulations in various
disciplines, or manipulatives in mathematics. Integration into the tools of the cur-
riculum makes the learning possible.
While VR is inherently isolating in its current instantiations (with the potential
exception of MUVE’s), AR actually has the potential to fit quite easily into DTP
environment. To begin with, AR can easily support teams of students working
together to solve a problem—as happens in Pokémon GO. A student or team can
explore their environment: recording readings of temperature, or PH; moving about
in a museum augmented by information; or moving about town in search of an
artifact. In Augmented Realities such as these, interesting artifacts and data can be
saved in a database, or collected in a spreadsheet. If the student wants to analyze the
data collected, then tools to assist in that analysis are necessary. Finally,
6 Infrastructures for Immersive Media in the Classroom 103

presentation capabilities would allow the student or team of students to share their
results with the class. Overall, the teacher must be able to fit this experience into the
different paths through the curriculum.
I believe that the success of these technologies in other domains, combined with
the presence of the digital and pedagogical infrastructure for the classroom, sug-
gests that it is time to move from research to practice.

References

comScore (2016). Cross-Platform future in focus, U.S. 2016. Downloaded May 27, 2016. http://
www.comscore.com/Insights/Presentations-and-Whitepapers/2016/2016-US-Cross-Platform-
Future-in-Focus.
Davis, S., Nesbitt, K., & Nalivaiko, E. (2014). A systematic review of cybersickness. In
Proceedings of the 2014 Conference on Interactive Entertainment (pp. 8:1–8:9). New York:
ACM. http://doi.org/10.1145/2677758.2677780.
Dede, C., & Richards, J. (Eds.). (2012). Digital teaching platforms: Customizing classroom
learning for each student. New York: Teachers College Press.
Goff, N. Maylahn, P., Oates, B., & Oates, R. [MDR, 2015]. State of the K-12 market 2015, Part
III: Educational technologies. Shelton, Connecticut: MDR.
IDC 2015. Smartphone Vendor Market Share. Downloaded, May 27, 2016. http://www.idc.com/
prodserv/smartphone-market-share.jsp.
ITU 2014. Final WSIS targets review: Achievements, challenges and the way forward.
International Telecommunication Union, Place des Nations, Geneva, Switzerland. Retrieved
May 29, 2016. http://www.itu.int/en/ITU-D/Statistics/Pages/publications/wsistargets2014.aspx
.
Kronqvist, A., Jokinen, J., & Rousi, R. (2016). Evaluating the authenticity of virtual environments.
Advances in Human-Computer Interaction, 2016, 3.
Lawson, B. D. (2014). Motion sickness symptomatology and origins. In Handbook of virtual
environments: Design, implementation, and applications (pp. 531–599). Boca Raton: CRC
Press.
Oculus (2016) Best practices guide. https://static.oculus.com/documentation/pdfs/intro-vr/latest/
bp.pdf.
Sinha, G. [BCC, 2016]. Virtual and augmented reality: Technologies and Global Markets.
Wellesley, MA: BCC Research.
[2016]. Digest of Education Statistics U.S. Department of Education. Retrieved February 13, 2017.

Author Biography

John Richards Ph.D., is an Instructor at the Harvard Graduate School of Education teaching
courses in Education and Entrepreneurship. He is founder and President of Consulting Services for
Education, Inc. (CS4Ed). CS4Ed works with publishers, developers, and educational organiza-
tions, as they negotiate the rapidly changing education marketplace to improve business–planning
processes, and to develop, evaluate, and refine products and services.
John was President of the JASON Foundation, and GM of Turner Learning–the educational arm
of CNN and the Turner Broadcasting System. Over the years, John has served on boards for a
variety of education groups including NECC; Cable in the Classroom; Software Information
104 J. Richards

Industry Association (SIIA), Education Market section; and the Association of Educational
Publishers (AEP). John’s projects have won him numerous awards including two Golden Lamps
and several CODIEs, as well as several EMMY nominations. He is a respected keynote speaker,
has been responsible for the publication of over 1000 educational products, and is the author/editor
of over 100 chapters and articles, and four books, including Digital Teaching Platforms, Teacher’s
College Press (with Chris Dede). He is the primary author of the Software and Information
Industry Association’s annual U.S. Educational Technology Market: Pre K-12 report.
Chapter 7
The Potentials and Trends of Virtual
Reality in Education
A Bibliometric Analysis on Top Research Studies
in the Last Two Decades

Dejian Liu, Kaushal Kumar Bhagat, Yuan Gao, Ting-Wen Chang

and Ronghuai Huang

Abstract Virtual reality has gained worldwide interest among the researchers in the
field of educational technology recently. This chapter presents an overview of virtual
reality research in education and also a bibliometric analysis was performed to evaluate
the publications on virtual reality from 1995 to 2016, based on the Thomson Reuters’s
Web of Science (WoS). A total of 975 related documents were analyzed based on their
publication patterns (documents types and languages, major journals and their publi-
cations, most prolific authors, most productive journals and their publications, and
international collaborations). Bibliometric results show that the number of article has
been increasing since 1995 exponentially. USA, UK and Chinese Taipei are the top 3
most productive countries/regions which are involved in virtual reality research in
education. The findings would help the researchers to understand current develop-
ments and barriers in applications of virtual reality in education.

Keywords Virtual reality
Education
Bibliometric analysis

7.1 Introduction

With the prosperity of technology and the sustained progress of society, there are an
increasing number of science fiction scenes starting to come in our real life. For
instance, the breakthrough of stimulation technology and the improvement of devices

D. Liu
K.K. Bhagat (&)
Y. Gao
T.-W. Chang
R. Huang

Smart Learning Institute, Beijing Normal University, Beijing, China
e-mail: [email protected]
D. Liu
e-mail: [email protected]
R. Huang
e-mail: [email protected]

© Springer Nature Singapore Pte Ltd. 2017 105

D. Liu et al. (eds.), Virtual, Augmented, and Mixed Realities in Education,
Smart Computing and Intelligence, DOI 10.1007/978-981-10-5490-7_7
106 D. Liu et al.

make individuals difficult to distinguish the virtual environment from real world. In
fact, a continuum of real-to-virtual environment (Milgram & Kishino, 1994) rather
than a dichotomy of real world or virtual environment has been taken into consider-
ation for the epistemology about the world, because of the development of science and
technology. There are both an augmented reality environment which is closer to
reality and a virtual reality which is similar to total vitality. Nevertheless, the aug-
mented reality and the virtual reality have plenty of similarities in technology char-
acteristics and practice application, the augmented reality, to some extent, could be
even considered as a subtype of virtual reality. Therefore, this chapter will discuss the
continuum environment in between based on virtual reality with a broader sense.
As one of the great progress in technology which will lead to the dramatic
changes in life style and organizational structure that humans have become
accustomed with, virtual reality (VR) takes unprecedented scenes into many fields,
especially education. Although it is just starting to scratch the surface in educational
applications, many researchers believe that VR has great potential for positive
educational outcomes (Bujak, Radu, Catrambone, Macintyre, Zheng & Golubski,
2013). VR is able to be considered as a mosaic of technology, supporting for the
creation of synthetic and stimulating a high level of interactions in both real
environment and three-dimension context (Mikropoulos & Natsis, 2011). The use
of VR in education does not only facilitate students’ motivation to participant in
learning activities but also promotes their ability of exploration and state their own
points of view (Dalgarno & Lee, 2010). Pantelidis (1993) pointed that the main
reasons for exploiting VR in the classrooms are the appeal for students to learn
actively, high level of interaction, and individualism learning style. A majority of
researchers and teachers have taken VR as a promising tool for improving the
effects of education and promoting the learning outcomes overall.
Nowadays, plenty of educational products based on VR technology such as
software and applications emerge in an endless stream, making researchers and
educators overwhelmed. In order to make VR more effective on education, it is very
essential to figure out some important questions related to VR firstly, such as the
definition and characteristics of VR, its related learning theories, and its current
application status in education.

7.1.1 Evolution of VR

Sutherland (1965), the father of computer graphics, put forward the basic thoughts
and classical description of VR at first. He then created the first Helmet-Mounted
Displays (HMD) and head tracking system in 1968, which was the embryonic stage
of VR. In 1980s, some typical VR systems began to appear gradually. For example,
the VIEW system developed by NASA’s Virtual Planetary Laboratory included the
devices like data gloves and head trackers. Foley (1987) comprehensively discussed
the concept, interface hardware, human-computer interactive interface, application,
and development and perspectives of VR in his article of “Interfaces for Advanced
7 The Potentials and Trends of Virtual Reality in Education 107

Computing”. Based on the previous studies, Lanier (1989), for the first time, pro-
posed the term of Virtual Reality and developed the VR technology to be relevant
products, promoting the development and application of VR. Since 1990s, with the
continuous development of technology, the simulation and emulation environments
based on VR extended to border areas from the pure research in laboratory,
including military, science, engineering, education and training, medicine, business,
art, entertainment, and other fields. With the industrialization and commercializa-
tion of VR technology in past two decades, 2016 was considered as the Year of VR
technology because many giant companies shift sights to the development and
application of VR, pushing VR into a new stage of comprehensive development.
Although VR has emerged for more than half a century, it has yet reached a
consensus on the definition in field. In general, the definition of VR can be elaborate
from two perspectives: technological perspective and psychological perspective
(Coelho, Tichon, Hine, Wallis, & Riva, 2006). From the technological perspective,
VR is a collection of diverse technologies with interactive means (Coelho et al.,
2006). Specifically, VR integrates a set of multiple media in a three-dimensional
environment such as audio, text, video, image, and so on. The difference of VR
from the traditional multimedia lies in its interactive characteristic (Riva,
Waterworth, & Waterworth, 2004). Therefore, VR can be defined as a
three-dimensional computer generated environment, which integrates diverse
technologies, updates in real time, and allows human interaction through various
input/output devices (Boud, Haniff, Baber, & Steiner, 1999).
From the perspective of psychology, VR was defined as a particular type of
experience instead of a technology (Coelho et al., 2006). Through connecting the
computers to one another via the Internet, several users are allowed to simultane-
ously participate in the same VR environment, engaging in a diverse set of social
interactions as well as creating different types of virtual contents (Nagy & Koles,
2014; Spence, 2008). Therefore, the interaction with the synthetic world, which
projected by VR, offers the people a feeling of immersion, which is not a tech-
nological component but a result of the interaction between man and environment
(Coelho et al., 2006). In addition, as another essential concept for people to have a
clear understanding about the psychological definition of VR, presence refers to the
user’s subjective psychological response to VR (Bowman & McMahan, 2007),
being as an experience common among different types of human experiences
independent of any technology (Coelho, et al., 2006). It is the psychological sense
of “being there” in the environment generated by VR. Users tend to behave as if
they are in the real life situation though cognitively they know they are not (Lee,
Wong, & Fung, 2010). Lombard (2000) even argued that the fundament of presence
in the immersive environment is that users fall to understand the role of technology
in his experience.
In summary, VR is a collection of diverse technology while it is more likely an
immersive experience with the sense of presence in learning.
108 D. Liu et al.

7.1.2 The Main Types and Characteristics of VR

Many researchers tried to classify the types for various VR produces emerging in
the market. Papagiannidis, Bourlakis, and Li (2008) divided VR system into two
different types: game-oriented VR such as World of Warcraft, and socially-oriented
VR such as Second Life. Game-oriented VR has precise rules and regulations,
limiting the availability of certain activities exclusively for specific characters while
socially-oriented counterparts grant their users practically unlimited freedom and
options to create their characters and to engage in virtual activities (Nagy & Koles,
2014). Based on the five-element typology of virtual communities suggested by
Porter (2004), Messinger, Stroulia, and Lyons (2008) extended the typology to a
border area, proposing a 5P model to classify different VRs. The 5P model included
(1) purpose, which refers to the content of interaction; more specific, whether there
is a specific purpose for the information or content being communicated among the
virtual community on a domain, (2) place, which refers to the location of interac-
tion. Besides determining the environment to be completely or partially virtual,
whether the users are collocated or geographically disperses also needs to be
considered, (3) platform, which refers to the design of interaction, focusing not only
on synchronous communication, asynchronous communication, or both, but also on
various platforms from Desktop VR to various VR platforms, (4) population, which
refers to the pattern of interaction, focusing on distinguishing characteristics of the
target user market besides on the size of the user group, and (5) profit, which refers
to the return on interaction, focusing on the way to obtains the economic gains
using VR from users. Some new taxonomy of VRs were also proposed in a more
specific domain. For example, in the field of education, Duncan, Miller, and Jiang
(2012) indicated a taxonomy from six aspects: (1) population, i.e. who the users are
and the disciplines; (2) educational activities, i.e. what activities the users are
performing; (3) learning theories, i.e. why the users are doing particular activities;
(4) learning environment, i.e. where the users are working; (5) supporting tech-
nologies, i.e. how the system supports the users; and (6) research areas, i.e. other
cases of learning specific research.
Through the development of technology and the increasing number of appli-
cations based on VR, the main characteristics of VR has been explored and dis-
cussed by many researchers. The most widely acknowledged concepts of all came
from Burdea and Coiffet (2003), who used three “I”s to identify the main charac-
teristics of VR technology: Immersion, Interaction, and Imagination. Immersion
refers to the immersive experiences of being there in the virtual environment with
computer-generated 3D images. Immersion mainly comes from multiple repre-
sentations and sensory simulations, from visual perception to auditory, tactile,
olfactory, gustatory, and motion perception. Baños et al. (2004) indicated that VR
are immersive 3D environments which are able to induce a stronger sensation of
presence. Immersive experience may also be induced by the other characteristic of
VR—Interaction, which means that VR system can detect users’ input signals and
make response immediately. People could use some sensor equipment to interact
7 The Potentials and Trends of Virtual Reality in Education 109

with objects in virtual environments with more natural ways as in the real world, for
instance, to “control” virtual objects directly by hand, and to receive force and
tactile feedback from them. Dawley and Dede (2014) pointed that VR offers
communication options for the users, who in virtual worlds can engage in real-time
and synchronous interactions (Nagy & Koles, 2014). Subsequent researchers fur-
ther extended the definition of Immersion from users’ interaction with the envi-
ronment and the objects within it to the interactions between different users.
Therefore, virtual environments usually include social elements like “avatar” and
“messaging services”. Avatars are the user’s representation, which serve as visual
interactive chat tool for users to communicate with one another (Hew & Cheung,
2010). Through avatars users can make real for their engagement with a virtual
world (Biocca, Harms, & Burgoon, 2003; Taylor, 2002). Imagination contributes to
the last main characteristic of VR, which refers to a brand new approach and means
provided by VR technology for people to learn the world and imagine the things
that do not exist in the real world, increasing perceptual and rational knowledge,
deepening understanding of concepts, and triggering new associations. However,
currently, few empirical researches could verify whether VR technology can
facilitate users’ creative thinking and their creative activities. The specialized pre-
sentation of abstraction symbols by VR technology, on the other hand, may limit
learners’ imagination, making the mental models constructed in users’ minds more
homogenized and hindering their creativity in the context of open-ended questions.
For instance, in a creative writing course, students are required to describe the
future cities. If students have roamed in a virtual future city, their descriptions
would be quite similar to the objects and scenes that they have just seen in the
virtual city.
In conclusion, immersion and interaction are considered as the most important
and the most widely accepted main characteristics of VR. Immersion focus on
verisimilitude and multiple simulations of the environment, while the latter focus on
the natural interaction in VR environment and interpersonal interaction between
different users.

7.1.3 VR in Education

VR has been considered as one of the most potential and promising tools to pro-
mote learning outcomes as mentioned above, there is a need to know how to use
VR in education and have a quite look about the applications that has been
exploited in classrooms with positive effects.

7.1.3.1 VR Related Learning Theories

The prerequisite for an effective educational application based on VR is its peda-

gogical approach and the learning theory that follows in order to fulfil the
110 D. Liu et al.

educational goals and reach the desirable learning outcomes (Mikropoulos &
Natsis, 2011). However, the absence of learning theories is common in designing
and developing VR products used for education, neither the rational of design nor
the user experience being considered. In fact, one of the significant challenges to
develop and to use VR in education is understanding the pedagogical and learning
theories that should inform the design and use of these VR systems (Fowler, 2015).
First, the key theoretical basis for applying VR on education is constructivism. The
constructivism suggests to take students as center in learning and teaching, not only
asking students to be the active body of information processing and meaning
construction, but also requiring teachers to be the guide rather than the instructor of
learning (Cunningham & Duffy, 1996). The contexts, activities, and social inter-
actions in the learning environment with constructivism keep challenging the
learners’ experience stored in their minds, promoting the construction of new
knowledge. A series of instructional strategies extended from constructivism, such
as situated learning, experiential learning, and collaborative learning, could be
applied into the teaching and learning in VR environments since they have the
similar features with VR.
A second learning theory related to educational VR is autonomous learning (also
known as self-directed learning or self-regulated learning), which refers to a situ-
ation where learners set their learning goals, select their learning methods, minor
their learning progress, and assess their learning outcomes when acquiring
knowledge (Zimmerman, 1994). In autonomous learning, the process of students’
autonomic exploring for knowledge construction is more important, teachers
playing a guiding role. Therefore, students should use the feedbacks from teachers
or environment to understand learning targets and acquire the ability of problems
solving. VR technology provides resources necessary for autonomous learning,
allowing students to select suitable learning environment based on their learning
requirements, to take an unlimited number of repetition and practice, and to check
learning outcomes by receiving feedback from environment. However, VR learning
environment has a higher requirement for students’ self-control ability. Teachers are
quite hard to minor students’ all learning behaviors in VR environment compared to
the face-to-face teaching and observation currently, especially for the students
wearing immersive output devices like head-mounted displays.
In this chapter, cognitive load theory (CLT) contributes to the last theory related
to using VR in education. CLT is a learning and instruction theory established to
coordinate instructional procedures with human cognitive architecture, with a
limited working memory being as the center (Sweller, 2003, 2004). Cognitive load
refers to the entire load imposed on working memory during human mental
activities, such as problem solving, thinking, reasoning, and so on. Different types
of cognitive load distinguished by CLT are associated with different instructional
designs and various cognitive load effects. When the amount of mental load
exceeds the capacity of working memory, overload will happen and mental pro-
cessing activities will be interrupted. The focus of cognitive load theory, therefore,
is on recognizing the role of working memory in cognitive process and making sure
that the cognitive load is to be controlled within the capacity of working memory.
7 The Potentials and Trends of Virtual Reality in Education 111

By multimedia modes, VR technology creates the learning environments with

multiple information delivered by different sensory modalities, such as sound,
images, texts, tactile cues, and even the simultaneous combinations of multiple
information. On the one hand, VR creates a highly realistic world, helping learners
to have an immersive learning experience; on the other hand, however, multiple
modalities of information and rich stimulation may lead to the working memory
overload in unit time and thus influence learning outcomes. Moreover, the
impropriate environmental settings and learning scripts may hinder students from
devoting their limited cognitive resources into the activities related to real learning
objectives. For example, multi-sensory stimulation in VR environments may induce
split-attention effect that students only focus on a certain stimulator and thus ignore
the real learning objective. Another example related to cognitive overload in VR
environments goes to the redundancy effect which may take place when using
multiple modalities of information on the same object in order to improve students’
presence and immersive experiment (e.g. the picture and words with the same
meanings appearing at the same time). Therefore, the content construction and
material presentation of VR learning environments should take account of cognitive
theories so as to suit learners’ cognitive processing and enhance learning outcomes.

7.1.3.2 Application of VR on Education

At present, the application of VR technology in the field of education is still on the

progress of preliminary attempt, and yet brought into the conventional classrooms
on a large scale. However, because of its characteristics of immersion and inter-
action, VR has broad prospects for application in different subjects. VR can create
various virtual learning environments, particularly for the objects which are difficult
to touch or even do not exist in the real world. Based on a number of previous
studies, we divide the application of VR technology in education into four types:
observational learning, operational learning, social learning, and academic research.
In the practical application, these four types are not mutually exclusive; instead,
they can be combined to use in a same virtual learning environment.

Observational Learning

The learners’ movement and behaviors can be extended into a 3D space with VR
technology. Learners are able to freely navigate in virtual environments, and obtain
initiative feelings for the things inside from different spatial perspectives. Learners
could, therefore, have a deeper understanding of the characteristics, construction,
and relevant processing of learning targets. In this chapter, the learning activities
carried out through multiple spatial special perspectives in a 3D virtual environment
is summarized as observational learning. The virtual campus is one of the earliest
application areas of VR technology, which stimulates the real campus using the 3D
virtual technology (e.g., Sourin, Sourina, and Prasolova-Førland (2006) for the
112 D. Liu et al.

Fig. 7.1 Virtual Campus (De Lucia et al., 2009)

virtual campus of Nanyang Technological University), and includes four types of

virtual space: public space, cooperate space, classrooms, and entertainment space.
The virtual campus can help students be familiar with the campus environment as
well as university facilities before they enter into the university, increasing their
adaptation ability to university life. The virtual campus also supports synchronous
lectures, rich plugins which can be added into the system boosting the interaction
between teachers and students (see Fig. 7.1).
VR technology can not only stimulate the scenes in real world, but also tran-
scend the limits of time, space, and even human’s physical senses to retrieve the
scenes that doesn’t exist in current society or to create a new world that is fully
imagined. Using VR technology, the physical space can be zoomed and transferred
to represent micro- and macro-worlds, making the abstract concepts concretized.
VR technology, therefore, can be applied widely in science education, for instance,
learners can learn about astronomy by walking on a virtual planet (Yair, Mintz, &
Litvak, 2001) or explore the structures of molecules and cells in the micro-world
with VR technology (Limniou, Roberts, & Papadopoulos, 2008). VR technology
can also be used while learning arts. For example, students can “go back” to the
reappeared historical scenes and events via the VR technology to “visit” the original
sites and observe the restored virtual antiques, through which they can learn history
by a close observation.

Operational Learning

The VR technology can offer a platform for tactile learning. Using the situated
learning environment created by VR, learners are allowed to operate the objects
with their own hands, to observe and to experience carefully. Simultaneously,
immediate feedbacks for learners’ operations and behaviors are provided, helping
the learners correct their wrong operations and understanding during learning. More
specific, by simulating the real scenes in skill training, VR technology provides the
learners opportunities to practice over and over again, facilitating the skills transfer
into the real tasks, for example, driving training and medical operation training.
7 The Potentials and Trends of Virtual Reality in Education 113

Compared with those training which may cost high or exist some dangers in the real
world, the VR technology can provide a training platform in a more convenient and
a safer way. In addition, when learning to understand some complex concepts, the
learners can observe their learning outcomes and examine their hypothesis by
operating or controlling the learning objects in the virtual world so as to adjust their
original understanding and construct the deep comprehension. For example, the
“Science Space” project developed with NASA’s fund includes a virtual platform
called “Newton World”, which can simulate the scenes where there is no any
gravity and friction. The learners can launch and catch the balls with different
masses via the “virtual hands”, through which they can predict, test, and elucidate
the physical phenomena by real operation to learn about Newton’s law of motion
and the law of conservation of energy.

Social Learning

The social learning under VR environment refers to a more extensive realm in

which the learners can study through the interaction and cooperation with others in
the simulated social scenes. 3D virtual environment provides a multi-user virtual
environment where the teachers and students are able to conduct social learning
activities, highlighting its characteristic of high interaction. Therefore, VR tech-
nology is considered to be suitable for distance education as it can overpass the
limits of physical distance. The students can participate in the real-time class
interaction and complete the group discussions through the multi-user virtual
platform. And with that, they can feel as they are in a class physically and also have
some senses of belonging to the group work. VR technology can not only facilitate
the cooperation among different learners in multi-user virtual platform, but can
establish the connections between the real learners and the virtual avatars, pro-
moting the information exchange and interaction in more abundant social com-
munication ways, including verbal and non-verbal modalities.

Scientific Research

Currently, the VR technology also has great importance in the scientific research for
some disciplines in addition to its application in teaching. Many academic insti-
tutions all over the world have had established the VR laboratories, especially in the
science areas. The VR technology is capable to simulate various science and
engineering environments conveniently, thus greatly decreasing the cost and risk of
conducting experiments in real laboratories. In addition, the VR technology can
stimulate or create some scenes and effects, which cannot be achieved in the real
world, through which the experiment conditions can be manipulated and controlled
flexibly. For example, in medical research, VR can visualize the inner organs and
make it accessible for the researchers to “operate” some virtual nervous tissue
(Morehead et al., 2014).
114 D. Liu et al.

7.2 Bibliometric Analysis

7.2.1 Data and Methodology

In this study data were extracted from Thomson Reuters’s Web of Science (WoS).
HisCiteTM software was used to analyse the data retrieved from the database. Many
researchers recommended WoS because it provides bibliographic data of individual
articles along with information about their cited references (Matthews,
Abdelrahman, Powell, & Lewis, 2016; Tseng, Chang, Tutwiler, Lin, & Barufaldi,
2013; Zhang et al., 2014). WoS was searched using the keyword search terms
(“virtual reality” or “immersion” or “augmented reality” or “mixed reality” or
“immersive learning environment” or “virtual embodiment” or “virtual world” or
“head mount display” or “virtual environment” or “sensory immersion” or “virtual
classroom” or “virtual training” or “immersive learning” or “augmented learning”)
and (“learning” or “education” or “training” or “simulation” or “interactive learning
environment”). In Thomson Reuters’s Web of Science (WoS), each article is
assigned to one or more subject categories. Education & Educational Research
(139), Educational Psychology (43), Special Education, (30), and Education,
Scientific Disciplines (27) are included in Education subject category.
Data retrieval was conducted within WoS for the period 1995–2016 through the
following steps (see Fig. 7.2). Firstly, the documents were searched using first
group and second group of keywords (Step 1). Secondly, all the searched results
obtained in the Step 1 were combined, and duplicates were removed (Step 2).
A total of 10,235 publications were identified. Thirdly, publications were refined by
subject categories in Education (Step 3). A total of 1228 documents were obtained.
In the next step, data obtained in the Step 3 were refined by source titles (Step 4).
To improve the reliability and validity of the results, the data cleaning was con-
ducted by examining the abstracts of the records obtained (Step 5). Finally, a total
of 975 publications were collected on 2016/09/27 for bibliometric analysis (Step 6).
There has been an exponential increase in virtual reality publications in education
over the past 20 years. The growing trend line in Fig. 7.3 shows that virtual reality
has got increased attention from the researchers.
Our analysis was limited because of the choice of the language as English. This
may have overlooked some of the important publications. In addition, we did not
include publications referred such as proceeding papers, dissertations, editorial,
book chapters, etc.
7 The Potentials and Trends of Virtual Reality in Education 115

Fig. 7.2 The data collection and cleaning procedure

Fig. 7.3 The growth of virtual reality publications in education from 1995–2016
116 D. Liu et al.

7.2.2 Empirical Results

7.2.2.1 Document Types and Languages

There were four document types were identified among 975 publications on virtual
reality in education. The most common document type is peer-reviewed journal
articles (930), accounting for 95.3% of the total. Proceeding papers were identified
to account for 3.3%. The other documents were review (8) and editorial letter (4).
All peer-reviewed journal articles were used for the further analysis because they
are more prevalent document type and provide more information to identify
research trends. As much of 99.7% of the journal articles were published in English.
This result shows that English dominates the official international academic lan-
guage for research in virtual reality.

7.2.2.2 Major Journals and Their Publication

Articles on virtual reality were published in wide range of 75 SCI/SSCI journals.

Table 7.1 lists the top 20 most productive journals in virtual reality research in
education, along with the total number of articles published, the number of LCS and
GCS. The major journals for virtual reality research included Computers &
Education, Journal of Surgical Education, International Journal of Engineering
Education, and Educational Technology & Society. In Computers & Education,
139 articles, or 14.9% out of the 930 journal articles, were published, and received
370 LCS and 3292 GCS. The journal Journal of Surgical Education ranked second
in terms of publication numbers, with 74 published articles and 31 LCS and 585
GCS. These results indicate that Computers & Education is the most outstanding
journal in virtual reality research.

7.2.2.3 Most-Prolific Authors

A total of 2383 authors (co) produced just one article, accounting for approximately
89% of the total authors. Table 7.2 lists the top 10 most-prolific authors, each with
no less than 5 articles. The most productive author was Roshan Aggarwal from
Faculty of Medicine, Imperial college London, U.K, with 8 articles, 6 LCS and 168
GCS. Most of the top productive authors are from English- speaking countries. This
result reveals that English-speaking countries is dominating virtual reality research
field.
7 The Potentials and Trends of Virtual Reality in Education 117

Table 7.1 Top 20 most productive journals based on total number of articles published
Journal TA LCS GCS
1 Computers & Education 139 370 3292
2 Journal of Surgical Education 74 31 585
3 International Journal of Engineering Education 39 18 179
4 Educational Technology & Society 38 44 406
5 Computer Applications in Engineering Education 35 33 173
6 British Journal of Educational Technology 33 110 828
7 Anatomical Sciences Education 31 84 414
8 Interactive Learning Environments 30 23 134
9 Medical Teacher 27 44 1422
10 Academic Medicine 25 13 672
11 Journal of Computer Assisted Learning 24 34 392
12 IEEE Transactions on Learning Technologies 22 12 136
13 Australasian Journal of Educational Technology 17 8 65
14 BMC Medical Education 16 0 183
15 Journal of Educational Computing Research 16 4 68
16 Journal of Science Education And Technology 16 39 235
17 Medical Education 16 79 943
18 ETR&D-Educational Technology Research And Development 14 54 408
19 Nurse Education Today 13 2 90
20 IEEE Transactions on Education 12 15 224
Note TA Total number of articles, LCS Local citation scores, GCS Global citation scores

Table 7.2 Top 10 most-prolific authors based on total number of articles published
Author Country TA LCS GCS
1 Aggarwal R UK 8 6 168
2 Darzi A UK 8 13 342
3 Wilson TD Canada 7 23 92
4 Goktas Y Turkey 6 5 14
5 Hwang GJ Chinese Taipei 6 7 85
6 Kneebone R UK 6 15 388
7 Passig D Israel 6 3 8
8 Ahmed K UK 5 3 34
9 Bjerrum F Denmark 5 0 16
10 Dasgupta P UK 5 3 34
118 D. Liu et al.

7.2.2.4 Geographic Distribution and International Collaboration

A total of 59 countries/regions contributed to virtual reality research. However, only

19 countries/regions (32.2% of the total) published more than 10 articles, and 15
countries/regions (25.4% of the total) produced only one article. USA was the most
productive country with 316 articles, followed by U.K (100). Table 7.3 lists the top
10 most productive countries/regions in virtual reality research field. Among the top
10 prolific countries/regions, 4 countries are English-speaking countries and their
total output accounted for 540 articles, 58.06% of the total. Interestingly, Chinese
Taipei and Peoples R China are the two emerging Asian countries/regions which
listed in top 10. Figure 7.4 displays the network visualization among the
countries/regions for virtual reality research.
The institutional distribution of virtual reality research is also uneven geo-
graphically. A total of 930 articles were distributed over 903 institutions, however,
610 institutions, or 67.5% of the total institutions produced only one article.
Table 7.4 lists the top 10 research institutions involved in virtual reality research.

7.2.2.5 Highly Cited Papers

From Table 7.5, it is clear that most of the research related to VR has been con-
ducted in medical education. Applications of VR in school and university teaching
and learning is yet still need a lot of attention from the educators and stakeholders.

7.2.2.6 Most Frequently Used Keywords

A total of 2103 keywords were extracted. This includes author keywords and
keywords plus. Keyword Plus are the indexing keywords which are provided by
WoS. Table 7.6 lists the top 10 keywords which were used frequently. The top 10

Table 7.3 Top 10 most Country/Regions TA LCS GCS

productive countries/regions
based on total number of 1 USA 316 469 5884
articles published 2 UK 100 131 1935
3 Chinese Taipei 94 153 938
4 Canada 70 85 643
5 Spain 64 56 584
6 Australia 54 42 563
7 Peoples R China 38 22 279
8 Netherlands 26 3 230
9 Turkey 23 20 202
10 Greece 21 29 408
7 The Potentials and Trends of Virtual Reality in Education 119

Fig. 7.4 The network of top 20 countries/regions involved in virtual reality research

Table 7.4 Top 10 productive research institutions of virtual reality research in education
Institutions Country/Region TA LCS GCS
1 Taiwan Normal University Chinese Taipei 22 48 227
2 Taiwan University of Science & Chinese Taipei 17 55 303
Technology
3 Central University Chinese Taipei 16 23 139
4 Harvard University USA 15 46 458
5 University of Toronto Canada 14 2 107
6 Arizona State University USA 12 14 120
7 Imperial College London England 12 11 282
8 University of Western Ontario Canada 12 28 136
9 University of Illinois USA 10 7 127
10 Bar-Ilan University Israel 9 8 26

high frequency keywords were used for 541 times, accounting for 25.7% of the
total keyword frequency appearances. From Fig. 7.5, it is found that there has been
wide application of virtual reality in medical education.
120 D. Liu et al.

Table 7.5 Top 10 most cited articles

Cited references TA
Seymour NE, 2002, ANN SURG, V236, P458, 34
DOI 10.1097/01.SLA.0000028969.51489.B4
Dalgarno B, 2010, BRIT J EDUC TECHNOL, V41, P10, DOI 10.1111/j.1467-8535. 33
2009.01038.x
Barab S, 2005, ETR&D-EDUC TECH RES, V53, P86, DOI 10.1007/BF02504859 31
Dunleavy M, 2009, J SCI EDUC TECHNOL, V18, P7, DOI 10.1007/s10956-008-9119- 30
1
Nicholson DT, 2006, MED EDUC, V40, P1081, DOI 10.1111/j.1365-2929.2006.02611. 29
x
Wu HK, 2013, COMPUT EDUC, V62, P41, DOI 10.1016/j.compedu.2012.10.024 27
Garg AX, 2002, ACAD MED, V77, pS97, DOI 10.1097/00001888-200210001-00030 26
Garg AX, 2001, LANCET, V357, P363, DOI 10.1016/S0140-6736(00)03649-7 25
Dede C, 2009, SCIENCE, V323, P66, DOI 10.1126/science.1167311 23
Hew KF, 2010, BRIT J EDUC TECHNOL, V41, P33, DOI 10.1111/j.1467-8535.2008. 23
00900.x

Table 7.6 Top 10 high Keywords TA

frequency keywords
1 Virtual reality 171
2 Interactive learning environment 64
3 Augmented reality 60
3 Simulation 54
4 Education 36
5 Second life 28
6 Virtual worlds 28
7 Laparoscopy 27
8 Teaching/learning strategies 27
9 Medical education 26
10 Immersion 20

7.3 Discussions

7.3.1 Empirical Findings from Previous Studies

Overall, the hypothesis that VR technology promotes learning has been supported
by a number of previous studies. For example, in a study about using VR in biology
classes on animal cells, researchers found that students had a better understanding
about the complexity of the natural world (Minogue, Jones, Broadwell, &
Oppewall, 2006). In another study on learning molecular chemistry, VR technology
with haptics feedback contributed to the understanding of concepts, and made the
learning process more interesting and interactive (Sato, Liu, Murayama, Akahane,
& Isshiki, 2008).
7 The Potentials and Trends of Virtual Reality in Education 121

Fig. 7.5 Network visualization of 30 keywords which meet the threshold of occurrence at least 10
times

The positive effects of VR technology on learning were also found in some

studies with 3D models, which were considered to be a normal form for VR
technology. Nicholson et al. (2006) conducted an experiment on the effects of
computer-generated 3D anatomical models on learning, and found a positive
relationship between the computer-generated 3D models they used and the scores of
students (Nicholson, Chalk, Funnell, & Daniel, 2006). Similar findings were also
indicated by another study in which researchers used Construct3D to help students
learn mathematics and geometry, and found that the knowledge presented with
Construct3D was easier to learn and facilitated the learning of geometric con-
structions (Kaufmann, Schmalstieg, & Wagner, 2000).
Some studies based on Second Life, which is a popular VR platform in edu-
cation, also revealed some positive effect on learning. The virtual campus based on
Second Life was found advantageous to improving user impressions concerning
presence, awareness, communication, and comfort with the virtual environment (De
Lucia, Francese, Passero, & Tortora, 2009). Another study carried out over a year
and half observing several groups of students found that the students who learned
electronic related subjects through mixed lessons combining traditional on-site
lessons with virtual lessons on Second Life platform had higher motivations for
learning and attending learning activities (Sierra, Gutiérrez, & Garzón-Castro,
2012).
Some studies on collaborate learning based on VR technology were also proved
positive in promoting learning, because students were able to discuss with their
classmates, share their findings, and demonstrate the operations of systems to their
classmates and teachers (Chen, Yang, Shen, & Jeng, 2007). When it comes to skills
training, VR technology also plays an important role. In the training of assembling,
VR technology was found to be better than traditional 2D engineering drawing
lessons (Boud et al., 1999). One of the main reasons for this was that assembly not
122 D. Liu et al.

only had a requirement for learning sequence, but also for motor behavior. VR
participants were able to investigate assembly sequences through a number of VR
conditions. However, Boud et al. (1999) also highlighted that one of the main
problems of VR technology is the lack of haptic feedback, which plays an important
role when manipulating objects. This problem was also found when researchers
compared VR technology and AR technology for training laparoscopic skills
(Botden, Buzink, Schijven, & Jakimowicz, 2007).
Although many researchers put their attention on the advantages and potential
benefits of the use of VR technology in the field of education, only a few of them
came up with powerful proof to support their opinions with empirical experiments
and quantitative data (Hew & Cheung, 2010). Some studies even showed that there
were no significant differences on learning outcomes between students who learned
with VR technology and students who learned in a traditional way. Merwin and
Wickens (1991) compared scientific data visualization performance with 2D plan
and 3D renderings, and found that 3D perspective supported short term performance
but had no benefit for longer term retention. Minogue et al. (2006) failed to find any
positive cognitive impacts when using VR technology to study animal cells, which
however may have been due to the inaccuracy of assessment by paper and pencil.
Similar results were found in Holmes’s study (2007) in which a post hoc Tukey test
failed to find any significant difference between three groups of learners in terms of
their test scores with regard to the learning of ecological concepts. Some researchers
even suggested that the use of VR technology in education might lead to lower
learning outcomes. Aretz (1991) divided learners into two groups and required them
to perform a helicopter flight simulation in which they navigated through a virtual
world of geometric objects. One group used a track-up map that always presented
their navigation information in the same orientation in their forward field-of-view
while the other group used a fixed north-up map. After the navigation, both groups
were asked to draw a map of the environment through which they had traveled, and it
was found that the group using the fixed map showed a better retention of the
position of the geographical features. Some researchers also pointed out a weakness
for Second Life in that students tend to get distracted by the computer as it gives
them access not only to the learning environment of Second Life but to all the social
networks and the Internet in general (Sierra et al., 2012).
To sum up, although there are some studies that failed to find the promising
effects and even indicated some disadvantages of VR technology in learning, the
characters and advantages of VR technology has been accepted by the mainstream
of researchers and practitioners, the majority of studies indicating that using VR
technology might make a great difference on learning, resulting in positive effects.
Nevertheless, the current studies mostly used observations, questionnaires,
self-reports, and other simple methods to get the findings, thereby lacking experi-
mental, quantitate supports. The effects of VR technology on learning needs to be
repeatedly tested and confirmed by more empirical studies. It is hard to have a
simple and standardized answer for the question of whether VR technology defi-
nitely contributes to learning, because it may be influenced by complex factors such
7 The Potentials and Trends of Virtual Reality in Education 123

as the setting, learning contents, and manipulating disparities, which also may be
the reasons why the conclusions of some current researches are contradictory.

7.3.2 Challenges of VR Application in Education

and Future Directions

VR technology is promising in education for it meets the principles of various

pedagogical theories and methods. In recent years, some relevant research sup-
ported that VR technology can function effectively in teaching and learning.
However, there are lots of challenges that VR technology has to face when applied
in education, including the technology, its application in teaching, and the learners’
experience (see Table 7.7).

Table 7.7 Ten challenges faced by VR technology

Categories Challenges Profiles
Technology 1. Controlling device cost and To develop more portable and cheap devices
improving portability and connect VR devices with modern
communication terminals like smart phones,
which can be applied in mobile learning
2. The improvement of To improve the product’s accuracy, reaction
environmental simulations speed and presenting various feedback
information including the sense of touch,
force and smell, which can strengthen the
sense of immersion
3. The improvement of interaction To improve the experience between humans,
experience devices and environments and develop the
systemic social tools which make it easy to
communicate in real-time, boosting
cooperation among users
Application in 4. Certify system content and To certify the teaching objects that fits the
teaching teaching strategy VR technology, design adequate system
contents and explore the effective teaching
strategies and principles in VR learning
environment
5. The avoidance of cognitive Cognitive overload should be fully
overload considered when constructing a virtual
learning environment. Both the scene design
and the organization of learning materials
will avoid causing overload in cognition
6. The supervision and evaluation To stress on the tracking, supervision and
of learning effects evaluation of the learning behaviors in the
VR learning environment. Furthermore,
more strict empirical researches should be
conducted focusing on the teaching effects of
this technology
(continued)
124 D. Liu et al.

Table 7.7 (continued)

Categories Challenges Profiles
Learners’ 7. The reduction of difficulty in To provide relevant trainings for teachers
experience using technology and students about technology use and
operation. Also, the expansibility of products
should be improved and the users are
allowed to edit and regulate the content by
themselves
8. Adapting to identity To help students adapt to the identity transfer
transformation and promoting in the VR environment better and to further
identity motivate the learners’ interests in
participating in learning activities through
improving the identity of virtual avatars
9. The protection of the privacy and In the open VR platforms, the protection and
data of users security of personal information matters a
lot. The industrial standards about product
development and opening should be
regulated
Comprehension 10. The integration of technology To strengthen the compatibility of VR and
and the applicable smart learning other education technologies, VR learning
environment environments and the real learning
environments. Also, to flexibly use various
technologies in teaching to create the
optimized smart learning environment based
on teaching goals

7.3.2.1 From the Perspective of Product Technologies

Control the Devices’ Cost and Strengthen Compatibility: Generally speaking,

the cost of VR devices is relatively high at present, especially those devices which
are fully-immersing through helmets. The output devices with high accuracy like
the VR helmets are too heavy to carry for a long time. The cost of this technology
has been constantly decreasing with the development of more and more VR
products. Nevertheless, it is necessary to develop more portable and cheaper
devices in order to widely spread and apply VR technology in education.
Furthermore, VR devices should be connected with modern communication ter-
minals including smart phones and pads to apply into fields like mobile learning.
Improve Environment Simulations: The highly immersive VR technology has
a strict requirement for the simulations of its learning environment. Only by
improving the VR simulations both in scene setting and operating experience can
the students achieve deep understanding under the natural sematic state in the
near-real learning environment. Some users will feel dizzy when they are using VR
products currently, especially after a long-time use, mainly due to incompatibility
between the VR visual stimulus and the self-sense stimulus of the users. Therefore,
it is needed to further improve the accuracy and reaction speed of the VR products
and try to reduce the delay of visual stimulus. Besides, most of the current VR
systems are still only focusing on offering visual and auditory information. But it is
7 The Potentials and Trends of Virtual Reality in Education 125

only by presenting various feedback including the sense of touch, force and smell
can the students generate the sense of immersion to the different aspects of the
learning objects, improving the comprehensive simulations of VR environments.
Improve the Interaction Experience: Although interaction is another eminent
feature of VR technology, the interaction experience of the current VR learning
system still needs to be improved. On the one hand, the interaction experience
between the users and the systems needs to be enhanced from the perspective of
technologies, allowing the users to manipulate and control the VR environment and
the objects in it in a more natural way. On the other hand, the tools for social
communication between individuals in VR learning systems have a limited range,
with the traditional verbal information as the major form of interaction. Although
some non-verbal information can be presented through body movements and facial
expressions of virtual avatars, the effectiveness of these communication cannot be
compared with social interaction in a real environment. Thus, besides improving the
more natural interaction between humans and devices and environments, VR
technology needs to be expanded with more social tools for convenient real-time
communication, supporting the cooperation among various users.

7.3.2.2 From the Perspective of Teaching Application

Certify System Content and Teaching Strategy: The first issue that needs to be
considered in teaching application is what to be taught and how to teach it with VR
technology. Though VR technology has the potential of being applied in numerous
disciplines, this technology’s advantages can only be fully present and the learning
effects can only be optimized by specifying the suitable application scale and
teaching objects. This makes clear the principles of presenting and constructing VR
environments and developing and designing more abundant VR system content
matching the teaching requirement. However, there is still a lack of systematic and
mature methodology in rendering VR technology to teach. The effective teaching
contents and strategies need to be further explored in the VR environment to find
out how to present the knowledge from textbooks in the environment, how the
students study by themselves in the VR environment, and how the teachers guide
the students in the VR environment.
Avoid Cognition Overload: The amount of multi-channel information carried
in the VR environment will deliver pluralistic information to the users, which
causes an overload in cognition and this affects learning outcomes if designed
inappropriately. The students may be distracted by the numerous functions and
simulated scenes of the VR world, disturbing their attention from the objective
learning contents. The students’ sense of presence can be strengthened by the
pluralistic information, but the delivery of information can also be repetitive and
redundant, causing a waste of cognition resources. The construction of VR learning
environment is different from other stimulation environments created for enter-
tainment and games, because the ultimate aim of a VR learning environment is to
deliver knowledge and to make students devote their limited recognition resources
126 D. Liu et al.

into the activities that have direct connections with the learning objects. Therefore,
the element of cognition load should be taken seriously in the construction of VR
learning environments. The design of all scenes and the presence and organization
of learning materials should follow the cognition process of the students, avoiding
cognition overload in learning. For example, the influence of VR to the cognition
process and learning outcomes, such as the number and layout of the objects in the
environment, the organization and presence of pluralistic media (such as sound,
words, images, animations), and the level of knowledge of the learners should be
considered.
Supervise and Evaluate the Learning Effects: When applied in actuality, the
VR helmet will create a fully-immersive learning environment in which the teachers
cannot distinguish students’ learning states according to the traditional behaviors
and reactions (like the expressions of distraction and confusion), making it difficult
for teachers to control the teaching process instantly. Thus, more auxiliary tech-
nologies should be involved in the VR system to help track, record and evaluate the
learning behaviors better, providing the feedbacks of learning effects for the stu-
dents and teachers. As for teaching research, it is a necessity to conduct experiments
with strict control and select the available effect indexes. This will contribute to
implementing empirical researches focusing on teaching effects in VR environment
with different learning objects and teaching methods. Furthermore, it is necessary to
conduct long-term longitudinal studies to examine the long-term effect of learning
outcomes in VR environments and explore whether it is possible to transfer what
students learn into the tasks in real environment.

7.3.2.3 From the Perspective of Dimension of Learners’ Experience

Reduce the Difficulty of Using Technology: As it is a new and complicated

technology, a VR system requires users to adapt to the particular ways of operation,
which may be difficult. To ensure using the VR technology fluently in classes, the
teachers should be trained about the technology to guide the students and solve the
problems of using it. Meanwhile, the students should also receive some relevant
training courses and practices before using the VR system to guarantee fluent use
and reduce the overload brought by the technology itself. In addition, considering
the poor expansiveness of most current VR products, users cannot edit and regulate
the contents according to their own needs, causing some inconvenience in practical
application.
Adapt to Identity Transformation and Promote Identity: The virtual identity
created by the students in a VR environment may have differences with the real one,
which will decrease the students’ sense of identity. The identity will affect the
students’ learning motivation in social environment. VR is a social stimulation
environment with a high interaction, especially in the multi-user virtual platforms,
in which the students will not finish their learning tasks and interact with others if
they lack the sense of identity. Therefore, in character design for virtual environ-
ments, it is necessary to consider how to help students adapt to the identification
7 The Potentials and Trends of Virtual Reality in Education 127

transformation and further stimulate their interests in participating in virtual

learning activities through promoting identity. In the future, researchers can study
the principles for the learners to use virtual avatars and the ways the avatars affect
the learning effects. For example, it can be studied whether students prefer to use
avatars which are close to the real identity or close to the ideal self. Moreover, it can
be researched how the characteristics of virtual characters affect the social inter-
action between them and the students, and how the students perceive the infor-
mation delivered from avatars.
Protection for the Privacy and Data of Users: The issue of personal privacy is
taken seriously by the educators of K-12 when applying VR technology (Dawley,
2009). The protection of the user’s privacy, especially the privacy of the
under-aged, needs thorough attention in VR platforms that are widely open.
Currently, there is no adequate industrial standards for the development and
opening of VR products and the relevant education products.
Taking the three dimensions into account, the last challenge of applying VR
technology in education is the technology integration of it, including promoting the
compatibility between VR technology and other education technologies and
between the VR environment and the actual learning environment. “VR plus
education” is a kind of education tool rather than a new education method, which
contributes to the traditional teaching mode instead of substituting traditional
classes. The current VR products lack the standards for open design, which will
cause some problems for the users who want to integrate other education tech-
nologies and teaching resources. It is worth studying how to combine VR tech-
nology and other education technologies, how to connect perfectly the VR
environment and the actual learning environment, and how to make VR contents
cover different disciplines. The aim of discussing and researching VR technology is
to better meet the teaching requirements and create an optimized smart learning
environment based on specific learning goals, where all kinds of technologies can
be flexibly exploited.

References

Aretz, A. J. (1991). The design of electronic map displays. Human Factors: The Journal of the
Human Factors and Ergonomics Society, 33(1), 85–101.
Baños, R. M., Botella, C., Alcañiz, M., Liaño, V., Guerrero, B., & Rey, B. (2004). Immersion and
emotion: Their impact on the sense of presence. Cyber Psychology & Behavior, 7(6), 734–741.
Barab, S., Thomas, M., Dodge, T., Carteaux, R., & Tuzun, H. (2005). Making learning fun: Quest
Atlantis, a game without guns. Educational Technology Research and Development, 53(1), 86–
107. doi:10.1007/bf02504859.
Biocca, F., Harms, C., & Burgoon, J. K. (2003). Toward a more robust theory and measure of
social presence: Review and suggested criteria. Presence, 12(5), 456–480.
Botden, S. M., Buzink, S. N., Schijven, M. P., & Jakimowicz, J. J. (2007). A comparison of the
promis augmented reality laparoscopic simulator versus lapsim virtual reality laparoscopic
simulator: What is the difference? World Journal of Surgery, 31(4), 764–772.
128 D. Liu et al.

Boud, A. C., Haniff, D. J., Baber, C., & Steiner, S. J. (1999). Virtual reality and augmented reality
as a training tool for assembly tasks. In Proceedings of IEEE International Conference on
Information Visualization, 1999 (pp. 32–36). IEEE.
Bowman, D. A., & McMahan, R. P. (2007). Virtual reality: How much immersion is enough?
Computer, 40(7), 36–43.
Bujak, K. R., Radu, I., Catrambone, R., Macintyre, B., Zheng, R., & Golubski, G. (2013).
A psychological perspective on augmented reality in the mathematics classroom. Computers &
Education, 68, 536–544.
Burdea, G. C., & Coiffet, P. (2003). Virtual reality technology (Vol. 1). Wiley.
Chen, C. H., Yang, J. C., Shen, S., & Jeng, M. C. (2007). A desktop virtual reality earth motion
system in astronomy education. Educational Technology & Society, 10(3), 289–304.
Coelho, C., Tichon, J. G., Hine, T. J., Wallis, G. M., & Riva, G. (2006). Media presence and inner
presence: The sense of presence in virtual reality technologies (pp. 25–45). From
Communication Topresence: Cognition, emotions and culture towards the ultimate commu-
nicative experience.
Cunningham, D., & Duffy, T. (1996). Constructivism: Implications for the design and delivery of
instruction. In Handbook of research for educational communications and technology
(pp. 170–198).
Dalgarno, B., & Lee, M. J. (2010). What are the learning affordances of 3-D virtual environments?
British Journal of Educational Technology, 41(1), 10–32.
Dawley, L. (2009). Social network knowledge construction: Emerging virtual world pedagogy. On
the Horizon, 17(2), 109–121.
Dawley, L., & Dede, C. (2014). Situated learning in virtual worlds and immersive simulations. In
Handbook of research on educational communications and technology (pp. 723–734).
New York: Springer.
De Lucia, A., Francese, R., Passero, I., & Tortora, G. (2009). Development and evaluation of a
virtual campus on Second Life: The case of Second DMI. Computers & Education, 52(1), 220–
233.
Duncan, I., Miller, A., & Jiang, S. (2012). A taxonomy of virtual worlds usage in education.
British Journal of Educational Technology, 43(6), 949–964.
Fowler, C. (2015). Virtual reality and learning: Where is the pedagogy? British Journal of
Educational Technology, 46(2), 412–422.
Foley, J. D. (1987). Interfaces for advanced computing. Scientific American, 257(4), 126–135.
Hew, K. F., & Cheung, W. S. (2010). Use of three-dimensional (3-D) immersive virtual worlds in
K-12 and higher education settings: A review of the research. British Journal of Educational
Technology, 41(1), 33–55.
Holmes, J. (2007). Designing agents to support learning by explaining. Computers & Education,
48(4), 523–547.
Kaufmann, H., Schmalstieg, D., & Wagner, M. (2000). Construct3D: A virtual reality application for
mathematics and geometry education. Education and Information Technologies, 5(4), 263–276.
Lanier, J. (1989). Virtual reality: A status report (pp. 272–279). Cyberarts: Exploring art and
technology.
Lee, E. A. L., Wong, K. W., & Fung, C. C. (2010). How does desktop virtual reality enhance
learning outcomes? A structural equation modeling approach. Computers & Education, 55(4),
1424–1442.
Limniou, M., Roberts, D., & Papadopoulos, N. (2008). Full immersive virtual environment
CAVE TM in chemistry education. Computers & Education, 51(2), 584–593.
Lombard, M. (2000). Resources for the study of presence: Presence explication. Retrieved
September, 3, 2000.
Matthews, A. H., Abdelrahman, T., Powell, A. G., & Lewis, W. G. (2016). Surgical
EDUCATION’S 100 most cited articles: A bibliometric analysis. Journal of Surgical
Education, 73(5), 919–929. doi:10.1016/j.jsurg.2016.05.011.
Merwin, D. H., & Wickens, C. D. (1991, September). 2-D vs. 3-D display for multidimensional
data visualization: The relationship between task integrality and display proximity. In
7 The Potentials and Trends of Virtual Reality in Education 129

Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 35, No. 5,
pp. 388–392). SAGE Publications.
Messinger, P. R., Stroulia, E., & Lyons, K. (2008). A typology of virtual worlds: Historical
overview and future directions. Journal For Virtual Worlds Research, 1(1).
Mikropoulos, T. A., & Natsis, A. (2011). Educational virtual environments: A ten-year review of
empirical research (1999–2009). Computers & Education, 56(3), 769–780.
Milgram, P., & Kishino, F. (1994). A taxonomy of mixed reality visual displays. IEICE
Transactions on Information and Systems, 77(12), 1321–1329.
Minogue, J., Jones, M. G., Broadwell, B., & Oppewall, T. (2006). The impact of haptic
augmentation on middle school students’ conceptions of the animal cell. Virtual Reality, 10(3–
4), 293–305.
Morehead, M., Jones, Q., Blatt, J., Holcomb, P., Schultz, J., DeFanti, T., … & Spirou, G. A.
(2014, March). Poster: BrainTrek-An immersive environment for investigating neuronal tissue.
In 2014 IEEE Symposium on 3D User Interfaces (3DUI), (pp. 157–158). IEEE.
Nagy, P., & Koles, B. (2014). The digital transformation of human identity towards a conceptual
model of virtual identity in virtual worlds. Convergence: The International Journal of
Research into New Media Technologies, 1354856514531532.
Nicholson, D. T., Chalk, C., Funnell, W. R. J., & Daniel, S. J. (2006). Can virtual reality improve
anatomy education? A randomised controlled study of a computer-generated three-dimensional
anatomical ear model. Medical Education, 40(11), 1081–1087.
Pantelidis, V. S. (1993). Virtual reality in the classroom. Educational Technology, 33(4), 23–27.
Papagiannidis, S., Bourlakis, M., & Li, F. (2008). Making real money in virtual worlds:
MMORPGs and emerging business opportunities, challenges and ethical implications in
metaverses. Technological Forecasting and Social Change, 75(5), 610–622.
Porter, C. E. (2004). A typology of virtual communities. Journal of Computer-Mediated
Communication, 1(3).
Riva, G., Waterworth, J. A., & Waterworth, E. L. (2004). The layers of presence: A bio-cultural
approach to understanding presence in natural and mediated environments. CyberPsychology
& Behavior, 7(4), 402–416.
Sato, M., Liu, X., Murayama, J., Akahane, K., & Isshiki, M. (2008). A haptic virtual environment
for molecular chemistry education. In Transactions on edutainment I (pp. 28–39). Berlin:
Springer.
Seymour, N. E., Gallagher, A. G., Roman, S. A., O’Brien, M. K., Bansal, V. K., Andersen, D. K.,
et al. (2002). Virtual reality training improves operating room performance: Results of a
randomized, double-blinded study. Annals of Surgery, 236(4), 458–463. doi:10.1097/01.sla.
0000028969.51489.b4.
Sierra, L. M. B., Gutiérrez, R. S., & Garzón-Castro, C. L. (2012). Second Life as a support element
for learning electronic related subjects: A real case. Computers & Education, 58(1), 291–302.
Sourin, A., Sourina, O., & Prasolova-Førland, E. (2006). Cyber-learning in cyberworlds. Journal
of Cases on Information Technology (JCIT), 8(4), 55–70.
Spence, J. (2008). Demographics of virtual worlds. Journal for Virtual Worlds Research, 1(2).
Sutherland, I. E. (1965). The ultimate display. Multimedia: From Wagner to virtual reality.
Sweller, J. (2003). Cognitive Load Theory: A special issue of educational psychologist. LEA.
Sweller, J. (2004). Instructional design consequences of an analogy between evolution by natural
selection and human cognitive architecture. Instructional Science, 32(1–2), 9–31.
Taylor, T. L. (2002). Living digitally: Embodiment in virtual worlds. In The social life of Avatars
(pp. 40–62). London: Springer.
Tseng, Y.-H., Chang, C.-Y., Tutwiler, M. S., Lin, M.-C., & Barufaldi, J. P. (2013).
A scientometric analysis of the effectiveness of Taiwan’s educational research projects.
Scientometrics, 95(3), 1141–1166. doi:10.1007/s11192-013-0966-z.
Yair, Y., Mintz, R., & Litvak, S. (2001). 3D-virtual reality in science education: An implication for
astronomy teaching. Journal of Computers in Mathematics and Science Teaching, 20(3), 293–306.
130 D. Liu et al.

Zhang, B., Liu, Y., Tian, C., Wang, Z., Cheng, M., Chen, N., et al. (2014). A bibliometric analysis
of research on upflow anaerobic sludge blanket (UASB) from 1983 to 2012. Scientometrics,
100(1), 189–202. doi:10.1007/s11192-013-1189-z.
Zimmerman, B. J. (1994). Dimensions of academic self-regulation: A conceptual framework for
education. Self-Regulation of Learning and Performance: Issues and Educational Applications,
1, 21–33.
 Part II
Case Studies of Immersive Learning
Chapter 8
Virtual Reality as an Immersive Medium
for Authentic Simulations
The Case of EcoMUVE

Chris Dede, Tina A. Grotzer, Amy Kamarainen and Shari J. Metcalf

Abstract This chapter describes a design strategy for blending virtual reality
(VR) with an immersive multi-user virtual environment (MUVE) curriculum
developed by the EcoLearn design team at Harvard University for middle school
students to learn ecosystems science. The EcoMUVE Pond middle grades cur-
riculum focuses on the potential of immersive authentic simulations for teaching
ecosystems science concepts, scientific inquiry (collaborative and individual), and
complex causality. The curriculum is inquiry-based; students investigate research
questions by exploring the virtual ecosystem and collecting data from a variety of
sources over time, assuming roles as ecosystems scientists. The implications of
blending in VR for EcoMUVE’s technical characteristics, user-interface, learning
objectives, and classroom implementation are discussed. Then, research questions
for comparisons between the VR version and the “Classic” version are described.
The chapter concludes with generalizable design heuristics for blending
MUVE-based curricula with head-mounted display immersion.

Keywords Immersive authentic simulation Ecosystems science



Scientific inquiry Complex causality Multi-user virtual environment
Virtual reality

C. Dede (&)
T.A. Grotzer
A. Kamarainen
S.J. Metcalf

Harvard Graduate School of Education, Harvard University, 13 Appian Way, Cambridge,
MA 02138, USA
e-mail: [email protected]
URL: https://www.gse.harvard.edu/faculty/christopher-dede
T.A. Grotzer
e-mail: [email protected]
A. Kamarainen
e-mail: [email protected]
S.J. Metcalf
e-mail: [email protected]

© Springer Nature Singapore Pte Ltd. 2017 133

D. Liu et al. (eds.), Virtual, Augmented, and Mixed Realities in Education,
Smart Computing and Intelligence, DOI 10.1007/978-981-10-5490-7_8
134 C. Dede et al.

8.1 Introduction

We live at a time of rapid advances in both the capabilities and the cost of virtual
reality (VR), multi-user virtual environments (MUVEs), and various forms of
mixed reality (e.g., augmented reality (AR), tangible interfaces). These new media
potentially offer extraordinary opportunities for enhancing motivation and learning
across a range of subject areas, student developmental levels, and educational
settings.
An attribute that distinguishes immersive from real world learning and poten-
tially makes it more powerful, is the ability to create interactions and activities in
mediated experience that are not possible in the real world. These include, for
example, teleporting within a virtual environment, enabling a distant person to see a
real-time image of your local environment, or interacting with a (simulated)
chemical spill in a busy public setting. However, while immersion is intrinsically
helpful for motivation and learning in some ways, it is not necessarily useful in
others. For example, in mastering complex knowledge and sophisticated skills,
students learn well in a Plan, Act, Reflect cycle (Lewin, 1946; Kieran & Saldanha,
2008): first they prepare for an experience that involves doing something they want
to master; then they attempt that performance; and finally they assess what went
well, what did not, why, and what they need to learn in order to execute a more
successful repetition of the cycle. Immersion is helpful for the Act part of the cycle
but, unless used carefully, can interfere with the Plan and the Reflect parts of the
cycle. This—and numerous other factors—make effective instructional design for
immersive learning complex.
To maximize the power of immersive learning, it’s important not to present
isolated moments in which VR, MUVEs, and AR merely provide short-term
engagement or fragmentary insight. Instead, extended experiences that immerse
students in rich contexts with strong narratives, authentic practices, and links to real
world outcomes are what truly unleash the transformational power of immersion.
For example, while showing a 3-D model of a human heart illustrating blood flow is
useful, immersing students in a virtual setting where they are applying knowledge
of the heart to save the lives of computer-based agents is much more motivating, as
well as effective in fostering a wide range of complex knowledge and sophisticated
skills.
That said, as discussed in Richards’ chapter, because of issues like simulator
sickness, isolation from the real world, and technical limits on shared VR experi-
ences, implementing an extended learning experience in VR only is challenging and
often sub-optimal. This chapter discusses ways to blend VR with
MUVE/monitor-based interfaces in ways that create a transmedia narrative
(Warren, Wakefield, & Mills, 2013), in which each modality contributes its own
rich capabilities to overall learning. To derive and illustrate generic design
8 Virtual Reality as an Immersive Medium for Authentic Simulations 135

heuristics for VR as a complement to other forms of immersion, we present a case

study of EcoMUVE, a MUVE-based curriculum for ecosystems science education,
describing the alterations needed to produce a version of this curriculum that blends
in VR.

8.2 EcoMUVE: Immersive Authentic Simulations

for Learning Ecosystems Science

EcoLearn is an educational research group at the Harvard Graduate School of

Education that explores the use of emerging immersive technologies to support
learning about the process of scientific inquiry and the complex causal dynamics of
ecosystems. Thus far, EcoLearn has developed a completed curriculum,
EcoMUVE, and is currently designing and studying EcoXPT, an extension of
EcoMUVE that adds experimentation that is authentic to ecosystems science, and
EcoMOD, a third grade curriculum that infuses computational modeling into
ecosystems science education. Details about EcoLearn overall and about our
completed augmented reality curriculum (EcoMOBILE) that complements our
MUVEs are available at http://ecolearn.gse.harvard.edu.
The EcoMUVE middle grades curriculum focuses on the potential of immersive
authentic simulations for teaching ecosystems science concepts, scientific inquiry
(collaborative and individual), and complex causality (Grotzer, Kamarainen,
Tutwiler, Metcalf, & Dede, 2013; Metcalf, Kamarainen, Tutwiler, Grotzer, & Dede,
2011). The curriculum has two MUVE-based modules, which center on pond and
forest virtual ecosystems. Each module consists of ten 45-min lessons and repre-
sents an ecological scenario involving complex causality. The curriculum is
inquiry-based; students investigate research questions by exploring the virtual
ecosystem and collecting data from a variety of sources over time, assuming roles
as ecosystems scientists (Fig. 8.1).
Overall, EcoMUVE enables internship-like experiences in immersive simulated
ecosystems that support authentic scientific practices, including collaborative
inquiry. This chapter discusses the advantages and potential problems of comple-
menting this MUVE-based learning experience with VR. The analysis sketches
possible modifications involved in reconceptualizing the curriculum’s technical
characteristics, user interface, learning experiences, and classroom implementation
for this type of blended immersion. (This redesign description focuses on the
EcoMUVE Pond curricular module; the Forest module is similar in its technical
aspects, curricular approach, and the types of redesign possible.) Then, research
questions for comparisons between the VR version and the “Classic” version are
described. The chapter concludes with generalizable design heuristics for blending
MUVE-based curricula with head-mounted display (HMD) immersion.
136 C. Dede et al.

Fig. 8.1 A student uses an avatar to explore a virtual ecosystem

8.2.1 The Learning Objectives of EcoMUVE Pond

EcoMUVE Pond represents a pond and its surroundings, including a nearby golf
course and housing development. Students visit the pond over a number of virtual
days, using their avatars to explore, see realistic organisms in their natural habitats,
chat with other members of their research team, and collect water, weather, and
population data. This monitor/avatar-based immersion can occur from either a
first-person or third-person perspective.
During a virtual summer, changes occur within the pond. These changes are
subtle, but detectable both visually and through measurements of water variables.
Eventually, an easily discernable “event” occurs—the larger fish in the pond die
overnight. Known as a fishkill, it may be viewed as part of a larger process in play
or as an event that grabs one’s attention and requires explanation.
In most problem-based scenarios, students are given the problem to solve.
However, in EcoMUVE they need to discern it. This design feature was included to
help students learn the importance of focusing on processes/change over time. Over
the course of six virtual weeks, students may notice the subtle visual changes as the
water clarity shifts and fish swim closer to the surface. Eventually the fishkill
captures students’ attention.
EcoMUVE Pond simulates an extreme eutrophication scenario in which the
proximal cause of the fishkill is low dissolved oxygen concentrations in the pond
during a particularly warm and windless night, but the ultimate cause is the process
of eutrophication driven by excessive fertilizer runoff followed by algae growth and
decomposition. As they examine the virtual world, students learn about processes
that lead to an increase (e.g. photosynthesis and mixing) or decrease (respiration (in
8 Virtual Reality as an Immersive Medium for Authentic Simulations 137

particular bacterial respiration) and warming temperatures) in dissolved oxygen

concentrations in a pond. To fully understand the fishkill, students must also rec-
ognize distant drivers of change, like fertilizer that was applied in the watershed and
ran off into the pond during a recent storm.
The following affordances for recognizing change over time and
process-oriented versus event-based causality are built into EcoMUVE.
A time-traveling calendar tool allows students to go back and forth in time to see
the subtle changes that they missed. They may talk to residents or collect data and
clues. Data they’ve collected is stored in a table so that they can analyze temporal
trends using a built-in graphing function. They can use digital tools that support
understanding the flows of matter and of energy through the ecosystem.
The curriculum supports students’ learning about typical ranges for water quality
measurements, and helps students see how ecological processes and interactions
result in variation in the measurements over time. Students conclude the unit by
constructing concept maps, which represent their understanding of the ecosystem
components related to the fishkill and are asked to support the relationships they’ve
identified with evidence they’ve collected.
Our research measures include students’ motivation to do science and feelings of
self-efficacy about their abilities as scientists, students’ understanding ecosystems
science content and practices, students’ abilities to perform individual and collab-
orative scientific inquiry, and students’ understanding of complex causality (Chen,
Metcalf, Tutwiler, 2014; Grotzer, Kamarainen, Tutwiler, Metcalf, & Dede, 2013;
Kamarainen, Metcalf, Grotzer, & Dede 2015; Metcalf et al., 2014). We also have
studied teachers’ perceptions of the practicality and effectiveness of the EcoMUVE
curriculum (Metcalf et al., 2016). We have found significant gains in all the factors
listed above. (For reasons of space, the detailed results of the numerous studies we
have conducted are referenced in our citations rather than summarized here.)

8.2.2 Blending VR into Features of the MUVE-Based Pond

Curriculum

Moving through the Pond virtual environment offers an opportunity to realize its
features (Fig. 8.2). Avatars can walk uphill to the housing development and down
along a drainage ditch where water is flowing into the pond; VR potentially pro-
vides a way to reinforce students’ perceptions of the topography, which is very
important in understanding the dynamics of a watershed.
The map icon displayed in Fig. 8.2’s upper-right-hand corner is helpful in
navigating through the overall environment. The various toolbars provide options
for collecting data and moving through various modalities in the world. Having so
many interface icons could be difficult to view in VR and would also likely
undercut a sense of presence; a VR interface that, as desired, reveals and conceals
these activity options would be necessary.
138 C. Dede et al.

Fig. 8.2 Students can collect water, weather, and population data, as well as chat with members
of their research team

Virtual agents (for instance, a park ranger, a local dog walker, and a landscaper)
offer pieces of information (Fig. 8.3). Physical artifacts, such as a bag of fertilizer,
also provide valuable data.
The interface for providing substantial amounts of information—and for dis-
playing the chat shown in Fig. 8.2—is based on large dialogue boxes, which in VR
would be distracting and possibly difficult to read. Finding ways in VR to com-
municate this information requires alternative interface modalities, such as audio
communication, as well as simultaneous storage of complex information for later
perusal. In our EcoXPT extension of EcoMUVE, for example, we use a
team-notebook that enables easily sharing information about the pond with related
comments. Gardner’s chapter provides additional illustrations of immersive com-
munication tools, and Kraemer’s chapter discusses the importance of collaboration
in learning.
Linked visual representations reinforce abstract concepts (Kamarainen, Metcalf,
Grotzer, & Dede, 2015); students can measure pond turbidity and link the mea-
surements to their experiences by seeing how murky the water looks on different
days (Fig. 8.4). Using a “submarine” tool, they can also shrink to microscope size
to see organisms within the pond that are invisible at normal scale (Fig. 8.5).
Slater’s chapter discusses the value this type of virtual embodiment provides.
8 Virtual Reality as an Immersive Medium for Authentic Simulations 139

Fig. 8.3 Talking to Manny and observing the bags of fertilizer

Fig. 8.4 Visual changes in turbidity of the pond on different days

The submarine tool is implemented so that students can travel directly up and
down in the pond, to observe organisms at various depths, and can turn in a circle at
any vantage point—but cannot navigate immersively into the water. While this
experience is currently implemented in 2-D (looking through a submarine window),
140 C. Dede et al.

Fig. 8.5 The submarine tool allows students to see and identify microscopic organisms

Fig. 8.6 The data table guides students in what to measure over time

reimplementing in VR would reduce the number of times students switch

interface-modalities.
Students measure environmental variables to collaboratively complete a Data
Table that numerically reveals changes over time (Fig. 8.6); they can then display
longitudinal graphs showing correlations among the data they have collected
(Fig. 8.7). These are easier to view and to manipulate through a monitor-based
interface, and VR offers no interpretive advantage.
8 Virtual Reality as an Immersive Medium for Authentic Simulations 141

Fig. 8.7 Graphs display correlations in how variables change over time

Fig. 8.8 The field guide provides information on organisms students find

Based on entries in the Field Guide (Fig. 8.8) from organisms they have found,
students can also construct a Food Web to show the flow of energy through the
ecosystem (Fig. 8.9). In addition, they can view an Atom Tracker feature to show
the flow of matter over time (Fig. 8.10). Both of these dynamics, not apparent
through sensory information, are important in understanding the causality of
ecosystems.
The Food Web tool is implemented outside of the virtual ecosystem, and is
easier to use without VR, since no immersion is involved. Both the Field Guide and
142 C. Dede et al.

Fig. 8.9 Students develop a food web for the pond ecosystem to show energy flows

the Atom Tracker display complex visual and alphanumeric information not easily
processed at the current image resolutions possible in VR. Since leaving VR would
disrupt the flow of activity and interpretation, the gist of this information can be
communicated via audio descriptions, as well as simultaneous storage of these
artifacts for later viewing in detail.
Students can use six interactive learning quests (Fig. 8.11), implemented outside
the virtual world, to learn more about content related to pond dynamics
(Chlorophyll a, Turbidity, pH, Nitrates and Phosphates, Dissolved Oxygen, and
Bacteria). There is no educational advantage from monitor-based viewing to porting
these to VR.
The Pond curriculum uses a “jigsaw” pedagogy; students work in teams of four
(Table 8.1) and are given roles (e.g., botanist, microscopic specialist).
Each student then performs data collection specific to his or her role in the
virtual pond ecosystem, sharing this data with teammates within the immersive
interface via tables and graphs. (As discussed later, collaboration in world is pos-
sible in the VR version only if it is implemented via a server.) Each team works
collaboratively to analyze the combined data and understand the ecosystem inter-
relationships. The curriculum culminates in each team creating an evidence-based
concept map representing their understanding of the causal relationships in
ecosystem and presenting it to the class (Fig. 8.12).
A video showing the “look and feel” of EcoMUVE is available at http://ecolearn.
gse.harvard.edu/ecoMUVE/video.php.
Overall, the curriculum offers complementary opportunities for observational
field-work and interpretive lab-work. This is an example of the rich activities
8 Virtual Reality as an Immersive Medium for Authentic Simulations 143

Fig. 8.10 An oxygen molecule describes a stage of its journey through the ecosystem

Fig. 8.11 Screenshot from interactive learning quest for chlorophyll

144 C. Dede et al.

Table 8.1 Students work in teams of four with complementary data collection roles
Naturalist Microscopic Water chemist Private investigator
specialist
Collect population Collect population Collect water Observe the weather
data for the pond data for the measurement data on different days;
organisms on microscopic on different days: collect
different days: organisms in the water temperature, measurements of air
largemouth bass, pond on different dissolved oxygen, temperature, cloud
bluegill, minnows, days: bacteria, phosphates, nitrates, cover, and wind
and great blue bluegreen algae, and turbidity, pH, and speed
herons green algae Chlorophyll a
Use the field guide Use the field guide Review your notes Talk to the
to learn about the to learn about the from the learning landscaper, golf
different fish species bluegreen algae, quests on course manager,
green algae and chlorophyll, utility worker,
bacteria turbidity, pH, ranger, birdwatcher,
nitrates and and other people
phosphates, and near the pond
dissolved oxygen
Use the graphs to Use the graphs to Use the graphs to Write down
look at the fish look at the algae look at each of the observations about
population data. population data. measurements you the pond and
How did each How did it change collected. Describe surrounding area.
population change over time. Write in words how each Take notes about
over time? Write down your ideas measurement changes you
down your ideas about why changes over time observed over time
about why
Use the graphs to Use the graphs to Write down any Use the graphs to
look at the heron look at the bacteria ideas about why the look at weather data.
population data. population data. water measurements How do air
How it change over How did it change might have changed, temperature, cloud
time? Write down over time. Write and how the cover, and wind
your ideas about down your ideas changes might relate speed change over
why about why to other things time? Write down
happening around your ideas about
the pond why
Use the atom tracker to find out what happens to the oxygen atom, the carbon atom, and the
phosphorus atom on different days
Work together to create a concept map that represents the causal relationships of the pond
ecosystem based on the whole team’s observations

Klopfer describes in his chapter. As a general design principle, VR is not necessary

and often cumbersome for data-based lab-work and related scientific briefings (e.g.,
the Field Guide, Atom Tracker, Learning Quests, and Food Web and Concept
Mapping tools), but can selectively enhance the observational field-work. Creating
a link from what is collected in the field (immersed in the virtual world) to what is
interpretable in the lab (face-to-face interactions using monitor-based resources) is
important. In-world, students could use a Notebook tool to store observations (e.g.,
8 Virtual Reality as an Immersive Medium for Authentic Simulations 145

Fig. 8.12 Team’s explanation including data and concept map

images captured of phenomena; interactions with in-world characters; information

from the data-table, Atom Tracker, and Field Guide) for later interpretation.
A teachers’ guide is available for helping students learn via the EcoMUVE
curriculum, including an overview of the ecosystems content, causal understanding
goals, pedagogical methods, technical information and a detailed day-by-day lesson
plan (Metcalf, Kamarainen, Tutwiler, Grotzer, & Dede, 2013). This would be
modified to highlight the types of learning opportunities VR makes uniquely
available.

8.2.3 Technical Characteristics for EcoMUVE Pond

EcoMUVE has two versions: server-based and stand-alone. The stand-alone version
lacks opportunities for team members to interact and does not collect logfiles, but is
useful in school settings where the EcoLearn team does not wish to provide server
access (which is a resource limited to our research sites). Student teams using the
server-based version can text-chat in-world and can easily share data among
teammates. However, this is not possible in the stand-alone version, in which
students must rely solely on classroom interactions and use more cumbersome
means to combine individually collected resources. The stand-alone version and the
server version have recently been redone in Unity 5, which supports advanced
features useful for research, such as HeatMaps that visually display the amount of
time students are spending in various locations.
146 C. Dede et al.

Some of the digital tools and supports (e.g., Food Web, Learning Quests) are
implemented in Flash and are accessed independently of EcoMUVE through a
browser interface. A concept mapping tool that enables preparing these represen-
tations digitally was developed for EcoXPT and could be added to EcoMUVE; this
allows students to easily link digital resources like data tables, graphs, and
screenshots. (The ability to take “screenshots” in VR would be useful for this
purpose.) Also, a tool is almost finished that will enable visualizing activity patterns
in the logfiles of students’ behaviors in the virtual world and in the virtual reality;
this will be helpful for students, teachers, and researchers (Dede, 2013). As dis-
cussed in Shute’s chapter, these activities can be automatically scored in real-time
to provide diagnostic feedback formative for further actions.
The teachers’ manual, professional development resources, and other support
materials are available in.pdf form. Teachers can use an online portal to create
accounts for each student’s avatar, to show groupings into teams, and to give
various permissions (e.g., chat with team members, access to parts of the world).
To complement the MUVE with VR, the Unity 5 source code will need to be
configured to work on a mobile phone, since the delivery platforms currently
affordable at scale in classrooms are devices similar to Samsung’s GearVR and
Google Cardboard. An important decision in terms of capability is whether the VR
app can function completely on the phone, or is connected to a server. The former is
easier in terms of implementation, but would have less functionality (parallel to
EcoMUVE stand-alone vs. EcoMUVE server). In either case, when a VR session
was completed, the phone or server would need to send information to the computer
on which the student is working, so that the two experiences were aligned in terms
of what has and has not been done (e.g., field guide entries, data collected).
To enable a user-interface in VR appropriate for that medium, access to interface
features such as the data collection tools would be toggled off and on by the student,
so that presence in the virtual world would be undercut by activation buttons in the
visual field. Also, all MUVE-based text boxes generated by clicking would be
replaced by audio files. For example, when clicking on a non-player character,
instead of a text-box appearing, a spoken response would be heard in the head-
phones. These would be appropriate to each situation (e.g., a young man’s voice for
Manny the landscaper, a neutral voice for the fertilizer ingredients). When acti-
vating a complex visual artifact like the Atom Tracker, the text would be read, and
the full artifact would be stored in the Notebook for later analysis outside of VR. In
contrast, brief numeric information (e.g., a turbidity reading for the pond) could be
displayed as it is in the MUVE interface.
Given the constraints of space in classrooms, movement in the VR world would
be virtual, as it is in the MUVE, rather than physical movement by students with
analogous displacements in the virtual world displayed to them. Also, to avoid
simulator sickness, a student’s time in VR should be limited.
8 Virtual Reality as an Immersive Medium for Authentic Simulations 147

8.2.4 Use-Case for EcoMUVE Pond

Below is described a typical scenario of a teacher using EcoMUVE Pond in her

classroom, alternating among the VR, MUVE, and monitor-based interfaces.
Part I: Getting to Know the Pond Ecosystem Starting in VR, students begin by
visiting the pond and becoming familiar with the species living there, the sur-
rounding environment (abiotic and biotic), and the local residents. When students
first visit the pond, they are most likely to pay attention to the things that they can
readily see: the fish, the frogs, the surrounding vegetation, and so forth (Grotzer
et al., 2014). In VR, important sensory inputs can be made more salient: the color
and turbidity of the water, weather variables like cloudiness and wind, shifts in the
color of grass, the slope of the land, and the degree to which large fish are staying
near the surface. Sounds also provide clues about variables like population density,
and VR can enhance sound through 3-D location and variations in intensity by
proximity to the source.
But soon students find out that, as ecosystem scientists, they must also attend to
things that they cannot see. Transitioning out of VR to MUVE, using the submarine
tool invites students to consider organisms that are invisible at the macroscopic
level: the many thousands of phytoplankton and zooplankton that play a critical role
in the pond ecosystem. By zooming in on these microorganisms, they will discover
a new world—so tiny that they cannot see it with their own eyes, but one that is
essential to life in and around the pond. On the second day, students will learn about
the energy relationships between these organisms by using the Food Web tool as a
monitor-based activity.
Part II: Measurement and Monitoring In the MUVE, students will be introduced
to the measurement tools for water, weather, populations, and the calendar tool so
that they can make observations and collect measurements on different days of the
simulation. Students will work in teams; within a team, each student has a different
role and must become an expert in certain aspects of measurement and monitoring
in the virtual world. Students develop their expertise by using the monitor-based
Learning Quests to gain an introduction to the major components of water quality.
Once students have completed the Learning Quests, and thus achieved a working
knowledge of water quality metrics, they may monitor these metrics during the
course of the module.
Shifting to VR, students will observe that after a period of relative consistency,
the numbers of phytoplankton begin increasing over time. Some students will
wonder why, while others will miss this clue entirely. As events unfold, this
dynamic could become part of a class discussion about why steady-state monitoring
is so important in understanding and managing ecosystem relationships (Grotzer,
Kamarainen, Tutwiler, Metcalf, & Dede, 2013).
Eventually the students should all notice that the phytoplankton is increasing; it
will become obvious as the water of the pond becomes greener and greener. What
the students are noticing is the process of eutrophication. Characterized by domino
148 C. Dede et al.

causality, the increase in plants leads to an increase in organisms that depend upon
green plants. It also leads to increased decay as the plants die. This, in turn, leads to
an increase in detrivores, such as amphipods and caddisfly larvae. Students should
notice changes in water measurements that they collect. If they are very observant,
they might also notice that the water is becoming a little murky and the temperature
in the pond is slowly rising.
However, there is something else happening that they cannot see, unless they
leave VR and use the submarine tool. The increased plant decay also leads to an
increase in bacteria, which are invisible to the naked eye—a non-obvious cause.
The bacteria that are eating the decomposing plants are thriving, but they are also
using up the dissolved oxygen in the water.
Very soon, many of organisms that thrived because of the increase in green
plants are gasping for oxygen. The most obvious of these is the fish (notice the fish
are swimming close to the surface on July 25th). With lower photosynthetic pro-
duction of oxygen and the use of oxygen by all of the bacteria, the fish are more
desperate on cloudy days and there are more fish at the surface, particularly just
before dawn. When the students visit the pond on July 28, there are dead fish along
the shore. Observant students will notice that the fish kill followed a couple of
cloudy days without wind.
Part III: Figuring Out What is Going On Using monitor-based tools, when
students have collected all of the data, they will start to notice patterns in the graphs
of the data over time. Students may notice changes in phosphates, nitrates,
chlorophyll a, oxygen, turbidity, weather variables, and population sizes. These
changes are related to one another and many of these changes give hints about what
caused the fishkill.
In order to put the pieces of the puzzle together, students are asked to examine
what variables have changed over time. In efforts to explain why these changes are
happening in the pond, the students will draw on observations and information they
have gathered elsewhere in the environment. The EcoMUVE environment provides
an open forum in which students can use scientific inquiry to pursue their own path
of discovery. Below we provide a few potential paths by which students could
uncover important parts of the fish-kill story.
In VR, as the students visit the pond on different days, most notice dramatic
changes in the color of the pond. By talking to one of the people in the world, the
students find out that the pond is not usually so green. Students may wonder what is
causing the pond to be so green. By using the water measurement tools and the
submarine, students will learn the connection between the green color and the tiny
floating plants (algae) in the pond. They realize that the pond is so green because
the algae population has increased.
Using the data tools, students may also notice that the level of the phosphates
changed quite a bit over time. As students try to understand why the phosphate
concentration changes over time, they may notice that the fertilizer that is being
used in the watershed contains phosphate. During their first visit, they see the
landscaper preparing to apply a fresh load of fertilizer to the lawns near the pond.
8 Virtual Reality as an Immersive Medium for Authentic Simulations 149

Students who visit the pond on July 6th will notice that there is plenty of rain falling
near the pond.
Transitioning out of VR, then reading the information on the fertilizer bag, and
gathering information from the Atom Tracker tool, students will learn that phos-
phates that have not been used by plants can be washed off of the ground when it
rains. The Atom Tracker tool will also help students recognize the critical role that
phosphate plays in supporting the growth of plant populations. Thus, students will
discover that phosphates likely had something to do with why the algae population
increased.
Using the population tool, or by observing organisms in the submarine, students
may notice a dramatic change in the number of bacteria that are in the pond. The
bacteria population increases many-fold just days before the dead fish are found.
Because many students recognize that bacteria can cause diseases, some students
may think that the bacteria caused the fish kill. When students use the Field Guide
to learn more about bacteria they will find out that many bacteria that live in ponds
are decomposers. Students may begin to think of the increase in bacteria as an effect
rather than a cause. Students will learn more about the role of bacteria through the
Atom Tracker and field guide and find out that bacteria thrive on dead plants and
animals. Students will also discover that bacteria use a lot of oxygen during the
process of decomposition. With this new information, the decline in oxygen on July
25th begins to make sense.
In VR, students may also notice that not all the fish died between July 25th and
July 28th. On July 28th, students can still find fathead minnows in the pond.
Transitioning out of VR, if students use the Field Guide to learn more about fathead
minnows, the students will find that these fish happen to be tolerant of high tem-
peratures, high turbidity, and low oxygen concentrations. This may give students a
clue about the variables that might have caused the other fish to die. The change in
dissolved oxygen that immediately precedes the fish kill may be a strong connec-
tion that draws the students’ attention. Further, the Atom Tracker tool will give the
students clues about the importance of dissolved oxygen for fish health.
Thus, in keeping with authentic science, there are multiple pathways that stu-
dents may take as they discover components that contribute to the fish kill. Rich
hints and clues are provided through each of the tools and non-player character
interactions built into the MUVE and the VR.
Part IV: Supporting Explanations with Evidence The final part of the
EcoMUVE involves students in generating multiple possible explanations and
working as a group to decide which hypothesis best explains the fish kill. The
process of exploring multiple explanations challenges students to support their
ideas with evidence; the teacher provides guidance as needed to accomplish this.
The group discussion will also help students construct an understanding of the pond
ecosystem that incorporates the complex causal relationships highlighted above.
Each group must then present their best, most-detailed explanation, along with
supporting evidence, to the rest of the class.
150 C. Dede et al.

Table 8.2 Illustrative series of daily lessons and support materials

Daily lesson Support materials
Day 1: BioBlitz: How many species can you find?
Day 2: Food Web online tool http://ecomuve.gse.harvard.edu/
foodweb/foodweb.html
Food Web Worksheet
Day 3: Explore changes in the pond over time http://ecomuve.gse.harvard.edu/
Pond_LQ.html
Learning quest worksheet
Day 4: Roles, learning quests, data collection Group worksheet
Role worksheets
Day 5: Atom tracker What is an atom handout
Atom tracker worksheets
Day 6: Ecological processes, looking at changes over time
Day 7: Representing causal relationships using concept Parachuting Cats into Borneo
maps Story
Day 8: Using evidence to support claims about causal
relationships
Day 9: Building cases—groups work together on final
concept map and prepare presentation
Day 10: Class presentations, final discussion

Below is an illustrative series of daily lessons (Table 8.2), assuming a 50 min

class period each day.
Note that many days have associated handouts and support materials.

8.3 Research Dimensions in Contrasting the “Classic”

and VR Versions of EcoMUVE

Parallel to studies documented by Gardner in his chapter, research is needed to

establish whether using HMD-based VR for some parts of the ecosystems expe-
rience (the “blended” VR version) might be more effective than the current
implementation with avatars in a MUVE (the Classic version). Doing these studies
is important, because VR does not intrinsically make every experience better in
terms of motivation and learning. In fact, indiscriminate usage of full sensory
immersion could make an educational experience worse. For example, for some
students:
• too much time in VR during a session could produce simulator sickness in some
students
• the richness and immediacy of VR could be overwhelming, undercutting a focus
on the aspects of the experience important for learning
8 Virtual Reality as an Immersive Medium for Authentic Simulations 151

• the lower visual resolution of VR could make detailed alphanumeric information

difficult to read
• the cumbersome VR interface could make it difficult to perform tasks requiring
sensorimotor control
These and other potential issues underscore that VR should be used only when it
adds value. Research studies can clarify the extent and severity of each issue.
What might VR contribute to EcoMUVE? Possible benefits are listed below:
• VR may make more apparent the topographic characteristics of the watershed,
such as the rise and fall of the landscape. This is very important in under-
standing ecosystems phenomena.
• VR may make the ecosystem experience seem more “real,” through increasing
students’ presence in the environment, enhancing transfer from the virtual set-
ting to the real world (Grotzer et al., 2014). In particular, as discussed earlier,
sensory inputs like sounds, the color and turbidity of the water, weather vari-
ables like cloudiness and wind, shifts in the color of grass, and the degree to
which large fish are staying near the surface could all be enhanced in VR.
• VR may enrich the aesthetics of the ecosystem, helping students appreciate its
beauty and richness, motivating them to preserve natural settings by being good
stewards.
• VR may increase the attraction that students feel to animals and plants in the
ecosystem, motivating students to care about these species. This is more likely
to happen if species in the virtual world are animated so that they move around,
interact with each other, and respond to user proximity.
• VR may increase collaboration with teammates in the setting by enhancing
interpersonal presence.
Research is needed to determine whether and how much these desirable out-
comes and other potential benefits of VR actually occur, as well as what types of
design strategies are needed to fully realize each enhancement. Based on the out-
come of the studies outlined above, design heuristics can be derived for blending
VR and MUVE/monitor-based experiences, drawing on the affordances of each
interface as appropriate for that particular part of the experience. Our work with
EcoMOBILE developed design heuristics for blending purely virtual environments
with real world experiences (Grotzer et al., 2014).

8.4 Design Heuristics for Complementing MUVEs

with VR

This section postulates design heuristics for converting a MUVE learning experi-
ence to a transmedia narrative incorporating VR. This list is suggestive rather than
complete and is largely based on the case presented in this chapter—so other types
152 C. Dede et al.

of monitor-based virtual worlds may require different or extended heuristics in

order to blend in VR effectively.
Relative Use of VR, MUVE, and Monitor Interfaces
• To enhance presence in VR, use artifacts in the world (e.g., opening a toolbox)
as a means to access to display activity-triggering buttons and other
non-immersive features that are constantly displayed on the edges of the
MUVE.
• To enhance presence in VR, use audio communication for complex or extensive
dialogue boxes displayed in the MUVE, storing those artifacts for later viewing
on the monitor.
• To enable high resolution viewing, display tables and graphs on the monitor
rather than immersively.
• To foster immersion and reduce unproductive shifts in equipment, develop the
narrative to minimize transitions between the VR and MUVE/monitor
interfaces.
• Given current technical limits, use MUVEs for in-world collaboration, but shift
this to VR when possible.
• To reduce issues with simulator sickness, limit time of each session in VR to
20 min.
As a general design principle, VR is not necessary and often cumbersome for
planning and reflecting activities (the P and R in the PAR cycle), but powerful for
the Action phase.
Useful Features in VR to Enhance Learning
• To preserve an experiential record for later interpretation, enable taking and
storing “screenshots.”
• To heighten presence and authenticity, use 3-D sound location and vary
intensity by proximity to source.
• To synchronize a user’s experience across devices, enable transfer of informa-
tion between VR and MUVE/monitor.
• To heighten authenticity and affective impact, animate the behaviors of entities
in the world (e.g., animals), so that they respond to the proximity of each other
and of the user.
• Given current limits on space in classrooms, make all movement in the world
virtually triggered rather than controlled by corresponding physical movements
by user.
Key considerations in these illustrative heuristics are presence, authenticity,
transfer, saliency, and identity. As discussed in earlier chapters, psychological
immersion is the mental state of being completely absorbed or engaged with
something, and virtual presence (place illusion, discussed in Slater’s chapter) a
particular form of psychological immersion, the feeling that you are at a location in
a virtual setting. As described in Jacobson’s chapter, Authenticity has dimensions of
8 Virtual Reality as an Immersive Medium for Authentic Simulations 153

context and activities (similar to the real world), as well as impact and value
(educational outcomes are close to students’ lives, satisfying real world needs).
Via rich stimuli and a 3-D point-of-view, head-mounted displays use sensory
immersion to deepen psychological immersion and presence, and some of the
heuristics above are designed to heighten this sense of presence. Further, if the
immersive experience is not fantastical, but instead designed as parallel to the real
world, VR can heighten a sense of authenticity.
Presence and authenticity are important for transfer, which is a key outcome in
learning experiences (Dede, 2009). Transfer is the application of knowledge learned
in one situation to another situation, demonstrated if instruction on a learning task
leads to improved performance on a transfer task, typically a skilled performance in
a real-world setting. A major criticism of classroom education today is the low rate
of transfer generated by conventional instruction (Schwartz, Bransford, & Sears,
2005). Even students who excel in schooling often are unable to apply what they
have learned to similar real-world contexts (Perkins, 1995). Authenticity and
presence address this challenge by making the setting in which learning takes place
similar to the real-world context for performance in work or personal life.
By themselves becoming part of phenomena, learners gain direct experiential
intuitions about how the natural world operates. Instructional design can make
those aspects of virtual environments that are useful in understanding scientific
principles salient to learners’ senses; and multisensory cues can heighten this
saliency. Adding multisensory perceptual information can aid students struggling to
understand complex scientific models (Dede, Salzman, Loftin, & Sprague, 1999).
Providing experiences that leverage human pattern recognition capabilities in
three-dimensional space, such as shifting among various frames-of-reference
(points of view), extends the perceptual nature of a visualization. In particular, by
using visualizations to make the important aspects of a complex experience salient
to learners, then fading these special supports over time, learners can perceive the
environment through the eyes of an expert (in EcoMUVE, as an ecosystem scientist
would), and then transfer this capability to real world settings.
All these factors culminate in the evolution of learners’ identities. The evolution
of an individual’s or group’s identity is an important type of educational outcome
for which immersive experiences situated in immersive media are well suited
(Dede, 2009). Reflecting on and refining an individual identity is often a significant
issue for students of all ages, and learning to evolve group and organizational
identity is a crucial skill in enabling institutional innovation and in adapting to
shifting contexts. Identity “play” through trying on various representations of the
self and the group in virtual environments provides a means for different sides of a
person or team to find common ground and the opportunity for synthesis and
evolution. Immersion is important in this process of identity exploration because
virtual identity is unfettered by physical attributes such as gender, race, and dis-
abilities. Authentic immersive simulations increase the value of participants’
explorations by providing realistic feedback on how the real world responds to
various patterns of individual and group behavior.
154 C. Dede et al.

In summary, VR is a promising medium for enhancing these and other aspects of

immersive authentic situations. However, full sensory immersion must be used
carefully to realize its full benefits. The design heuristics and research dimensions
this chapter articulates are a step towards understanding the differential power of
immersive media to deepen motivation and learning.

Acknowledgements EcoMUVE was supported by the Institute of Education Sciences, U.S.

Department of Education, R305A080514 to Chris Dede and Tina Grotzer (EcoMUVE: Advancing
Ecosystems Science via Situated Collaborative Learning in a Multi-User Virtual Environment) and
the National Science Foundation, REC-0845632 (CAREER: Learning About Complex Causality
in the Classroom) to Tina Grotzer.

References

Chen, J. A., Metcalf, S. J., & Tutwiler, M. S. (2014). Motivation and beliefs about the nature of
scientific knowledge within an immersive virtual ecosystems environment. Contemporary
Educational Psychology, 39(2), 112–123.
Dede, C. (2009). Immersive interfaces for engagement and learning. Science, 323(5910), 66–69.
Dede, C. (2013). Opportunities and challenges in embedding diagnostic assessments into
immersive interfaces. Educational Designer, 2(6), 1–22.
Dede, C., Salzman, M., Loftin, B., & Sprague, D. (1999). Multisensory immersion as a modeling
environment for learning complex scientific concepts. In W. Feurzeig & N. Roberts (Eds.),
Computer modeling and simulation in science education (pp. 282–319). New York: Springer.
Grotzer, T. A., Kamarainen, A., Tutwiler, M. S., Metcalf, S., & Dede, C. (2013). Learning to
reason about ecosystems dynamics over time: The challenges of an event-based causal focus.
BioScience, 63(4), 288–296.
Grotzer, T. A., Powell, M. K., Courter, A. K., Tutwiler, C., Metcalf, M. S., & Dede, C. (2014).
Turning transfer inside out: The affordances of virtual worlds and mobile devices in real world
contexts for teaching about causality across time and distance in ecosystems. Technology,
Knowledge, and Learning, 19(3). Available on-line, December 24, 2014. doi:10.1007/s10758-
014-9241-5.
Kamarainen, A., Metcalf, S., Grotzer, T., & Dede, C. J. (2015). Exploring ecosystems from the
inside: How immersion in a multi-user virtual environment supports epistemologically
grounded practices in ecosystem science instruction. Journal of Science Education and
Technology, 24(2), 148–167.
Kieran, C., & Saldanha, L. (2008). Designing tasks for the co-development of conceptual and
technical knowledge in CAS activity. In G. W. Blume & M. K. Heid (Eds.), Research on
technology and the teaching and learning of mathematics (Vol. 2, pp. 393–414). Cases and
perspectives Charlotte, NC: Information Age Publishing.
Lewin, K. (1946). Action research and minority problems. Journal of Social Issues, 2(4), 34–46.
Metcalf, S. J., Chen, J. A., Kamarainen, A. M., Frumin, K. M., Vickrey, T. L., Grotzer, T. A., et al.
(2014). Shifts in student motivation during usage of a multi-user virtual environment for
ecosystem science. International Journal of Virtual and Personal Learning Environments,
5(4), 1–15.
Metcalf, S. J., Kamarainen, A. M., Cooke, C. B., Turkay, S., Grotzer, T., & Dede, C. (2016).
Teacher perceptions of the practicality and effectiveness of immersive ecological simulations.
In K. Terry, & A. Cheney (Eds.), Utilizing virtual and personal Learning environments for
optimal learning (pp. 22–45). Hershey, PA: Information Science Reference. doi:10.4018/978-
1-4666-8847-6.ch002.
8 Virtual Reality as an Immersive Medium for Authentic Simulations 155

Metcalf, S. J., Kamarainen, A., Tutwiler, M. S., Grotzer, T. A., & Dede, C. J. (2011). Ecosystem
science learning via multi-user virtual environments. International Journal of Gaming and
Computer-Mediated Simulations, 3(1), 86–90.
Metcalf, S. J., Kamarainen, A., Tutwiler, M. S., Grotzer, T. A., & Dede, C. J. (2013). Teacher
perceptions of the practicality and effectiveness of immersive ecological simulations as
classroom curricula. International Journal of Virtual and Personal Learning Environments, 4
(3), 66–77.
Perkins, D. N. (1995). Smart schools. NY: Simon & Schuster.
Schwartz, D. L., Bransford, J. D., & Sears, D. (2005). Efficiency and innovation in transfer.
In J. P. Mestre (Ed.), Transfer of learning from a modern multidisciplinary perspective (pp. 1–
51). Greenwich, CT: Information Age.
Warren, S., Wakefield, J. S., & Mills, L. (2013). Learning and teaching as communicative actions:
Transmedia storytelling. In L. A. Wankel & P. Blessinger (Eds.), Increasing student
engagement and retention using multimedia technologies: Video annotation, multimedia
applications, videoconferencing and transmedia storytelling (pp. 67–94). Boston, MA:
Emerald Group Publishing Limited. doi:10.1108/S2044-9968(2013)000006F006.

Author Biographies

Chris Dede is the Timothy E. Wirth Professor in Learning Technologies at Harvard’s Graduate
School of Education (HGSE). His fields of scholarship include emerging technologies, policy, and
leadership. From 2001 to 2004, he was Chair of the HGSE department of Teaching and Learning.
In 2007, he was honored by Harvard University as an outstanding teacher, and in 2011 he was
named a Fellow of the American Educational Research Association. From 2014 to 2015, he was a
Visiting Expert at NSF, Directorate of Education and Human Resources. Chris has served as a
member of the National Academy of Sciences Committee on Foundations of Educational and
Psychological Assessment, a member of the U.S. Department of Education’s Expert Panel on
Technology, and a member of the 2010 National Educational Technology Plan Technical Working
Group. In 2013, he co-convened a NSF workshop on new technology-based models of
postsecondary learning; and in 2015 he led two NSF workshops on data-intensive research in the
sciences, engineering, and education. His edited books include: Scaling Up Success: Lessons
Learned from Technology-based Educational Improvement, Digital Teaching Platforms:
Customizing Classroom Learning for Each Student, and Teacher Learning in the Digital Age:
Online Professional Development in STEM Education.

Tina A. Grotzer is a Faculty Member at the Harvard Graduate School of Education and a
Principal Research Scientist at Project Zero. She directs the Causal Learning in a Complex World
Lab. Her research focuses on how causal reasoning interacts with complexity and on developing
supports for complex causal learning and public understanding of science. Tina received a Career
Award from the National Science Foundation (NSF) in 2009 and a Presidential Early Career
Award for Scientists and Engineers (PECASE) in 2011. She is the author of Learning Causality in
a Complex World (2012), lead author of the Causal Patterns in Science series, and Co-PI with
Chris Dede on the EcoXPT and EcoMOD projects.

Amy Kamarainen is a senior research manager and principal investigator at the Harvard Graduate
School of Education where she collaboratively manages grant-based education research projects,
most recently the EcoXPT, EcoMOBILE and EcoMOD projects. Amy is an ecosystem scientist
who holds a B.S. in Zoology from Michigan State University and a Ph.D. from the University of
Wisconsin—Madison. Her Ph.D. work focused on studying the movement and fate of pollutants in
aquatic ecosystems using environmental sensors, historical data, and models. She applies her
156 C. Dede et al.

understanding of ecosystems science and education research to the design and evaluation of
technologies that support science learning inside and outside of the classroom. Amy’s professional
interests concern the application of these technologies to creative spaces like Citizen Science,
STEM learning, and place-based education. The Ecological Society of America named Amy an
Ecology Education Scholar in 2011.

Shari J. Metcalf is the Project Director of the EcoXPT and EcoMOD projects at the Harvard
Graduate School of Education, and was Project Director of EcoMUVE. She holds a SB and SM
from MIT, and a Ph.D. from the University of Michigan, where she designed and developed
Model-It, a software tool for students building models of dynamic systems. Her prior research
focuses on educational technology projects in science, math, and sustainability education,
including research on computer-based modeling and simulation tools for middle school science
students. Her professional interest centers on learner-centered design and emerging technologies,
and the use of modeling, simulation, and immersive environments to support inquiry-based STEM
learning.
Chapter 9
Systems to Support Co-creative
Collaboration in Mixed-Reality
Environments

Michael Robert Gardner and Warren W. Sheaffer

Abstract This chapter examines the use of mixed-reality technologies for teaching
and learning, particularly for more active and collaborative learning activities. The
basis for this work was the creation of the MiRTLE platform—the Mixed Reality
Teaching and Learning Environment. We report on some of the lessons learnt from
using this platform on a range of different courses and describe how different
active/collaborative approaches were used. We also provide evidence of the effect
of these different approaches on the overall student attainment and discuss the
implications on the use of this technology. We then consider some of the techno-
logical research being done to develop these mixed reality learning spaces and the
affordances offered by this approach. Finally we reflect on the tensions between the
pedagogy and technology and consider the implications for the wider systems that
support teaching and learning and co-creative collaboration in mixed-reality
environments.

Keywords Mixed-reality Virtual worlds Blended delivery
Co-collaboration


Maker-spaces Multi-user virtual environment

M.R. Gardner (&)

School of Computer Science and Electronic Engineering, The University of Essex,
Colchester, Essex CO43SQ, UK
e-mail: [email protected]
URL: http://www.csee.essex.ac.uk/staff/mgardner/index.html
W.W. Sheaffer
Department of Mathematics and Computer Science, Saint Paul College, Saint Paul, MN
55102, USA
e-mail: [email protected]

© Springer Nature Singapore Pte Ltd. 2017 157

D. Liu et al. (eds.), Virtual, Augmented, and Mixed Realities in Education,
Smart Computing and Intelligence, DOI 10.1007/978-981-10-5490-7_9
158 M.R. Gardner and W.W. Sheaffer

9.1 Introduction

As discussed in the introductory chapter, the concepts of immersion, presence and

engagement are clearly important aspects in the effective use of virtual environ-
ments in education. In a Multi-User Virtual Environment (MUVE), properly
designing the virtual learning space to improve the immersive experience,
enhancing the visual and audio capabilities to heighten a participant’s sense of
tele-presence and developing activities to engage participants in collaborative and
active problem solving, as discussed in Kraemer’s chapter, can all contribute to an
improvement in student academic achievement within these spaces.
The notion of learning by creating new knowledge is enhanced when partici-
pants in a virtual environment collaborate to creatively solve problems that are
designed to improve their knowledge in a targeted area of study. We call this
process co-creative collaboration.
Early in our research efforts we observed issues with traditional classroom based
video-conferencing systems that supported the belief that extending the virtual
environment into the physical world of a classroom and bringing the classroom into
the virtual environment could achieve higher levels of presence within such envi-
ronments thereby improving student achievement. To address these issues our
thought was to mix video streaming and virtual reality to create a mixed reality
environment. We began a research and development program that produced a
platform that we called the Mixed Reality Teaching and Learning Environment, or
MiRTLE (Gardner & O’Driscoll, 2011).
While our empirical research supports the concepts on which MiRTLE is based
we felt that it was very important that the technology be tested in the actual
classroom in order to develop a broader assessment of the efficacy of this tech-
nology in education. We developed a cooperative research program with Saint Paul
College in Minnesota, USA who agreed to construct and operate a large-scale
implementation of MiRTLE, use it in a standardized first course in computer sci-
ence and to record their observations concerning its utility. Also, as the initial
research was based on using MiRTLE with the Health and Human Sciences
department at the University of Essex, it was felt that this extended trial at St. Paul
with the Computer Science department would also help to assess whether this
approach could be generalized across different disciplines.
When designing any teaching activity the learning objectives should be first and
foremost and should guide the use of any technology that is being used. Based on
our research at Essex, the trials at Saint Paul and other experiments we feel there are
particular challenges in designing effective learning activities using this techno-
logical approach (Gardner & Elliott, 2014). This chapter reflects on this initial work
and suggests that when applying such technology we need to put in place systems
that support not only the technologies being used, but also the wider pedagogical
and behavioral aspects effected by this platform.
We begin this chapter by highlighting some of the challenges involved in
designing effective technology mediated learning experiences. We then describe the
9 Systems to Support Co-creative Collaboration … 159

MiRTLE project and review both the qualitative and quantitative findings from the
field trials at Saint Paul College and some of the other technological research being
carried out at the University of Essex. We conclude by considering what the
implications may be for the development and use of similar Multi-User Virtual
Environments (MUVEs) particularly when used to support co-creative collaborative
learning activities.

9.2 Building Better Learning Spaces

With a plethora of technological advances at our fingertips, we have the ability to

increase access to technology and content, display content from the
internet-of-things, visualize abstract concepts within immersive environments,
interact with peers and colleagues remotely both synchronously and asyn-
chronously, and the list goes on. Part of the difficulty with the rapid advances in
technology is our ability to rapidly design, develop and research educational spaces
on two levels. First there is a technical and infrastructure level that needs to be
iteratively tested to understand the logistics behind delivering working immersive
learning environments. Secondly, there is the need to address the use of these
embedded technologies for the purpose of learning and the creation of pedagogical
situations that harness the affordances of the embedded capabilities in a way that is
meaningful for learning.
At the University of Essex the use of the laboratory classroom called the
iClassroom (Dooley et al., 2011) has allowed for design and testing of new
immersive environments to be completed in a real space and with real students,
measuring the efficacy of the technical and pedagogical solutions. This experi-
mental space affords the opportunity to refine and redefine what intelligent spaces
look like and how they can best be used to maximize positive learning outcomes.
Moving forward, as we begin to better understand the affordances and iteratively
create design guidelines, our hope is that eventually a prescriptive framework
emerges that informs both the practice of technical development and also the
deliberate incorporation of technologies into both the learning space and the ped-
agogy through which students learn. At a wider level, the intention is to have
demonstrated how the research is addressing understanding affordances, structuring
experiences, and creating constructivist, collaborative processes, in mixed-reality
smart environments. Ultimately the aim is to create more engaging, effective and
rewarding learning experience for students.
Taxonomies have been developed that describe the technical aspects of
immersive technologies (such as Milgram & Kishino, 1994), in addition to
frameworks for describing the learning affordances of virtual learning environments
(mainly multi-user virtual environments) such as developed by Dalgarno and Lee
(2010), but none have been able to sufficiently and completely capture the multiple
levels and complex interactions between them in terms that can be beneficial to
designers and researchers for accurately describing the technologies with which
160 M.R. Gardner and W.W. Sheaffer

they are working and the affordances of those technologies for learning. Continuing
this investigative journey into the design of immersive learning environments, and
being able to better define and classify new tools and research is imperative.

9.3 MiRTLE

Our first project that combined real and virtual worlds was MiRTLE (Gardner &
O’Driscoll, 2011). The objective of the MiRTLE (Mixed Reality Teaching &
Learning Environment) project was to provide an online virtual classroom to
augment live lectures. This was inspired by the observation that even if remote
students were able to watch a live lecture remotely (for example using video
conferencing or other similar technology), they often would choose to watch the
recorded session instead. The main reason for this (as determined from observations
of actual classes and interviews with teaching staff and students) was that there was
very little perceived value in their participation in the live event, as often there was
only limited means (if any) for them to interact with the people in the live class-
room. This meant that the recorded version of the event usually offered an equiv-
alent experience with the advantage that they could also choose to watch it in their
own time. MiRTLE provided a mixed reality environment for a combination of
local and remote students where both dispersed and local students are able to see
and talk with each other, in addition to the teacher. The environment was intended
to augment existing teaching practice with the ability to foster a sense of com-
munity amongst remote students. In this sense, the mixed reality environment links
the physical and virtual worlds. Using MiRTLE, the lecturer in the physical
classroom is able to deliver the class in the normal way but the classroom also
includes a large display screen that shows avatars of the remote students who are
logged into the virtual counterpart of the classroom. This scenario is relevant to any
classroom event where not all of the participants can attend in person. Thus the
lecturer will be able to see and interact with a mix of students who are present in
both the real and virtual world. In terms of the conceptual framework for VR
outlined in the Introduction (Chap. 1), MiRTLE is a Multi-User Virtual
Environment (MUVE) (Fig. 9.1).
From the initial qualitative evaluation of MiRTLE at the University of Essex,
based on a small number of live classroom experiments, valuable issues were
identified that have implications for future uses of this technology. For example, it
showed that there was the opportunity for impromptu and naturalistic social
interaction between virtual and physically present students. Also from anecdotal
evidence, it was clear that the teachers recognized the potential value of the system,
reporting that once students are logged-in and settled, the MiRTLE environment
had a minimal impact on normal patterns of teaching, and the teachers perceptions
of the learning occurring in their teaching environment. It also suggested that
MiRTLE could facilitate a breaking down of the barriers between the virtual and the
physical, supporting more spontaneous social exchanges between virtual and
9 Systems to Support Co-creative Collaboration … 161

Fig. 9.1 MiRTLE classroom

physically present students, and increase the sense of presence for the learners and
teachers involved.

9.4 Systems to Support Co-creative Collaboration

in Mixed-Reality Environments

9.4.1 The Teaching Context—The Need for Blended

Pedagogy Systems

Based on the initial research and the encouraging feedback from the use of
MiRTLE at the University of Essex, we felt that it was important to carry out a
more in-depth assessment of this approach in the classroom. We also hoped to
better understand the wider pedagogical and behavioral aspects affected by the use
of this technology. In this section we review the results of a field trial conducted
over six years involving roughly 600 students in which the MiRTLE platform was
utilized to deliver a standardized first course in computer science. During this trial
the course syllabus, learning objectives, assessments, instructor and assignments
were held constant.
During the 2008–2009 academic year the computer science department at Saint
Paul College, MN, USA, implemented a MiRTLE installation in order to begin
distance delivery of coursework.
MiRTLE was selected as the primary platform for distance learning for the
following reasons:
162 M.R. Gardner and W.W. Sheaffer

1. MiRTLE brings the classroom to the distance learner and the distance learner to
the classroom in a very engaging manner.
2. MiRTLE is a mixed realty platform that extends the virtual world that supports
it. When this virtual world is not being used for lecture activities it can be easily
utilized as a virtual world platform on which students and faculty can com-
municate and collaborate. See Gardner and O’Driscoll (2011).
3. Based on the Open Wonderland platform, MiRTLE easily supported the typical
teaching tools used in a first course in computer science.
4. With a properly configured client computer MiRTLE is easily accessible and
simple to use.
5. A MiRTLE installation can support alternative platforms such as Second Life or
OpenSim with very little effort.
6. A MiRTLE facility is not disruptive, it does not require the faculty or students to
significantly change their educational routine.
A general-purpose classroom was remodeled into a MiRTLE classroom in 2008.
The general arrangement of this classroom is shown in Fig. 9.2. Life size avatars
are projected on the rear wall of this facility and lectures are captured and dis-
tributed using an inexpensive security camera mounted on the ceiling of the
classroom. For a wider discussion on the general infrastructure for virtual and
augmented reality in the classroom see Richards (Chap. 6).
The field trials began following the construction of the MiRTLE facility.

Fig. 9.2 MiRTLE classroom layout

9 Systems to Support Co-creative Collaboration … 163

Two measurements were considered reliable indicators of student success and

data was collected on these:
1. Student achievement—measured by the improvement in the score on a stan-
dardized examination administered during the first class meeting and subse-
quently used as the final examination in the course.
2. Student retention—measured as the percentage of students attending the course
relative to the initial enrolment in the course.
The first field trial began in September 2008 and ended in May 2010 that
involved two class-groups per term for four terms (8 cohorts of 30 students) for a
total of 240 students. Data from the eight class-groups was aggregated.
Based on the aggregated data, first meeting scores on the standardized exami-
nation were around 30% and end of term scores were around 75%. For those
attending physically in the classroom the ending scores were 75.3% and the
MiRTLE attendees scored about 74.4%. This seemed to indicate that there was
essentially no difference in achievement between students learning on the MiRTLE
platform and students who attended lectures.
The overall retention of students was about 61%. Weekly attendance trends are
shown in Fig. 9.3. As can be seen, retention was slightly better over the course of
the term for the students attending via MiRTLE.
During this trial the classes began with about 15% of the total enrollment
indicating that they would be attending via MiRTLE. Over the course of the term
students who planned to attend the classes in person began to migrate to the virtual
platform. The attendance by the end of the term was typically 40% attending
virtually and 60% attending in person.
We feel the results show that MiRTLE when used in this manner provides
distance students with similar learning outcomes to those in the physical classroom.
This is considered a successful trial since MiRTLE met all of its design expecta-
tions, with students performing as well on the MiRTLE platform as in the lectures

Fig. 9.3 MiRTLE retention

164 M.R. Gardner and W.W. Sheaffer

and it appeared that as students became more comfortable with the MiRTLE
platform they migrated to it.
Due to the initial success of MiRTLE the computer science faculty elected to run
a second field trial beginning in September of 2010 utilizing the MiRTLE platform
and a learning management system to offer the course in a blended delivery format.
The idea was to move the static curriculum content to the asynchronous learning
management system and utilize weekly meetings to answer questions and conduct
lecture reviews of the material for the week. Students were also encouraged to
attend the weekly meetings remotely using the MiRTLE facility and to meet vir-
tually on it to collaborate on problems together.
The second field trial was terminated in May of 2011 due to lower student
retention and measureable decreases in learning achievement. The frequency and
nature of student complaints and interventions by the college administration also
played a role in electing to end the trial. The data gathered from this trial involved 4
class groups over two academic terms with 4 cohorts of 30 students each for a total
of 120 students. Data from the four groups was aggregated.
Based on the aggregated data the first meeting scores on the standardized
examination were again around 30%. However the final examination scores
dropped from 75% as observed in the first trial to 63%. Overall student retention
dropped from 61 to 52% of initial enrollment. Students made little or no use of
MiRTLE and did not utilize the virtual world meeting rooms available to students in
these sections as in the earlier trials with MiRTLE. The retention experience is
shown in Fig. 9.4.
This outcome was obviously worse than the previous computer science courses
that used MiRTLE for synchronous instruction. A lengthy retrospective review was
done which included students who had dropped the blended courses, participants from
earlier field trials and students and faculty who participated in a similar trial within the
mathematics department. The following are considered key findings from this:
1. There appeared to be a critical coupling between pedagogical design and the
technological platforms used in the course.

Fig. 9.4 Blended delivery

lecture review
9 Systems to Support Co-creative Collaboration … 165

2. The weekly review sessions in the blended sections were essentially useless if as
few as 10% of the attendees had not prepared for them.
3. The failed in-person sessions were the primary source of student complaints and
were very disruptive to course conduct.
4. Students abandoned the use of MiRTLE and collaboration within the virtual
environments due to their feeling of a loss of course structure.
Since MiRTLE had been successfully used in a traditional course during the first
trial the group decided to re-examine the pedagogy in light of the technological
approach being used. This led to the development of an alternate model of course
design and a modification of the instructional pedagogy. This model envisions a
course design of sequential learning modules that utilize a mixture of behaviorist
and constructivist-based pedagogies. Figure 9.5 illustrates this general design.
This pedagogical approach utilizes active/collaborative learning laboratories for
in-class instruction and divides online learning into behaviorist and constructivist
learning activities. Behaviorist activities are primarily used for instruction in the lower
order cognitive skills as identified by Bloom et al. (1956) they include knowledge,
comprehension and application. The remainder of the online coursework which
includes online discussions, online interactive laboratories and collaborative problem
solving in the MiRTLE virtual environment, follow the constructivist paradigm.
Constructivist activities are primarily used for instruction in the higher order cognitive
skills as identified by Bloom et al. (1956) that include create, evaluate and analyze.
They represent approximately 80% of all learning activities.
The computer science department began a third field trial in September 2012
which lasted through May 2014. This trial utilized the modified pedagogical model
with the option of in-person or MiRTLE attendance. The trial involved two course

Fig. 9.5 Revised pedagogical model for a module

166 M.R. Gardner and W.W. Sheaffer

groups per term for four terms (8 cohorts of 30 students) for a total 240 students.
Data from the eight groups was aggregated.
Based on the aggregated data, first meeting scores on the standardized exami-
nation were around 30% and end of term scores were around 81%, which was an
improvement over the traditional lecture led classes. For those attending physically
in the classroom the ending scores were 80.3% and the MiRTLE attendees scored
about 82%, again both higher than in the traditional delivery format. This seemed to
confirm the results of the first field trial that there is essentially no difference in
achievement between students learning on the MiRTLE platform and students who
attended lectures.
The overall retention of students was about 74%. This was a dramatic
improvement over the second trial that utilized blended delivery and also an
improvement over the first trial which used a traditional delivery format. Weekly
attendance trends are shown in Fig. 9.6. As can be seen, retention was slightly
better over the course of the term for the students attending in person rather than
with MiRTLE.
Table 9.1 summarizes the student achievement based on the increase in their raw
scores on a standard exam given at the beginning and conclusion of each course
section, and the end of term retention rates of those students who were initially
enrolled in the course.
Based on these experiments, we believe the following observations are relevant.

Fig. 9.6 Blended delivery—

active/collaborative

Table 9.1 Comparison of student achievement

Student retention and achievement
Configuration Trial period Student achievement (%) Student retention (%)
Traditional 2008–2010 45.3 60.4
MiRTLE 2008–2010 44.4 62.8
Blended—lecture 2010–2011 33 51.8
Traditional—active 2012–2014 50.3 75.8
MiRTLE—active 2012–2014 52 72.1
9 Systems to Support Co-creative Collaboration … 167

Traditional or blended coursework utilizing the MiRTLE platform for remote

students had learning outcomes similar to those who attended the course in the
physical classroom. From the anecdotal evidence collected, we believe the effec-
tiveness of MiRTLE lies in the level of presence the learner has within the virtual
classroom environment, the ease with which the participants adopt the technology
and the relative ease that learners using this platform have in interacting with other
people in the lecture hall.
A simple unifying observation can be made for the tests conducted at Saint Paul
to date. When implementing any system of technologically supported learning, care
should be taken to consider the overall course delivery mechanisms, systematically
recognizing as key system components the nature of the platform, the structure of
the pedagogy and the capacity of both students and instructors to adjust their
respective approaches to learning and instruction in the new environment.

9.4.2 Mixed-Reality Research at the University of Essex

A key finding from the Saint Paul experience was that it illustrated the clear benefits
from using a constructivist learning approach, particularly when applied to active
learning and creative problem solving, as discussed in Schneider’s chapter. This
also resonates with the technological research being carried out at the University of
Essex. In Gardner and Elliott (2014) it was reported how our research addressed the
core themes of understanding affordances, structuring experiences, and creating
constructivist, collaborative processes, in mixed-reality smart environments. This
was based on the iClassroom testbed and included the use of MiRTLE and the
BReal Lab which is described below. In this section three areas have been high-
lighted relevant to the topic of this chapter. Part one will discuss the work on
extending the capabilities of the original MiRTLE platform, the second part will
examine the development of co-creative maker spaces, and the third area will
examine how to use these spaces as a lens for the assessment of the learning
outcomes of students in mixed-reality environments.

9.4.2.1 Extending MiRTLE

In addition to the University of Essex and Saint Paul College, other institutions such
as the University of Hawaii have had success in deploying their own MiRTLE
installations. The University of Hawaii is carrying out innovative work (Schmidt,
Kevan, McKimmy, & Fabel, 2013) to extend the MiRTLE concept. For example,
their HoloDeck system allows physically present students to interact with virtual
students by using a mobile tablet based application.
The University of Essex is expanding the fairly limited instructional pedagogy
supported by the original MiRTLE platform to include more collaborative learning
activities, and incorporating a range of mixed and augmented reality technologies
168 M.R. Gardner and W.W. Sheaffer

into the MiRTLE learning space. The work also aims to enhance the collaborative
learning processes through the use of a variety of different user interfaces such as
tablets and immersive glasses to enable a greater level of collaboration and com-
munication between a range of users that include expert and novice students.
A similar approach is described in Dede et al. (Chap. 8) where a range of research
questions for comparisons between a VR and “Classic” version of an educational
simulation are presented together with some generalizable design heuristics for
blending MUVE-based curricula with head-mounted display devices.
A new platform has been developed (dubbed MiRTLE+), which enables a group
of four learners with different levels of expertise (two novices and two experts) to
collaborate on the same task at the same time. For the purposes of this initial study,
the learning task was chosen to be a commonly used card game called UNO.
The UNO card game was chosen as a learning task because of its relative simplicity
and due to the fact that card games are commonly learnt by practicing with experts
rather than reading from books or manuals.
In this scenario, the players work in pairs with the intention that the novice
player can learn the game and associated playing strategies by observing their
partners game play. The novice will also learn the game rules by communicating
with their team member (the expert). Based on this learning activity, we have
divided the group into four possible scenarios. This division is based on two factors;
the students’ location (real or virtual world) and their level of expertise (expert
(E) or novice (N)). This is used to determine whether their location (in the real or
virtual space) has an affect on the learning outcomes and their overall performance.
Figure 9.7 illustrates these different evaluation scenarios.

Fig. 9.7 MiRTLE+ evaluation scenarios

9 Systems to Support Co-creative Collaboration … 169

The iClassroom smart environment supports this interaction by using a range of

augmented and mixed reality collaboration tools. This forms the basis for quanti-
tative and qualitative evaluation studies that measure the learning effectiveness of
the new mixed-reality environment. With regard to the evaluation techniques, the
evaluation of the students’ sense of presence includes the use of a subjective
questionnaire that have been customized and extended based on the Presence
Questionnaire (PQ) developed by Witmer and Singer (1998).
In the first phase (see Fig. 9.8), where students used tablet computers (for an
augmented reality view) and PC screens (for a mixed reality view), the initial results
showed that the immersive platform was more effective in promoting the novices’
knowledge of the basic and strategic rules for UNO, than novices who played and
learnt using a traditional web-based approach. The analysis also indicated that there
were statistically significant differences in the levels of presence between the
groups: which included a control group, a web group and the experimental group.
In the second phase of the evaluation, the augmented view within the tablet was
replaced with an augmented reality headset (MetaOne) and the virtual world view
with an immersive Oculus Rift headset which is used by the remote learners.
Figure 9.9 presents the results from the evaluation of the student’s sense of pres-
ence between the main groups. This clearly shows that the face-to-face control
group had the highest score, followed by the phase 2 MiRTLE+ system, then the
phase 1 MiRTLE+ system and finally the web-based group. These results clearly
indicate that the more immersive technologies had a positive impact on the stu-
dent’s sense of presence when participating in the learning activities.
In both phases of the evaluation the overall findings are that the learning
effectiveness and the students levels of engagement in the learning activity were
significantly higher when using the augmented reality and immersive devices when
compared to the web-based group. This is illustrated in Fig. 9.10, which shows a
comparison of the learning effectiveness of the phase 1 and 2 MiRTLE+ platforms
with a web-based group and a control group (which met face to face). However, as
expected, overall the students who worked together face-to-face performed the
highest. This probably indicates that the augmented and immersive technologies

Fig. 9.8 MiRTLE+ augmented and virtual worlds

170 M.R. Gardner and W.W. Sheaffer

Fig. 9.9 Measuring presence between groups

Fig. 9.10 Comparing learning effectiveness between groups

still have some way to go in order to fulfill their potential, but there are clearly some
advantages when compared to less immersive web-based interfaces.

9.4.2.2 Mixed-Reality Co-creative Maker-Spaces

We believe the next generation of e-learning can bring with it a fundamental shift
towards informal, interactive and business driven learning, away from the more
9 Systems to Support Co-creative Collaboration … 171

traditional formal e-learning approaches. This will be accompanied by a host of

opportunities and challenges for both the learners themselves and the institutions
delivering the courses.
Many successful examples exist of project-based collaborations built by
multi-disciplinary student teams that transcend this. For example, children in coding
clubs (e.g. coderdojo.com) mentor each other using programs like Scratch and have
subsequently self-organized to build teams to compete in robotics competitions.
Undergraduates in engineering courses collaborate with business students to create
new product prototypes; often these projects emerge from undergraduate work and
evolve towards competitive programs, and may then be reinvented as postgraduate
projects.
Maker spaces are a type of non-formal learning environment more active in
technology orientated, and other more informal creative and innovative tasks. These
workplaces have the potential to bring together people with common interests in
technology, computer science and digital art to socialize and work. Typically all of
the participants will have the common motivation for creativity, commitment to
innovation, openness to collaboration and the promotion of experiences of learning
by doing (rather than being required to use the space as part of a formal learning
task). Here, new and flexible concepts of virtual laboratories and virtual experi-
ments offer an appropriate platform for education and communication. Examples of
current Maker space initiatives in Europe include Madrid (e.g. medialab-prado.es),
Barcelona (e.g. vailets-hacklab.org, codeclubcat.org), Toulouse (e.g. tetalab.org),
Lisbon (e.g. fablablisboa.pt) and Porto (e.g. opolab.com).
In the last decade a number of fully software based virtual laboratories in dif-
ferent fields have been developed. In most cases they are specific to an educational
context and do not offer possibilities for generalization to a platform applicable to a
wider class of disciplines (Potkonjak et al., 2016). This has led to the emerging idea
of using virtual environments that enable exploration of new challenges and con-
ceptualisation of solutions through collaboration. This Maker space concept
includes a range of creative, design and end user engagement toolkits as well as
models enabling the use of a range of scientific tools, which could be composed and
implemented in an appropriate virtual laboratory. For example at the University of
Essex we have developed the ‘BReal Lab’ (Peña-Rios, Callaghan, Gardner, &
Alhaddad, 2015—see Fig. 9.11) which provides an innovative approach to
enabling geographically distributed learners to collaborate around physical engi-
neering laboratory activities, such as building robots. This mixed-reality space
allows students to share both virtual and physical electronic components in order to
jointly create new robotic applications (as distinct from how MiRTLE used audio
and video as a mixed-reality medium). It facilitates the combination of software and
hardware components, created by different students in different parts of the world,
into a single working system. The BReal Lab has been successfully tested with a
range of groups of dispersed learners and the preliminary results (Peña-Rios et al.,
2015) showed a positive learner’s experience opening up new opportunities for
online education.
172 M.R. Gardner and W.W. Sheaffer

Fig. 9.11 BREAL mixed-reality maker space

9.4.2.3 Virtual Observation Lens

There are many issues involved in creating meaningful collaborative learning tasks,
including task allocation, communication, evaluation and assessment. Working
with peers is not just a matter of gathering people together; successful collaboration
depends on who the members are, what the tasks are and how the members are
assessed individually and as a group. Learners usually acquire new knowledge
while practicing learning activities. Consequently, rather than looking at the final
product as evidence of learning, instructors should assess the whole process.
However, collecting data to trace the students’ behaviors in 3D virtual environ-
ments is typically more challenging than in face-to-face traditional learning sessions
(Chiang & Schallert, 2012).
Several issues can arise in assessing a group of learners in these environments.
First observing the users’ behavior dynamically and collecting evidence of learning
can be challenging when using a virtual world. The main problem is the abundance
of data generated by the VW. Secondly numerous skills, such as communication
and negotiation skills could be gained by students when working on collaborative
activities, however often it is difficult to automatically detect this type of evidence
in these spaces. Third labeling and recognizing the evidence of individual users in
9 Systems to Support Co-creative Collaboration … 173

real-time is particularly difficult where several learners are contributing at the same
time, which makes automatically tracking performance difficult (however, this can
also be true in real-world settings). Some of these issues were highlighted by
Gardner and Elliott (2014), in which we found that “learning within technology
creates a pedagogical shift that requires teachers to think about measuring outcomes
in non-traditional ways”.
Shute et al. in Chap. 5 examines the role of diagnostic assessment in immersive
learning environments by introducing the concept of stealth assessment, and pro-
vides an example of stealth assessment within a game environment. Our research
(Felemban, Gardner, & Callaghan, 2016) is also exploring stealth assessment
mechanisms but within a multi-user environment. We are aiming to create a novel
computational framework that enhances the observation and recording of collab-
orative learning activities in order to evaluate the individual and group learning
outcomes. In particular, the aim is to create a virtual observation model that can
map between observing and evaluating students in physical settings with observing
and assessing them in virtual worlds. The focus is more on providing methods to
identify and classify learning evidence and assess group working than mapping all
these elements to specific learning outcomes. To do so, we use an agent-based
approach to support the recording and labeling of learning evidence from virtual
activities and therefore simulate the observations normally made by a teacher.
At this time, the BReal Lab (see above) has been extended to provide a number
of tools to address these issues. The platform utilizes techniques from multi-agent
systems to track users’ actions and predict the learners’ acquired skills and
knowledge. It has two different kinds of agents: software agents and natural agents.
The software agents track learners and collect different users’ clicks and actions,
while the natural agents allow users to perform peer evaluations of each other to
evaluate the quality of their performance. When the students are taking part in
learning activities, they will be asked to act as agents and evaluate their peers via a
sliding rating scale. A key issue will be to avoid any unnecessary disruption or
cognitive overhead whilst the students are doing this. These agents are employed to
record both implicit and explicit data that will be analyzed to determine the learning
evidence and individual students’ performance. The agents will communicate and
work together in real time to collect this evidence, which will then be analyzed by a
computational model based on the use of fuzzy logic. The intention is that this
model will fully describe and map the characteristics of different learning objectives
and will provide a mechanism to map between in-world events and individual
learning achievements (based on the underlying data) through inferences derived
from the fuzzy rule set.
The central part of this research is the observation lens model that will determine
how to analyze the data that is captured by the agents. In normal practice in order to
observe students in the classroom, educators should consider numerous criteria to
gain an insight into the students’ learning processes and outcomes. As a starting
point the work of Borich (2016) has been taken to create a virtual observation
model (see Fig. 9.12), which can be applied to the collaborative virtual world in
order to evaluate the learning activities taking place. The virtual observation model
174 M.R. Gardner and W.W. Sheaffer

Fig. 9.12 Observation lens

defines different levels of granularity for observing students and recording evidence
of collaborative learning, from high to low-level observations.
Starting at the lowest level, the first level of the hierarchy is the Events Detection
lens that simulates an instructor who is watching a collaborative activity from a
general viewpoint without looking deeply into the content of what is happening. In
this level, the observer monitors the activity by understanding that a sequence of
events is occurring and capturing these events without judging the meaning of the
actions. It is envisaged that examples of the events that can be observed and
collected from the virtual world activities will include avatar event logs, chat logs,
object event logs and student self-rating logs.
The second level is the Learning Interactions lens, which considers a deeper
view of the social and environmental interactions. The social interactions are
between peers, and the environmental interactions are between students and the
virtual world. Evaluating the quality and quantity of collaborations and interactions
infers whether the learners have valuable interactions and if they are active learners
in their groups. It also determines the amount of sharing and interaction among
students. It is possible to infer the quantity and the quality of learners’ interactions
by creating fuzzy rules based on the teachers’ viewpoint. Examples of these rules
are:
• The number of a learner’s contributions in the chat log during a period com-
pared with other learners.
• The number of a learner’s contributions in using the virtual objects during a
period compared with other learners.
• The number of a learner’s contributions in completing tasks during a period
compared with other learners.
• The rating scores for a student from other members in a given period.
The third level is the Students’ Success lens. It represents the actions taken by
teachers when they are observing students’ success by counting the number of
correct answers; the number of right answers reinforced or acknowledged events,
and the number of delayed corrections.
9 Systems to Support Co-creative Collaboration … 175

The fourth level is the Performance Outcomes lens. This level simulates the
observer tracking students in depth to identify the skills and knowledge that they
have acquired from the learning activities. Because our research focuses mainly on
educating learners on technical subjects using collaborative learning activities,
students can acquire both collaborative and technical skills based on the activity
they are doing. Thus, this lens should help the virtual observer assess the knowl-
edge and skills of the individual learners.
The first two layers of the observation lenses and the mechanisms to assess how
best to translate teacher generated observation rules into descriptive logical rules are
currently being developed. The plan is to determine the observing rules that should
be created and when the rules should be activated. This will require the creation of a
fuzzy model, which will be validated through a series of expert evaluations. In terms
of assessing this approach for collecting learning evidence, the intention is to assess
how much data should be collected to yield sufficient evidence of learning, and how
much should be collected from natural agents and how much from software agents.
In this work a determination will need to be made as to how different types of data
can be automatically grouped to demonstrate meaningful evidence of learning.

9.5 Discussion

The Saint Paul case study clearly shows that the technology is only one part of the
picture, and the other aspects (particularly behavioral aspects) are equally important
when designing new learning activities.
This work has also shown that the mixed-reality approach can work well, par-
ticularly when incorporated into a sound pedagogical/behavioral approach. The
evidence from the Saint Paul case study also reinforces this.
MiRTLE was deliberately very simplistic in terms of the pedagogy being sup-
ported (but effective because it was so simple). MiRTLE was successful not
because it was technically advanced but because it was so natural for the instructors
to use, requiring no new training or lesson planning. Effective use of any of these
platforms in an educational setting is totally dependent on the structure of the
pedagogy. Here pedagogy is king and technology is servant. This might explain the
reason for the relatively slow uptake of virtual and mixed-reality where often due to
the complexity of the platforms a lot of effort is spent on the technology and little
effort on how to apply it. Here the relationship the technological system has to the
environment it will be operated in is a key success factor. In addition the work
being done at the University of Essex to extend the MiRTLE platform particularly
with the use of new devices and more collaborative learning activities, shows great
promise, and seems to lead to more engaging, and effective learning (when com-
pared to more traditional approaches).
Once this approach becomes more embedded, it should then be possible to
leverage more of the affordances of using virtual and mixed-reality spaces for
176 M.R. Gardner and W.W. Sheaffer

learning. For example, the Essex work to create an assessment lens should help
students and instructors better assess the learning outcomes achieved.
A key aspect arising from this work is the challenge of designing more effective
virtual spaces for learning. Our research has shown some of the benefits from taking
a more constructionist approach to learning. This is backed up by other research
(Ross, 2013) that proposes a more student-centred pedagogy in which students are
more involved and active in their learning and construction of knowledge. This has
led to increased emphasis on active and collaborative learning activities and sub-
sequently the creation of new Maker spaces (as discussed above) that incorporate a
significant leaning role. The importance of the design of the environment on the
learning outcomes is also reflected in other studies (Barrett, Zhang, Moffat, &
Kobbacy, 2013; Perks, Orr, & Al-Omari, 2016), that have focused on changes to
the physical environment of classrooms and schools and its impact on the students
as a space for learning.
Designing and building effective online spaces can be demanding and time
consuming for stakeholders and also often requires high levels of technical
expertise. User customization has the potential to improve engagement and expe-
rience in virtual spaces, much in the same way as personalizing homes makes it feel
and work better for its inhabitants. Thus, to achieve better online experiences, such
3D spaces need to be easier to build and well designed. Also, such environments
need to provide its users with a rich experience and greater flexibility to suit their
needs. Jacobson further discusses some of these issues in Chap. 3 where he focuses
on the challenge of creating authenticity in immersive design as a principle for
guiding authors of educational immersive media.
Recent studies have found that the influence of physical and psychological
design parameters influence the users experience in built environments. These
parameters are linked to the ‘responsiveness’ of such environments. Virtual worlds
show great promise for increased visualization, immersion and enhancing the users’
experiences but the current evidence suggests that they struggle to compete with
their physical counterparts. One approach moving forward would be to capitalize
on findings from the design of built environments to try to develop better designs
that can be used beneficially in 3D online worlds. According to Bentley (1985) a
built environment should provide its users with flexible settings and opportunities to
maximize the choices available in their environment. Such environments with these
affordances are said to be ‘responsive’ and can be characterized according to well
defined architectural and user characteristics. The challenge in creating new mixed
reality spaces for learning is to combine the benefits of good architectural design
with the flexibility and customizability that is afforded by the use of these digital 3D
spaces.

Acknowledgements We would like to acknowledge the work of the team of researchers in the
Immersive Education Lab in the School of Computer Science and Electronic Engineering at the
University of Essex that supports much of the research described in this paper. This particularly
includes the contributions of Prof. Vic Callaghan, Dr. Anasol Peña-Rios, Ahmed Alzahrani,
Samah Felemban and Salako Oluwatimilehin Samuel.
9 Systems to Support Co-creative Collaboration … 177

References

Barrett, P., Zhang, Y., Moffat, J., & Kobbacy, K. (2013). A holistic, multi-level analysis
identifying the impact of classroom design on pupils’ learning. Building and Environment, 59,
678–689.
Bentley, I. (1985). Responsive environments: A manual for designers. UK: Routledge.
Bloom, B. S. (Ed.), Engelhart, M. D., Furst, E. J., Hill, W. H., Krathwohl, D. R. (1956). Taxonomy
of educational objectives. In Handbook I: The cognitive domain. New York: David McKay Co
Inc.
Borich, G. (2016). Observation skills for effective teaching, UK: Routledge.
Chiang, Y., & Schallert, D. (2012). Instructional design meets politeness issues in virtual worlds.
In S. D’Augustino (Ed.), Immersive Environments, Augmented Realities, and Virtual Worlds:
Assessing Future Trends in Education: Assessing Future Trends in Education (p. 123).
Information Science Reference, Hershey: PA, USA.
Dalgarno, B., & Lee, M. J. W. (2010). What are the learning affordances of 3-D virtual
environments? British Journal of Educational Technology, 41(1), 2010.
Dooley, J., Callaghan, V., Hagras, H., Gardner, M., Ghanbaria, M., & Alghazzawi, D. (2011). The
intelligent classroom: Beyond four walls. proceedings of the intelligent campus workshop
(IC’11) held at the 7th IEEE Intelligent Environments Conference (IE’11), Nottingham.
Felemban, S., Gardner, M., & Callaghan, V. (2016). Virtual observation lenses for assessing
online collaborative learning environment. iLRN 2016 Santa Barbara Workshop, Short Paper
and Poster Proceedings from the Second Immersive Learning Research Network Conference.
Gardner, M. and Elliott, J. (2014). The Immersive Education Laboratory: Understanding
affordances, structuring experiences, and creating constructivist, collaborative processes, in
mixed-reality smart environments. EAI Endorsed Transactions on Future Intelligent
Educational Environments, 14(1), e6. ISSN: 2409-0034.
Gardner, M., & O’Driscoll, L. (2011). MiRTLE (Mixed–Reality Teaching and Learning
Environment): From prototype to production and implementation. Workshop ‘Learning
activities across physical and virtual spaces’ (AcrossSpaces). EC—TEL 2011 conference,
Palermo (Italy), September 20–23, 2011.
Milgram, P., & Kishino, A. F. (1994). Taxonomy of mixed reality visual displays. IEICE
Transactions on Information and Systems, E77-D(1), 1321–1329.
Peña-Rios, A., Callaghan, V., Gardner, M., & Alhaddad, M. (2015). Experiments with
collaborative blended-reality laboratory technology for distance learners. Published in the
iLRN 2015 Prague Workshop, Short Paper and Poster Proceedings from the inaugural
Immersive Learning Research Network Conference.
Perks, T., Orr, D., and Al-Omari, E. (2016). Classroom re-design to facilitate student learning: A
case study of changes to a university classroom. Journal of the Scholarship of Teaching and
Learning, 16, 53–68.
Potkonjak, V., Gardner, M., Callaghan, V., Mattila, P., Guetl, C., Petrovic, V., & Jovanovic, K.
(2016). Virtual laboratories for education in science, technology, and engineering: A review.
Computers & Education, 95, 309–327. (Elsevier).
Ross, K. C. (2013). Creating dialogical spaces in blended environments: A case study of classroom
design. Practical Applications and Experiences in K-20 Blended Learning Environments, 280.
Schmidt, M., Kevan, J., McKimmy, P., & Fabel, S. (2013). The Holodeck: Mixed-reality teaching
and learning environment at the University of Hawaii, Manoa. Virtual Education Journal.
Witmer, B., & Singer, M. (1998). Measuring presence in virtual environments: A presence
questionnaire (Vol. 7, No. 3, pp. 225–240). Presence: MIT Press.
178 M.R. Gardner and W.W. Sheaffer

Author Biographies

Michael Robert Gardner has a B.Sc. (Hons) in Computer Science (1984) and a Ph.D. in
Computer Science (1991) from Loughborough University in the UK. Previously he worked for 15
years as a Research Scientist at the British Telecommunications (Adastral Park) research labs at
Martlesham Heath in the UK. He then setup and became the Deputy Director of the Institute for
Social and Technical Research at the University of Essex. Currently he is a Senior Lecturer in the
School of Computer Science and Electronic Engineering at the University of Essex. In this role he
is also Director of the Immersive Education Laboratory. He has published over 100 papers which
include conference, journal and book articles. His primary research interests are in understanding
the affordances of immersive education technologies and developing systems to support
co-creative collaboration in mixed-reality environments. He is on the board of the newly created
Immersive Learning Research Network (iLRNetwork), which aims to develop and support a
community of educators, scholars, and practitioners dedicated toward research in and on
digitally-enhanced immersive learning environments.

Warren W. Sheaffer is a visiting Professor at the University of Essex and is a faculty member in
the department of Mathematics and Computer Science at Saint Paul College. He served as the
department chairman of the Computer Science department at Saint Paul for 15 years implementing
a variety of new programs and technologies including many which utilize virtual reality in
education. His area of technical interest is man-in-the-loop optimization systems for scheduling
problems within stochastic networks. His area of pedagogical interest is the use of virtual reality
and visualization systems to improve learning outcomes. He is currently serving on a standards
committee for computer science education for undergraduates.
Chapter 10
Massively Multiplayer Online Roleplaying
Games and Virtual Reality Combine
for Learning

Eric Klopfer

Abstract The places where Virtual Reality (VR) can really make a difference in
learning are those in which the VR can bring a truly unique experience to students.
The simulated online world of games is an ideal way to take advantage of the
capabilities of this new technology. Games provide a set of structures that not only
scaffold learners in solving complex problems but also provide a great deal of
freedom to explore personally interesting pathways. In particular, Massively
Multiplayer Online Role Playing Games (MMOs) offer an environment that sup-
ports social learning and exploration around increasingly challenging problems. VR
can greatly enhance MMOs through opportunities for more natural and expressive
communication and collaboration as well as ways to visualize the complex infor-
mation resulting from interactions in this space. When this approach is applied in an
educational context, learners can be presented with challenging problems, requiring
participation from multiple players around realistic scientific concepts. As this
genre moves forward it can explore interesting hybrid approaches that combine VR
with Augmented Reality (AR) and traditional displays to meet the needs of schools,
teachers, and learners.

Keywords Games
Learning
Science education
Collaborative learning

10.1 Introduction

I was recently reading an announcement (Whitmore, 2016) about a new school that
will have virtual reality (VR) as a central theme. In discussing the ways in which
they would use VR, this description was offered:

E. Klopfer (&)
Scheller Teacher Education Program, Massachusetts Institute of Technology, 77
Massachusetts Avenue, NE48-328, Cambridge, MA 02139, USA
e-mail: [email protected]
URL: http://education.mit.edu

© Springer Nature Singapore Pte Ltd. 2017 179

D. Liu et al. (eds.), Virtual, Augmented, and Mixed Realities in Education,
Smart Computing and Intelligence, DOI 10.1007/978-981-10-5490-7_10
180 E. Klopfer

Imagine this: Students enter the lab, strap on a virtual reality headset (just like the HTC
Vive Andrew [one of the people at the school] had me put on during a demonstration), and
instead of playing video games, students will enter a fully immersive and scientifically
accurate virtual reality chemistry lab. They will see their lab partner, and the two will
discuss what they will do in the experiment, just as students would do in a traditional lab.
Let’s lay out a simple experiment: Does adding salt affect the boiling point of water? The
student would reach out with hand controllers, take a graduated cylinder, fill it with water,
measure out the salt, light a Bunsen burner, add a thermometer, track the boiling point —
and then repeat the experiment without adding salt.

As new technologies are introduced, they are often initially adapted to previous
practices. Going back in time, the first films were simply movies of live plays. Later
films explored cameras that moved around, went outside and shot on location. And
today we see a diverse use of this medium. More recently, we saw the first wave of
Massive Open Online Courses (MOOCs) broadcasting videos of lectures (Papano,
2012). This medium has also started to change, incorporating more features of the
unique online environment and scale that they offer.
So this usage of VR is a predictable initial foray for the technology. Before we
dream up new kinds of interactive experiences, we recreate existing practices in a
new medium (see Dede’s “Old Wine in New Bottles”). While the pathway to such
experiments may allow for rapid development, boiling water in VR is unlikely to
lead to great advances in 21st century learning. Rather than recreating 20th century
laboratory practices to demonstrate known phenomena, we should be doing exactly
what this passage passes over. Instead of having students replicate those prior
practices, they should be playing video games.

10.2 Why Video Games?

Why should students be playing video games instead of conducting labs? That is
actually a false dichotomy. They should be playing video games and conducting
labs. Some of those labs should be in real life. If Bunsen burners are unavailable,
then draw upon kitchen science. Experiments in virtual reality can permit working
in domains that are difficult or impossible to recreate in the real world. Experiments
can take place on long time scales like that of evolution, at microscopic levels in
chemistry experiments, or on the inside of a nuclear reactor.
But those labs can also be embedded in or connected with games. A game world
can provide context, identity, and purpose for conducting such investigations and
make the experience more meaningful and (while it might seem counterintuitive) it
can also make them more authentic. For many students, isolated labs have little
meaning. Breeding peas to determine whether wrinkled or smooth is dominant
(long after Mendel established this) might seem quite routine. But breeding plants
to find the variety that will help cure a (fictional) illness in an online world can
situate the activity in a more personally meaningful context.
10 Massively Multiplayer Online Roleplaying Games … 181

While some look to games merely to “make” an activity fun, their real advantage
comes in providing structure around activities that allow for freedom and autonomy,
yet also provide enough boundaries to help guide learners. There are many defini-
tions of games, some of which focus on narrative, identity, or the separation of play
and reality. My definition has focused specifically on the structures of games. These
five principles describe these structures or “gaminess” (Klopfer, 2015).
• Interesting Decisions—This is Sid Meier’s (the creator of Civilization) hallmark
of game design. Good games are defined by the choices that players make.
Deciding between heads and tails is not interesting, but making decisions that
are informed by previous experiences and insights often is.
• Consequences to Decisions—There needs to be some set of consequences to the
outcomes of the player’s decisions. Those outcomes can be positive or negative,
providing feedback to the player as to how those decisions are valued in the
context of the game.
• Clearly Defined Goals—The game should provide a set of constraints and
opportunities so that the player either knows a priori, or can easily discover what
they are trying to accomplish. In good games, the player can choose between
multiple different goals and can pursue them in a way that they see fit. For
example, a player may choose to accumulate the most wealth, conquer the
world, or broker peace.
• Visible Measurable Feedback—The player needs to know how they are doing.
Ideally, they get this kind of feedback along multiple dimensions. This is not just
a single measurement of score, but might include level of achievement, health,
or gear that they have collected. This feedback may reflect “stealth assessments”
that are a part of the game (see Shute in this volume).
• Underlying model or systems—There should be a coherent set of rules that
define an underlying system. In digital games this might take the form of a
simulation or set of simulations. But it can also be a comprehensive set of rules
that define the mechanics of a non-digital system (Fig. 10.1).
There are many ways that these principles can be manifest in games. They could
be provided through structured narratives, quest structures or other game systems
and mechanics. Many great games embody many or all of these structural princi-
ples. These could be strategy games like Civilization, battle arenas like League of
Legends, or Massively Multiplayer Online Role Playing Games (MMOs) like
World of Warcraft. The latter is a particularly good example.
World of Warcraft (WoW) is a game set in a fictional world occupied by two
warring factions consisting of fantasy races like elves, dwarves, orcs and even
humans. Players choose a faction, race and class that specifies their roles, like a
magical cleric or a hunter. They are then given a series of tasks to complete in the
world by numerous non-player characters (NPCs) that inhabit the world. These
tasks (or quests, as they are known) might be about collecting a certain number of
items from a dangerous place in the world, or killing a terrible beast that has been
attacking your folk. Players attempt to complete these tasks, granting them new
182 E. Klopfer

Fig. 10.1 World of Warcraft quest from http://www.mobygames.com/images/shots/l/91220-

world-of-warcraft-windows-screenshot-the-quest-log-shows-you.png

skills and items. But they do so in a context that includes many other real players
who inhabit the world. Most of those players are on the same side and are there to
help. There are some quests that can only be completed by working with others.
Such “dungeons” require collaboration with small groups. As players gain more
experience they are challenged to collaborate in large groups (dozens of people)
through “raids.” Thus, the game exemplifies gaminess in the following ways:
• Interesting Decisions—The game offers players a variety of choices, beginning
with their faction, race and class. As their characters level up, players need to
continually make decisions about their specialization. While the world isn’t truly
“open” it has many places to explore and players need to choose where to go
and what quests to take on. In battle, players choose from the array of abilities
and weapons on hand to accomplish the task.
• Consequences to decisions—Succeeding in a quest leads to rewards. These
include money, items and experience that can be used to level up. The more
interesting consequences come from the choices about how to increase and
specialize abilities. One can be a generalist, but at the cost of not being really
great at anything. Or one can specialize in one domain, but be at a loss when
another set of skills is needed.
• Clearly defined goals—For some players, WoW is about leveling up as quickly
as possible. For others it is about obtaining some particularly rare item that can
10 Massively Multiplayer Online Roleplaying Games … 183

be worn around. For still others it is about showing leadership within a group
with whom one is collaborating. This diverse array of goals makes the game
interesting to a range of players.
• Visible measurable feedback—On the way to accomplishing different sets of
goals, Wow provides many levels of feedback. Players have wealth, items,
experience, achievements, and reputation, to name just a few. Different players
can choose to value these in ways that are personally meaningful and relate to
their goals.
• Underlying model or system—WoW consists of many related systems. There
are systems of weapons and spells. There are also systems that govern where
and how often particular items are found. Discovering and debating the rules of
those systems becomes an important thread in online discussions where players
discuss their theories.

10.2.1 Educational Video Games—The Story of Radix

Many attempts have been made over the years to combine video games and
learning. In some cases that means applying the superficial components of games
like scoring or shooting to an otherwise mundane learning activity. The game Math
Blaster exemplifies this kind of gamification. Math problems appear in the sky and
the player needs to shoot down the right answer. There is no connection between
the game play and the learning. The problems floating in the sky could just as well
be vocabulary words.
Compare that game to The Logical Journey of the Zoombinis (Hancock &
Osterweil, 1996), another long-lived educational math/logic game. But in this game
the player never actually sees math problems. Instead, they solve mathematically
modeled problems that are situated in a world inhabited by lovable creatures the
player is trying to save (Fig. 10.2).
There have also been attempts at using commercial video games directly in the
classroom. One such attempt (Steinkeuhler & Duncan, 2009) centered on WoW. In
particular, it focused on the ways in which scientific discourse was incorporated
into online discussion about the game. Players craft theories about how the systems
work within the game and then collect and analyze data to test those theories. It
might be about where the highest probability location for a particular item is, or the
right sequence of spells to take down an enemy. Another topic was how players
should best work together to accomplish complex tasks. Steinkuehler and Duncan
found that many players used some fairly sophisticated theories and engaged in
many levels of scientific discourse.
Players also invest a tremendous amount in their characters, developing them
over months or years. At one point in time the most secure login system that I used
was for my WoW character. It was more secure than my login for my bank, as I
knew my character was nearly irreplaceable. That investment in the character
immerses the player in the world in a deep and rich way. When that character
184 E. Klopfer

Fig. 10.2 The Logical Journey of the Zoombinis (https://external-wiki.terc.edu/display/ZOOM)

showing a puzzle in which players must get all of the characters across the bridges based on an
unknown logical rule

investment is combined with the social investment that players make—connecting

to other players with whom they collaborate—it makes for a truly immersive
experience.
Building on this idea, we (Clarke-Midura, Rosenheck, Haas, & Klopfer, 2013)
created an educational MMO, Radix, about mathematics and biology. Radix is an
MMO set on an earthlike planet in a Renaissance-like era of knowledge. Players
take on the role of a novice in an underground society trying to use knowledge to
make the world a better place. The quests that players take on relate to core content
in either science or mathematics in one of several topic areas. For example, there is
a quest line in mathematics that requires players to use geometry to reconstruct
broken down buildings. Players are given tools that help them compute and con-
struct the shapes that they need. Another quest line involves understanding human
body systems. The player takes on the role of a physician who must isolate the
system that is malfunctioning based on the symptoms patients present with. The
challenge is to make these activities authentic, as discussed in Jacobson’s chapter,
even in a fantasy world.
Within each of these areas the quests get more challenging as the player pro-
gresses. A first challenge might simply be about understanding the underlying tool.
10 Massively Multiplayer Online Roleplaying Games … 185

The next quest might be about applying the tool in a basic way. Each subsequent
quest makes the challenge more difficult. Failure along the way results in another
opportunity to complete the quest.
Radix was designed in such a way that those failures, as well as the successes,
are informative. Many of the quests were built on principles of Experiment
Centered Design (XCD) (Conrad, Clark-Midura, & Klopfer, 2014), a variant of
Evidence Centered Design (Mislevy, Almond, & Lucas, 2003), which is also used
in designing Stealth Assessments (see Shute in this volume). In XCD learners are
challenged to conduct experiments. Each experiment provides results. Based on
those results players can take further action, which might include further experi-
ments. By interpreting these data we can deduce what students do and do not
understand. For example, if a player is challenged to get “true breeding” (plants that
breed to create plants with identical characteristics) medicinal plants, they would
conduct a series of genetics experiments. If they breed two plants and get a mixture
of offspring we can understand a lot based on what they do next—do they breed the
offspring, go back and find new parent plants, or simply breed them again? This
methodology gives the players a lot of choice and agency, but still allows us to
better understand the model of the system that they have in their heads. In turn, the
game can provide feedback to the students or their teachers based on these out-
comes (Fig. 10.3).
Over time players unlock more capabilities in their characters and discover new
parts of the world. They can also collaborate with peers and classmates on different

Fig. 10.3 Radix, an educational Massively Multiplayer Online Roleplaying Game (http://
radixendeavor.org)
186 E. Klopfer

tasks, sharing data and knowledge to make the tasks more tractable. This form of
collaboration, however, only scratches the surface of what is possible in these
spaces.

10.3 VR + MMO + Learning

There is a huge potential for MMOs in VR. While some aspects of MMOs are
adequately handled in 2D, or even better handled in this way, there are some
aspects that can greatly benefit from this technology. Some early attempts already
exist (see Looking Glass, 2016) and others are on the way (https://orbusvr.com).
While Minecraft isn’t an MMO, it shares some common facets, and it, too, has a
VR version.
There are several reasons why MMOs in particular stand to gain a lot from VR
implementations.
• Immersion—The most obvious reason is immersion, which I will interpret here
as “investment.” Part of the success of MMOs is immersion (similar to narrative
immersion). There is the immersion in the character, social interactions and the
world itself. Really being invested in a character means being invested in the
world. VR can make the player a part of that world. A player becomes more
invested in the fate of the world, the assets that they must protect or seize to
accomplish their goals, if they feel that world is real. A player in a VR version of
a game like Radix can benefit from the immersion in the character, interactions
and the world, which are enhanced by removal of other distractions and a deeply
connected first person perspective. See the world as if the player and the
character are one can change the perspective of that player (see Slater in this
volume).
• Presence—Presence is also fairly obvious and what I see as feeling like you are
really in the virtual world. Complex tasks within WoW are notoriously chal-
lenging and require real time understanding of a multitude of data and factors.
Screens of knowledgeable players (Fig. 10.4) contain dozens or hundreds of
buttons, meters and heads up displays to keep them apprised of the situation.
Bringing the player into the world allows them to experience and visualize that
information in entirely new ways. Displays can be embedded in the world, or
better yet, the ability to focus and perceive elements within the 3D space can be
used to directly convey information for the player. The best representations and
interfaces will need to be designed for these worlds to take advantage of the
increased 3D perception, and situational awareness. For example in a game like
Radix, rather than having detailed tools broken out into a zoomed in view in
another panel, the tool could be used in context directly in the virtual world.
• Collaboration—The biggest possible contribution of VR to MMOs has to be
around collaboration and social interaction (discussed in Kraemer’s chapter).
What really makes MMOs interesting and unique are the Massive and
10 Massively Multiplayer Online Roleplaying Games … 187

Fig. 10.4 World of Warcraft raid interface showing the kind of displays and information players
use. https://noggenfoggerblog.files.wordpress.com/2013/06/raidui1.png

Multiplayer parts. Sometimes that interaction is simply around two players who
run into each other in some part of the world, exchanging ideas or helping each
other on a task in which they are both involved. I’ve been continuously sur-
prised and elated by the friendly help I receive from other players who just are
passing by. In most cases the game is designed such that tasks become easier
when players collaborate, encouraging this behavior. For example, players
might be collecting unique samples of some flowers in the world. When two
players encounter each other they might share some of the samples that they
have each found.
Other times the collaboration is a lot more intense; every player in a group
numbering dozens must coordinate their efforts down to a fraction of a second to
take down “the boss,” an opposing powerful NPC. In these cases information
must flow through multiple channels, including not only the meters and displays
shown above, but also audio and chat channels. In a game like Radix, this could
be mean a biologist finding the genetic weakness of invading monsters, while a
chemist creates substances that could attack those weaknesses.
But VR has the potential to make collaboration and social interaction much
more productive and natural. This is why Facebook, the massive social network,
purchased Oculus, one of the first producers of VR consumer hardware. They
see VR as the next frontier for social interactions. Recent announcements (Lee,
2016) have started to give indications of what this will look like. Players might
need to hold two sides of a lens to reflect light in the proper direction to activate
a secret entrance, or simultaneously cleave a DNA segment to edit genes.
188 E. Klopfer

While the typical Facebook interaction might be different than the kind of social
interaction that makes up an MMO, many of the same principles will apply. It
should be easy to see and understand the actions and expressions of your peers
in the virtual space. It should be natural to interact with shared objects. It should
be obvious what people are doing and where they are headed. These same traits
are useful in an MMO.
In fact, the next generation of social interactions in MMOs may look quite a bit
different than the ones in current games. We may see more interpersonal and
expressive interactions enabled by higher bandwidth, greater graphics fidelity,
new controllers and VR headsets. MMOs might try to blur the lines between the
game and the network of people working in the game so that other kinds of
social interactions are supported, keeping players in the game world longer.
Getting back to the initial story on VR in schools, VR does show the promise of
supporting greater collaboration around shared artifacts. Those could be test tubes
filled with salt water for conducting a traditional lab. But a much more exciting
opportunity is that those could be test tubes filled with alien DNA for analysis, or
used to collect and analyze chemicals from a toxic waste site. These scenarios not
only involve substances and scenarios that are unobtainable in real life, they can
also be situated in a context that provides a rationale and in which the learner can
immerse themselves. They can become something beyond a student in a class doing
a lab, perhaps a scientist, adventurer, or pioneer (as illustrated in Dede’s chapter).
This is important for helping students develop an identity.

10.4 Innovation on the Horizon

So why aren’t we seeing educational VR MMOs yet? There are a number of

reasons. They require a lot of time and money to create, which means that before
anyone can be successful in that space there are a few innovations that need to be
worked out. One set of challenges involves the barriers that any MMO might face,
which include the massive domain of content to be authored, which require
authoring tools, which could be made accessible to individual content creators.
Other VR specific challenges include:
• Interfaces—The existing set of interfaces for managing information and inter-
actions within MMOs have been designed for relatively small flat screens. Yes,
those can simply be ported to VR, but then there won’t be a big advantage for
VR. Instead, new interfaces that can both be situated in 3D space and that can
display information in novel ways need to be created. People’s abilities to focus
on elements in the world and to perceive the relationship between information
and objects will change in VR, and that can lead to exciting new visualizations.
• Controls—MMOs are notoriously complex and often require the use of the
whole keyboard for navigation, use of abilities, and control of displays. Using a
10 Massively Multiplayer Online Roleplaying Games … 189

keyboard with an opaque VR display adds new challenges to using this kind of
control scheme. New VR controllers for navigating in 3D space and performing
complex operations are starting to emerge. These controls need to be adapted to
the space of MMOs to support the kinds of interactions they need.
• Settings/Context—Many MMOs are set in space or Tolkienesque lands inhab-
ited by mythical creatures. But VR opens up the opportunity to situate these
games in other kinds of places. They could be real societies, historical sites, or
scientific microcosms. Being immersed in these kinds of spaces in VR becomes
an exciting opportunity.
• Pacing—Playing an MMO is often a serious time commitment. Some MMOs
have sought to break this time constraint by chunking play into smaller pieces.
Being immersed in VR, particular when considering school context, likely needs
to further advance this work of breaking play into smaller, manageable chunks
of time.
• Collaboration—As mentioned above, this is perhaps the biggest and most
exciting challenge for MMOs. What are the new forms of collaboration that can
be fostered through VR in terms of social interactions and shared objects?
Can players communicate with a nod of their head or wave of their hand? And
can they interact with the same warp drive motor to try to repair it? This will
make for rich game play and educational experiences. There may also be
in-game and out-of-game collaboration opening up a range of mixed reality
experiences. Perhaps one player is in VR, and the other is in the real world,
feeding them live data through a mobile device that is incorporated in the game
or simply using a flat display to provide a different perspective on the world.

10.5 Next Generation VR MMO for Learning

So what might a next-generation MMO for learning look like? I might start with a
premise something like that of Radix, being part of a group trying to use science
and math to make the world a better place. But perhaps it is set in the near future, in
a world created through mapping and filming of actual cities. Ideally, this might be
created via an openly shared mapping database that could be used for many VR
applications, thus reducing the cost and barrier to entry for creating rich VR spaces
for learning.
Given the current context, the problems to be solved might be based on real
issues like fighting emerging diseases, breeding crops to cope with global climate
change, or tracing the source of air pollution hundreds of miles away. Players might
specialize in different domains like genetics, air quality, or data analysis. In the VR
world, they can become that character, and take on that unique identity.
In one scenario, perhaps players need to manage an emerging disease in the
southern United States (think something similar to Zika). They need to assemble a
team consisting of someone who is role-playing a doctor, another who is a DNA
expert, another who is an entomologist, and one who is a mechanical engineer,
190 E. Klopfer

along with several others. They need to build a trap to collect mosquitoes for
analysis and must collaborate to assemble the pieces and build the machine. They
use specialized controllers to move the parts around, often requiring two people to
lift together, and haptic feedback indicates when the pieces are in place.
Meanwhile, another team is collecting blood samples and analyzing them.
Instead of using the traditional analysis tools, they can shrink down into the
bloodstream and examine cells and molecules as they whiz by. One player needs to
keep an eye on what is coming down the bloodstream and communicate to make
sure the players don’t get washed away.
Yet another team is in command central on traditional computers, monitoring the
situation and advising the other players in real time. Flat displays might still be
useful in the context of such activities as displaying maps, text, images and
allowing seamless access to real world information such as laboratory equipment,
documents, and peers.
The game might even include some Augmented Reality, relaying real time
information to players’ phones throughout the week. Augmented Reality (AR) can
mean many different things. There is tabletop AR, as one might see through a
specialized phone app showing a 3D model that pops out of a book, room scale AR,
as seen on platforms such as Hololens, and landscape scale AR, which has been
popularized in Pokémon Go. Landscape scale AR has quite a history in education
(e.g. Dunleavy, Dede, & Mitchell, 2009; Klopfer, 2008; Squire & Jan, 2007). In
much of this work, AR is used to add an interesting but fictional layer onto the real
space, such as a school yard or nature center of museum, in which players are
exploring. Also, in these experiences players work on teams to solve complex
problems. The AR nature of the experience allows them to rely quite a bit on reality
for communication using familiar devices like phones or face-to-face communi-
cation to work together.
In this case, blending in AR might mean that players can get updates on situ-
ations they are monitoring, and their own city might be a part of the
problem/solution as well. So as they come to and from school, or go about their
after school chores, they can be pushed new information based on their location and
scan areas for problems/solutions using a simulated heads up display on their
phone. This borrows from the genre of Alternate Reality Games that blend game
play and the players’ lives (McGonigal, 2008). Perhaps in this scenario players are
combating an outbreak of a disease in the virtual world, but they can pick up virtual
medical supplies at any of thousands of medical facilities geotagged across the
country. Or perhaps players need to pick up certain elements only found in par-
ticular regions of the world and then share them within the game.
The teacher could also interact in a variety of ways. They could drop in through
VR to experience what many of the students are doing, teleporting from one
location to another. Many teachers might find it too difficult to immerse themselves
in VR because doing so means losing sight of the whole class. A more likely
scenario might be a single display on a flat screen. It will allow teachers to track
what their students are doing in real time, allowing them to see who is struggling
and who is succeeding and to offer students real-time assistance.
10 Massively Multiplayer Online Roleplaying Games … 191

10.6 The Reality of School

Conducting an activity like the one described above in most schools would provide
a significant challenge. For one, if they have VR capabilities, it might be on a single
workstation for demonstration, or with relatively simple devices like Google
Cardboard, which don’t allow for highly interactive environments. It will be some
time before schools have the technical capabilities and resources to conduct these
activities. That means emphasizing one design principle—differential experiences
on different devices. The kinds of hybrid activities in which some are in VR and
others are on traditional displays can create immersive experiences that still draw
upon the unique advantages of VR without trivializing the unique aspects of VR.
But there are more fundamental changes to both school and games that need to
happen to facilitate this activity, as discussed in Richards’ chapter.
First, the activity needs to be pervasive. If it is just a matter of short experiences
in the computer lab, then these experiences will have little impact on student
learning. The activity itself can be pervasive through mobile extensions and explicit
connection to the curriculum. Students can monitor the scenario, communicate, and
perhaps even launch automated exploration through mobile devices. This allows
them to be immersed in the experience even when they aren’t in VR.
Second, the activity needs to be persistent throughout the year, or at least a
longer period of time. It will take some setup to make the activity work, and that
investment in time can be justified by maintaining the activity over a longer period
of time. Additionally, this persistence allows students to develop and explore their
identity within the virtual space, investing the time and resources to specialize their
characters and make them unique. This is the investment we see in commercial
MMOs, and we should see similar investment in educational MMOs.
Finally, school needs to change some. These kinds of activities are great for
getting students to take new perspectives, develop identities, learn how to collab-
orate around complex tasks, and challenge misconceptions. But it doesn’t compress
a lot of factual learning into a short period of time. Leading scholars and educators
agree that that the kinds of learning promoted in this sort of experience is the kind
of learning that we need to be offering students today. We just need to align that
perspective with what actually goes on in school.

References

Clarke-Midura, J., Rosenheck, L., Haas, J., & Klopfer, E. (2013). The Radix endeavor: Designing
a massively multiplayer online game around collaborative problem solving in STEM. In
Proceedings of the Computer Supported Collaborative Learning Conference (CSCL),
Wisconsin, Madison, 15–19 June 2013.
Conrad, S., Clarke-Midura, J., & Klopfer, E. (2014). A framework for structuring learning
assessment in a massively multiplayer online educational game: Experiment centered design.
International Journal of Game-Based Learning, 4(1), 37–59.
192 E. Klopfer

Dunleavy, M., Dede, C., & Mitchell, R. (2009). Affordances and limitations of immersive
participatory augmented reality simulations for teaching and learning. Journal of Science
Education and Technology, 18(1), 7–22.
Hancock, C., & Osterweil, S. (1996). Zoombinis and the art of mathematical play. Hands on!,
19(1). Cambridge, MA:TERC.
Klopfer, E. (2008). Augmented learning: Research and design of mobile educational games.
Cambridge: MIT Press.
Klopfer, E. (2015). Learning games are hard fun. Retrieved from http://onlineteachered.mit.edu/
files/games-and-learning/week-1/Games-Week1-Klopfer-as-presented-compressed-pdf.pdf.
Leen, N. (2016, October 6). Facebook shows how you’ll hang with friends in VR. Engadget.
Retrieved from https://www.engadget.com/2016/10/06/oculus-shows-how-you-ll-hang-with-
friends-in-vr/.
Looking Glass. (2016, August 29). Virtual reality MMORPG—people will never leave their
homes. Retrieved from http://lookingglass.services/virtual-reality-mmorpg/.
McGonigal, J. (2008). Why I love bees: A case study in collective intelligence gaming. The
ecology of games: Connecting youth, games, and learning, 199, 227.
Mislevy, R. J., Almond, R. G., & Lukas, J. F. (2003). A brief introduction to evidence‐centered
design. ETS Research Report Series, 2003(1), i–29.
Pappano, L. (2012). The year of the MOOC. The New York Times, 2(12), 2012.
Squire, K. D., & Jan, M. (2007). Mad city mystery: Developing scientific argumentation skills
with a place-based augmented reality game on handheld computers. Journal of Science
Education and Technology, 16(1), 5–29.
Steinkuehler, C., & Duncan, S. (2009). Scientific habits of mind in virtual worlds. Journal of
Science Education & Technology, 17(6), 530–543.
Whitmore, R. (2016, September 14). A D.C. charter wins $10M to invent virtual reality.

Author Biography

Eric Klopfer is Professor and Director of MIT’s Scheller Teacher Education Program and The
Education Arcade. His research focuses on the development and use of computer games and
simulations for building understanding of science, technology, engineering and mathematics. The
games that he works on are designed to build understanding of scientific practices and concepts as
well as critical knowledge, using both mobile and web-delivered game platforms. Klopfer’s work
on simulations focuses on students’ understanding complex systems through critical thinking and
connecting computer programming with scientific practice and real-world issues. He is the
co-author of the books Adventures in Modeling, The More We Know, and the upcoming Resonant
Games, as well as author of Augmented Learning. His lab has produced software that includes the
UbiqBio line of mobile biology games, the Massively Multiplayer game, and The Radix Endeavor,
as well as platforms like StarLogo TNG for modeling complex systems, Taleblazer for creating
augmented realities, and Gameblox for making games online. His work also includes a series of
Massive Open Online Courses known as edTechX, which cover educational technology and
games. His work has been funded by federal agencies including NIH, NSF, and the Department of
Education, as well as the Gates Foundation, the Hewlett Foundation, and the Tata Trusts. Klopfer
is also the co-founder and past President of the non-profit Learning Games Network (http://www.
learninggamesnetwork.org).
Chapter 11
Embodied Education in Mixed
and Mediated Realties
Some Results and Principles for VR Content
Design

Mina C. Johnson-Glenberg

Abstract This chapter provides a summary of some of this lab’s immersive media
and embodied STEM learning research. It focuses on the integration of gesture in
learning, and a new gesture-based assessment. A taxonomy for embodiment in
education is included. The chapter concludes with several design principles that the
Embodied Games Lab has culled over the years while creating educational content
that maximizes the affordances of virtual and mixed reality technologies and
meshes those with best pedagogical practices.

Keywords Mixed reality Virtual reality Science education


Embodied learning Taxonomy for embodiment

11.1 Introduction

This chapter provides a summary of some of the research ongoing in embodiment

in education using both virtual and mixed realities, hereafter called immersive
media (see introductory chapter for this book). It ends with several design principles
that the Embodied Games Lab has culled over the years while creating optimal
embodied educational content for mixed and virtual reality platforms. The term
‘embodied’ can mean many different things, here it means that learners are moving
their bodies in a mediated environment in a manner that aids in comprehension; it
encompasses using representational gesture as well as constructing models with
virtual and real manipulables.

M.C. Johnson-Glenberg (&)

Department of Psychology, Arizona State University, 6695 South Rockford Drive, Tempe,
AZ 85283, USA
e-mail: [email protected]; [email protected]
URL: http://www.embodied-games.com

© Springer Nature Singapore Pte Ltd. 2017 193

D. Liu et al. (eds.), Virtual, Augmented, and Mixed Realities in Education,
Smart Computing and Intelligence, DOI 10.1007/978-981-10-5490-7_11
194 M.C. Johnson-Glenberg

Action and movement hold a special place for many educators. Maria Montessori
writes, “Movement, or physical activity, is thus an essential factor in intellectual
growth, which depends upon the impressions received from outside. Through
movement we come in contact with external reality, and it is through these contacts
that we eventually acquire even abstract ideas.” (Montessori, 1966). There are several
timely and important questions related to meshing embodiment with mediated edu-
cational content. Computer simulations have a long track record now of increasing the
learning of content. Rutten et al., (2012) writes, “In most cases simulation conditions
showed improved learning outcomes, with effect sizes up to 1.54.” Interactive games
and simulations of science phenomena are increasingly being used to supplement
education, designers need to know how to create optimal interactive content.
In our lab we often ask, how can we harness the affordances of virtual and mixed
reality to create embodied, constructivist content? As discussed in Schneider’s
chapter, best pedagogical practices may depend on the amount and type of
embodiment in a lesson. But, how do we determine just how “embodied” an edu-
cational simulation is? Without some sort of ranking or categorizational schema, it is
impossible to run experimental studies on the efficacy of embodiment and mediated
education. The technology is moving rapidly and our theories and design principles
need to keep pace. Early work on a taxonomies for VR and user interfaces run the
gamut from the technical (Coomans & Timmermans, 1997) to the philosophical
(Biocca, 1997). More recently, Lindgren and Johnson-Glenberg proposed six pre-
cepts for designing embodied content in mixed reality spaces (Lindgren &
Johnson-Glenberg, 2013). The field would benefit from a taxonomy that codified
embodiment in educational content, to that end, this section reviews embodiment
and concludes with a proposed taxonomy on embodiment in education.

11.1.1 Embodied Learning

Human cognition is deeply rooted in the body’s interactions with its physical
environment (Glenberg & Kaschak, 2002; Wilson, 2003), as discussed in Slater’s
chapter. Multiple research areas now support the tenet that embodiment is a
powerful underpinning of cognition. The various domains include (but are not
limited to): cognitive psychology (Barsalou, 2008; Glenberg, 2008), social psy-
chology (Niedenthal, Barsalou, Winkielman, Krauth-Gruber, & Ric, 2005), lin-
guistics (Lakoff & Johnson, 1980), mathematics (Lakoff & Nunez, 2000), gesture -
as it relates to learning and language (Goldin-Meadow, 2009; Hostetter & Alibali,
2008), gesture and math (Nathan et al., 2014); theater (Noice & Noice, 2006), and
even using dance to learn computer programming (Parmar et al., 2016).
As the above list suggests, theories of embodied cognition have implications for
a wide range of human activities and mental processes, and thus can be brought to
bear on the full spectrum of learning modalities from the refinement of motor skills
to cultivating creative expression to socio-emotional learning. An intriguing
demonstration of how cognition is intertwined with the actions of the body is found
11 Embodied Education in Mixed and Mediated Realties 195

in an fMRI study where participants listened to words related to various body areas
(“lick”, “pick”, and “kick”) and brain activation was observed in the sensorimotor
areas associated with performing those actions (Hauk, Johnsrude, & Pulvermüller,
2004). For example, reading “lick” activated motor and premotor areas associated
with the face and tongue, suggesting that sensori-motor areas are still important and
activated in the adult brain during language comprehension.
Also relevant are studies showing a direct effect of physical enactment on
cognitive processes. In the Self Performed Tasks (SPT) domain, Engelkamp and
colleagues compared participants who heard a list of unrelated action phrases with
participants who performed the actions. The consistent finding was that the
self-performing participants recalled more of the phrases than those who merely
heard the phrases (Engelkamp & Zimmer, 1994). There is increasing evidence that
body movement such as gesture can serve as a “cross-modal prime” to facilitate the
retrieval of mental or lexical items (Hostetter & Alibali, 2008). If physical move-
ment primes mental constructs such as language, then perhaps increasing an indi-
vidual’s repertoire of conceptually-grounded physical movements will provide
fertile ground for new knowledge structures to be developed.
It is on this premise, that adding a motor signal or trace will enhance learning,
that our view of embodied education is based. Much of the educational content in
western education is instructed using abstract symbols, namely the symbols of
language (words and syntax) and the symbols of mathematics. For these symbols to
be meaningful to learners they must be based in something outside of the system of
symbols themselves. Body perception and action, and experiences based on per-
ception and action, what Barsalou’s (2008) calls “perceptual symbols” provide a
mechanism for this grounding. It may be the case that when the appropriate sen-
sorimotor systems are engaged via action, then the converging inputs can create
stronger and more stable memory traces and learning is enhanced and retained for
longer.
Our lab focuses on using representational gestures, these are either captured with
external sensors, or with hand controls linked to the newest HMD’s (e.g., HTC
VIVE, Oculus TOUCH). But, there are also immersive realms of movement that can
be explored with Tangible User Interactions (TUI). Schneider (Chap. 12) describes
work using blocks that can be tracked with QR codes. The blocks can now rep-
resent any manipulable content and physically moving these blocks encourages
learners to be strategic. We would consider these to be meaningful gestures as well
and to prime spatial cognition, among other constructs.

11.1.2 The Taxonomy for Education in Embodiment

In some ways, the body is a primordial display device, a kind of internal mental simulator
(Biocca, 1997).

One of the driving goals of the author’s lab is to create educational content that
meshes the affordances of virtual and mixed reality technologies with best
196 M.C. Johnson-Glenberg

pedagogical practices to result in optimal learning. Educational content is never

simply embodied or not, there are probably degrees. Reading a text-only passage
that is visually evocative is embodied, albeit we would deem that experience as low
embodied. If perceptual symbols are always activated (Barsalou, 1999), then it is
problematic to state that some content evokes zero embodiment. As a field, we need
more methodical descriptors for the levels of embodiment in lessons.
To this end, a taxonomy with four degrees of embodiment for new media has
been proposed following a mathematical “weak ordering” system
(Johnson-Glenberg, Birchfield, Koziupa, & Tolentino, 2014a; Johnson-Glenberg,
Megowan-Romanowicz, Birchfield, & Savio-Ramos, 2016). The degrees depend
on three constructs that are not strictly orthogonal. These are: (a) amount of
sensori-motor engagement, (b) how congruent the gestures are to the content to be
learned, and (c) amount of immersion experienced by the user. The first two
constructs are not unrelated, because for a gesture to be congruent, there must be
some amount of sensori-motor engagement. Nonetheless, within these constructs
magnitudes can vary affecting the overall degree.
In the Taxonomy for Embodied Education, the continuous spectrum of
embodiment is binned into four degrees with the 4th being the highest. The anchor
points of the 4th degree, high in all constructs, and the 1st degree, low in all
constructs are well justified. The taxonomy represents an improvement beyond the
simplistic claim that educational content is either embodied or not.
The embodied degrees and some examples are explicated below.
4th degree = All three constructs are rated as being high. (1) Sensorimotor
engagement—A technological platform is available that can map (e.g., via motion
capture, etc.) the whole body, or multiple limbs. Therefore the body can act as the
controller and action on the display surface is linked to the users’ actions. If
locomotion is included then visual parallax is also engaged and this is important
(Campos et al., 2000) as it further increases sensori-motor activation. Multimodal
effects (e.g., auditory and haptic cues) are present and these increase sensorimotor
activation. (2) Gestural congruency—Within a lesson there are multiple instances
of gestures that drive the system, and those gestures are consistently designed to
map to the content being learned, e.g., spinning the arm makes a virtual gear spin
the same speed and same direction on the screen. This is congruent to and aids in
the learning of the educational goal. (3) Sense of immersion—This is a multi-
componential construct. Even the authors in this book use the term differently and
we have agreed to define the term where it differs from the reference in the
introductory chapter. In our lab, immersion is defined as the interactional outcome
of three components. First, the field of vision (FOV) that is covered; second, how
engaging the content this (this includes emotional engagement as well); and third,
the degree to which the user loses awareness of the real world (others use the word
presence). Slater (chapter in this book) and others have published extensively on
immersion and presence (Slater, Spanlang, & Corominas, 2010a; Slater & Wilbur,
1997), and several measures of immersion have been developed. In Slater’s lab
immersion is a property of the system, and he reserves the term presence for the
human experience. We conflate the two terms for ease of expression, and because
11 Embodied Education in Mixed and Mediated Realties 197

presence and immersion are often non-orthogonal. In one of our studies in this
chapter, we further simplify immersion by operationalizing it with only the first
component of FOV. Display areas vary from smart phones screens to wrap-around
360° Head Mounted Displays (HMDs) used in virtual reality. Displays that do not
have borders in the field of vision are considered higher in the immersion construct.
3rd degree = (1) Sensorimotor engagement—The whole body could be used as
the controller, but the user remains in one place (e.g., standing at an Interactive
Whiteboard). At least one large physical gesture (beyond finger movement) should
be present and linked to the content. (2) Gestural congruency—The system should
contain one or more instances a gesture that is well-mapped to the content.
(3) Sense of immersion—A large screen display or floor projection should induce
the learner to perceive the environment as immersive; however, borders are usually
present on the periphery.
2nd degree = (1) Sensorimotor engagement—Learner is generally seated, but
there is some upper body movement of the arm or fingers. (2) Gestural congruency—
this is probably not a defining construct in the lesson, although there is always some
interactivity (e.g., a finger swipe to advance, or a flick-wrist-forward action while
holding a smart phone to simulate casting a fishing reel), (3) Sense of immersion—
The display covers less than 50% of the field of vision; borders and real world are
always present no matter the fixation point (e.g., a 16 in. monitor, or tablet-sized
screen).
1st degree = (1) Sensorimotor engagement—Learner is generally seated, there is
some upper body movement, but usually just for a key press. The learner is pri-
marily observing a video/simulation. (2) Gestural congruency—Low. There is no
learning-related mapping between gesture and content, the users’ movements are
elicited primarily for navigation (e.g., tap for next screen). (3) Sense of immersion
—Low. The display covers far less than 50% of FOV and borders/real world are
always present.
The taxonomy results in eight configurations for the four degrees. Table 11.1
lists the degrees and magnitude of the three constructs binned into low and high.
There are two configurations in the 2nd and 3rd degrees that would be odd to
consciously create, but they are nonetheless possible, so they are included for
symmetry’s sake. Why design in a large movement, e.g., a jumping jack, and have
it map to a lesson that has nothing in common with the action, i.e., make the first
gear in a gear train spin to the right?

Table 11.1 Construct magnitude within degrees in the Embodied Education Taxonomy; H High,
L Low
Degree 4th 3rd 3rd 3rd 2nd 2nd 2nd 1st
Embodiment construct
Sensorimotor H H H L L L H L
Gestural congruency H H L H L H L L
Immersion H L H H H L L L
Cells in Bold It would be odd to require a large movement that was poorly mapped to the content
to be learned
198 M.C. Johnson-Glenberg

11.1.3 Construct 1—Sensori-Motor Activity

Being Active
It appears that doing the action helps people to learn. Recent work with younger
learners suggests there are neural differences when children are active versus pas-
sive during a learning experience. When 5- to 6-year-old children actively
manipulated an object while hearing a new label and then heard the label again,
motor areas of their brains were more likely to be activated upon subsequent
viewing compared with when they were only allowed to passively watch an
experimenter manipulate the named object (James & Swain, 2011). A compelling
example of passive versus active science learning comes from Kontra’s lab (Kontra,
Lyons, Fischer, & Beilock, 2015). Participants who physically held two bicycle
wheels spinning on an axle learned more about angular momentum compared to
those who observed a partner holding the wheels. Kontra et al. then used fMRI to
reveal that the action group did better than the observe group on knowledge tests,
and that the level of the BOLD signal in the brain motor regions of interest (left
M1/S1) significantly predicted test performance for both groups. They tout this as a
model that explains how physical experience, relative to observation, increases
“activation of the sensorimotor systems important for representing dynamic phys-
ical concepts.” (p. 6).
If Goldin-Meadow et al.’s postulation is correct that gesturing helps to off-load
cognition (Goldin-Meadow, 2011) and free up resources for learning, then perhaps
educational designers should consider methods of teaching science content that
make use of motoric components and gestures. If gestures help to free cognitive
resources, then we should see gains in learning when content is difficult and par-
ticipants are encouraged to gesture during encoding.

11.1.4 Construct 2—Gestural Congruency

STEM (Science, Technology, Engineering, and Math) topics may benefit from
being taught in an embodied manner using new media because many of the con-
cepts are abstract and difficult to grasp. However, the gestures need to be designed
to be congruent to the task learned (Kang & Tversky, 2016; Segal, 2011).
Congruent means that there is overlap between the representational gesture and the
construct being learned. Koch et al., (2011) report that participants react faster in a
Stroop condition using congruent gestures (up movement attached to word
“happy”) compared to incongruent gestures (down movement for “happy”) using a
large physical 28 in. slider (Koch, Glawe, & Holt, 2011). Glenberg and Kaschak
(2002) explore embodiment by varying the direction of button pushes to signal
sentence comprehension. They found that participants take longer to react when
there is a mismatch between the sentence meaning (‘close the drawer’) and the
button push direction (‘towards your body’).
11 Embodied Education in Mixed and Mediated Realties 199

Antle and others call these “body metaphors”; her Sound Maker mixed reality
system (Antle, Corness, & Droumeva, 2009) uses viable physical mappings for
volume, tempo, pitch and rhythm. For example, tempo was associated with speed of
movement through the room, pitch was associated with movement up and down in 3D
space. She counsels for “interactional mappings that preserve structural isomor-
phisms between lived experience and the target domain”, what we call congruency.
As an example in the science education and mixed reality domain, middle school
students move their bodies along a hypothesized asteroid trajectory (Lindgren,
Tscholl, Wang, & Johnson, 2016) to learn about gravity. They walk quickly
through the body-mapping platform as they are laser scanned and their movements
control a virtual asteroid. Participants who were in the embodied condition, i.e., the
whole-body congruent activity, showed significant learning gains, higher levels of
engagement, and more positive attitudes towards science than the control group
whose movements were not mapped.

11.1.5 Construct 3—Immersion

The third construct is immersion. Coomans and Timmerman (1997) state that
immersion is “…the feeling of being deeply engaged… (in) a make believe world
as if it was real.” (p. 279). Immersion is a term somewhat in flux in America and
needs to further operationalization for education. Coulter and colleagues (2007)
showed that students who learned how to treat a head trauma victim via a pro-
prietary virtual HMD (which they deemed the “full immersion” condition) showed
significantly better learning than students who learned on a laptop monitor (deemed
the “partial immersion” condition). In the paper, it is not explained why those
condition definitions were chosen. A meta-analysis by Miller and Bugnariu (2016)
showed that for high-immersion virtual environments, treatment response was
overwhelmingly positive for those with autism spectrum disorder who were
learning social skills. However, in a small n correlational study run by Bailenson’s
group (2012), participants who learned multiple environmental messages while in a
VR shower recalled less content. Participants filled out a five item Physical
Presence questionnaire and those with higher presence scores, recalled less content
on a cued recall task. This significant negative correlation had not been predicted,
and the authors speculate that after a highly vivid sensory experience, participants
may have had limited cognitive resources left over to dedicate to the memory task.
Thus, the idea that an immersive environment will indiscriminately enhance
learning has not been fully supported. We do not know if the study included many
high gestural congruency mappings to the eco-messages to be remembered. The
issue of seductive details may take on more importance as environments become
more immersive. For education, less may be more.
When immersion, as it relates to learning, is rigorously operationalized we will
be better able to weave it into our content and assess its unique effects on learning.
The most thoughtful definition so far, using the system level analysis, comes from
200 M.C. Johnson-Glenberg

Slater and Wilbur (1997). The immersive capability of an environment depends on

the following five components: inclusive, extensive, surrounding, vivid,
and matching (Slater & Wilbur, 1997). Inclusive refers to whether signals per-
taining to the physical world have been eliminated (e.g., joystick, weight of
wearables, etc.). Extensive refers to the number of sensory modalities that are part
of the experience. Surrounding refers to the visual presentation including field of
vision (FOV) and the degree to which the physical world is shut out. Vivid refers to
the fidelity and resolution of the simulation. Matching refers to whether the
viewpoint of the environment is modified to match the user’s perspective (e.g., in an
HMD when the user moves left, the environment moves as well). Issues associated
with vividness and matching have been mitigated the past several years. The
majority of researchers use surveys but we also note that physiomarkers have been
used (i.e., heart rate, skin conductance, pupil dilation, etc.). As an example of a
bio-marker, Slater et al. (2010b) use heart rate deceleration and show that decel-
eration significantly correlates with key emotional experiences reported on a post
evaluation survey. Immersion is complex, and may not be well-measured by one
metric because it is not one dimensional. At any point users may feel present in “…
one of three places: the physical environment, the virtual environment, or the
imaginal environment… Presence oscillates among these three poles”, Biocca
(1997).
In the next two sections, two studies are described that controlled for amount of
embodiment (which includes the construct of immersion). In the first, a virtual
learning environment is used, and in the second, a mixed reality environment is
used. The chapter ends with several best practices for design.

11.2 Virtual Learning Environment Study

11.2.1 Electric Field—Embodiment of Abstract Content

It has become increasingly cost effective to use sensors like the Microsoft Kinect to
map movements and then drive simulations with users’ body actions. As designers,
we want to understand how adding gestures and movement affect science learning.
We make the comparison in this study between a low embodied condition—
watching science simulations on a larger projection surface—to a high embodied
condition where learners use gestures to actively construct models on the same
sized projection surface. We varied the amount of embodied content, the amount of
active generation of the gestures, and whether there was a narrative story line that
held together the seven simulations. We do not address the null results of the
narrative condition here, but those results can be read elsewhere (Johnson-Glenberg
& Megowan-Romanowicz, 2017).
The study was a between subjects 2  4 design. The first factor was time with a
pretest and immediate posttest; the second factor was condition with four levels
11 Embodied Education in Mixed and Mediated Realties 201

described in more depth below. In addition, two types of tests were administered,
the first was a more verbal assessment that used a keyboard for input, and the
second was an innovative gesture-based assessment, called the Ges-Test, that we
hypothesized would be more sensitive to revealing learning gains in the embodied
conditions. All participants were exposed to the same learning “text cards”, written
in a low embodied expository style (no anthropomorphization or highly evocative
language).
The Manipulated Conditions.
1) Symbols and Text (S&T)—In between the text card sections, the control S&T
group answered quiz questions that included only text and symbols for equations
and questions. Participants read the short multiple choice text-only questions
that appeared after each content section. After each text section there were four
multiple choice questions designed to reinforce what had just been read and to
equate for time between conditions. Thus, no graphics nor simulations were
seen or acted upon between sections.
2) Low Embodied—In the Low Embodied condition, participants watched ani-
mations or simulations that were pre-created (similar to viewing a video). The
participants could start the animations but they did not actively control the
action within the animations.
3) High Embodied—The final two conditions (3 & 4) are called High Embodied. In
condition 3, the Microsoft Kinect sensor mapped key body joints and motion.
After the instructional text sections, participants were able to physically interact
with the seven simulations. As an example, in the simulation called Scuff-n-
Spark participants would actually scuff their feet up and down a section of carpet
and “pick up electrons”. The rate that participants moved their knees drove the
simulation on screen. The more electrons they accumulated, the more blue dots
showed up the bottom of the screen in the Scuffometer (see Fig. 11.1) and the
higher the qNet on the virtual hand rose. When participants decided the qNet was
high enough they could reach forward with their real hand and the virtual hand
on-screen would move toward the door knob and result in a spark. The dynamic
relationship of Coulomb’s Law stayed on screen (upper left corner). Bodily
actions altered the size of the symbols in the equation in real-time (e.g., q1 was
mapped to the charge build-up of negative electrons). This is a powerful
instance of embodiment with symbols and instantiates representational fluency.
4) High-EMB-Narrative—The 4th condition was the same as the 3rd except that
seven graphic narrative cut scenes were inserted before the simulations. A cut
scene is a comic-style graphic with text bubbles that appeared and faded.
Videos and games can be downloaded at https://www.embodied-games.com/
games/all/electric-fields or https://www.youtube.com/watch?v=eap7vQbMbWQ.
202 M.C. Johnson-Glenberg

Fig. 11.1 The Scuff-o-meter screen shot with the dynamic equation in the upper left corner

11.2.2 Embodied Assessments

Two assessments were administered at pretest and posttest, they were invariant.
First, a content knowledge pretest was taken on a computer. This was a more
traditional non-gesture based assessment using the keyboard as the input device.
Second, the Ges-test was taken on the Wacom™ Intuous Pro (15.1 in. or 38.4 cm
active diagonal). All participants confirmed they had never used a Wacom before.
This is essentially a large tracking pad with high accuracy for touch. The Wacom
was placed on the table beneath the 16 in. diagonal testing computer monitor. To
keep the assessment as haptic and embodied as possible the stylus was not used,
instead participants drew with a fingertip on the Wacom surface.
The mechanic was a double tap that would signal ‘start tracking’ and a drag
across the screen would result in a line or vector onscreen. Figure 11.2 shows a
student who drew towards the right and slowed down with time. As the finger
moved across the Wacom, users saw a colored line trailing behind a larger circle
that represented the touch. Every 100 ms white dots were placed inside the colored
line.
The placement of the white dots is a visual cue for speed, similar to the motion
map concept used in Modeling Instruction (Hestenes, 1987). Users should be able
to feel if they are accelerating, but the visual feedback of the white dots as a motion
map also allowed users to see that when the finger moves faster the dots spread
further apart. The dots get closer together as the user is slowing down before
stopping on the far right.
11 Embodied Education in Mixed and Mediated Realties 203

Fig. 11.2 Screenshot of the Wacom Ges-Test, the finger is used to draw charge movements and
vectors. The white dots represent finger location similar to a motion map

11.2.3 Results for Electric Field Study

Participants (N = 134) were randomly assigned to condition. We are interested in

the difference between the average of group 1 and 2 compared to groups 3 and 4.
That is, the split between the low embodied groups and the two high embodied
active groups. All four groups were equated on the Ges-Test at pretest. A linear
regression analysis revealed a significant active embodiment effect on the Ges-Test,
F(1, 132) = 3.77, p < 0.05. The low embodied groups gained 2.03 (SD 7.03) and the
high embodied groups gained 4.79 (SD 8.36), for a moderate Effect Size (Cohen’s
d) of 0.35.
Interestingly, the traditional keyboard-based knowledge test did not reveal a low
versus high embodied effect. The traditional keyboard test revealed only that the
three embodied groups (i.e., the low embodied, high embodied and high embod-
ied + narrative) performed significantly better than the control symbols and text
group, F (1, 164) = 4.23, p < 0.04, Effect Size (Cohen’s d) = 0.38.

11.2.4 Conclusions for Electric Field

With motion capture technology becoming more cost effective and entering edu-
cation, and virtual/mixed realities meshing with the embodiment that motion cap-
ture enables, it is more timely than ever for researchers and educational content
designers to use a codified language. As a field, we are in need of studies that
explicate the most efficacious components in embodied science lessons, and we
204 M.C. Johnson-Glenberg

need a method for discussing the degrees of embodiment. For the four conditions in
this study, the conditions map to the degrees in the following manner:
Control, symbols and text = 1st, the low embodied condition (observe simula-
tions) = 2nd, the high embodied and high embodied/narrative (gestures construct
simulations) conditions = 4th. The study above highlights that being active and
embodied benefits learning, in addition, we must also be creative about the type of
metrics to assess learning.
New immersive media need more procedural and embodied assessment mea-
sures. The chapter by Shute (Chap. 5) describes creative methods for assessing
learning during gameplay and active simulations. For the electric field study, a new
gesture-based metric was designed and it proved to be more sensitive to the learning
gains associated with motion and vectors. When knowledge was assessed via the
larger tablet format that facilitated gestures, the two active high embodied groups
scored significantly higher on knowledge gains. This suggests that more sensitive
and embodied metrics should be developed that also assess knowledge that is
gained in a more embodied manner.

11.3 Second Example Study—Mixed Reality

with SMALLab

The second study compares a mixed reality environment with two of the more
common learning platforms found in schools. The six counterbalanced lessons were
designed to be either high or low in embodiment and to maximize the affordances
of each platform. A full description of this study can be found in (Johnson-Glenberg
et al., 2016) in Frontiers in Psychology. To ask the question of how learning gains
are affected by platform crossed with embodiment, three different platforms were
selected. The first was SMALLab which stands for Situated Multimedia Arts
Learning Lab. With an overhead projector, one dozen Optitrack motion-capture
infrared cameras, and a very large projected floor display (21 foot or 252-in.
diagonal) a highly immersive learning platform was created. Figure 11.3 shows a
learner surrounded by the graphical “motion map” feedback she created during the
centripetal force lesson.
All lessons also included auditory feedback, the pitch increased as speed
increased on the tethered objects. The second condition used the Promethean
ActivBoard, an Interactive Whiteboard (IWB-78 in. diagonal) as the learning
platform. The third condition used a traditional desktop computer with a monitor
and mouse set-up (16 in. diagonal). The hypothesis was that the high embodied, 4th
degree SMALLab platform which encouraged larger sensori-motor activation
(participants could physically spin objects and locomote) and included a greater
sense of immersion (defined by FOV) would result in better learning and greater
delayed learning gains.
11 Embodied Education in Mixed and Mediated Realties 205

Fig. 11.3 SMALLab

interface with learner in
Centripetal Force simulation

While designing the pedagogy of the lessons, we were careful to use instances of
implicit scaffolding to introduce each component of the centripetal force equation
(FC = mv2/r) to the learners. This is also a technique advocated by the designers of
the PhET simulations (Johnson-Glenberg et al., 2014c). In addition, interface
components were added one step at a time after the learner demonstrated mastery.
E.g., each bar graph (first speed, then force) was introduced alone and participants
had to verbalize the purpose of the component before the experimenter added
another.
One of the greatest differences between the low and high embodied conditions in
this study was that we varied gestural congruency. In the high embodied condition,
SMALLab participants swung a tethered bob overhead and they could feel cen-
tripetal force as it acted on the arm and body; this provided a sense of authenticity,
as discussed in Jacobson’s Chap. 3. This is highly congruent to the concept to be
learned, i.e., velocity is directly proportional the amount of centripetal force. In the
low embodied conditions, the virtual bob that was spun was not “well mapped”
because a horizontal slider was used to increase or decrease the speed of the bob.
Again, in the high embodied conditions, the physical act of spinning the bob
overhead or using an interactive tracking pen on the IWB, or spinning the mouse in
the desktop condition would be considered high embodied because it corresponded
directly to and drove the speed of the virtual bob. In juxtaposition, in the low
embodied conditions, the speed of the bob was driven by lateral placement of a
virtual slider—left for slower, to the right for faster. To see a video to help
206 M.C. Johnson-Glenberg

conceptualize, please visit www.embodied-games.com/games/all/centripetal-force

or https://www.youtube.com/watch?v=oFiXtcXRpVE.
Tangibles constructed for the MR platform. One of the exciting aspects of
working in mixed reality environments is that it is possible to construct manipu-
lables that interface between physical gesture and the virtual displays. For the
SMALLab high embodied condition, two new manipulables were constructed. The
“swinger” had two components: a plastic handle that connected to string and in the
end was a tracked ball (the bob) in which mass could vary. The “flinger” was a
dually tracked manipulable that was used to assess the learning of trajectory at point
of release. (A common misconception that many novices believe is that an object
released from circular motion will continue to travel on a circular path, this is called
the impetus misconception.) The flinger is shown in Fig. 11.4. The brass lever on
the top of the handle serves as the release mechanism.
Participants physically spun their entire bodies around in a circle holding the
flinger in front. They then released the bob at a time of their choosing to hit a target
on the floor. That is, the large tennis ball (the ‘bob’ part) of the unit would dis-
engage when the brass lever was pressed, and the ball would fly from the tracked
handle. The smaller tracked spheres on both components are covered with
retro-reflective tape and this allows the IR cameras to map the positions of both the
handle and the moving bob. Participants are able to observe how the bob flies at the
point of release. They begin to address the impetus misconception regarding tra-
jectory at point of release. See Fig. 11.1 for an image of the learner who missed the
target, but sees the mediated feedback on the floor projection regarding the path

Fig. 11.4 “The Flinger”

constructed for SMALLab, the
smaller spheres are mapped in
realtime by the 12 IR cameras
in the ceiling
11 Embodied Education in Mixed and Mediated Realties 207

after release. The bob flew in a straight line (the curled line represents the hand-held
part that continued to spin with the participant). This is another fine instance of
representational fluency.
Low embodied conditions. In the low embodied SMALLab condition, partici-
pants used a different rigid-body trackable object. This was an extant 3D-printed
manipulable/wand. Participants in this condition used the wand to control the vir-
tual slider projected on the floor and to signal the release from spin via raising the
wand in the Z axis. In all three low embodied conditions (including the IWB and
desktop), the participant controlled speed of spin with a horizontal virtual slider.

11.3.1 Results SMALLab and Centripetal Force

The 109 college-age participants were randomly assigned to one of six 50 min-long
lessons. These experimental conditions were derived from two factors. The first
factor, amount of embodiment had two levels: (1) low and (2) high and the second
factor of platform had three levels: (1) a mixed reality immersive environment
SMALLab, (2) an interactive whiteboard system, and (3) a mouse-driven desktop
computer. The levels were crossed resulting in six lessons. Pretests, posttests, and
one-week followup (retention) tests were administered resulting in a 2  3  3
design. In this study we also gave two different types of knowledge subtests, one
that relied on more declarative knowledge and was administered on a keyboard.
The second subtest relied on more generative and gesture-based knowledge, e.g.,
hand-drawing vectors.
Regardless of experimental condition, ALL participants made significant
immediate learning gains from pretest to posttest, F(1, 99) = 459.89, p < 0.001.
There were no significant main effects or interactions due to platform or embodi-
ment on immediate learning. One week after posttest, 63% of participants returned
and took both subtests again. From posttest to followup the level of embodiment
interacted significantly with time, such that participants in the high embodiment
conditions performed better on the subtest devoted to generative knowledge
questions. On the generative subtest only, there was a significant interaction
between level of embodiment and posttest to follow-up (i.e., delayed learning
gains), F(1, 62) = 4.83, p < 0.03. Platform was not predictive of either immediate or
retention scores.

11.3.2 Conclusions for Centripetal Force

We posit that better retention of certain types of knowledge can be seen over time
when more embodiment is present during the encoding phase. This sort of retention
may not appear on more traditional factual/declarative tests.
The retention effect was only seen on the test that was more sensitive to embodi-
ment, i.e., the generative test where participants had to draw vectors to show the path at
208 M.C. Johnson-Glenberg

the point of release. Some readers might consider all three of these platforms to be
virtual reality, including the one on the typical computer monitor, but we would call
that a virtual environment and not virtual reality. Using the definition in Chap. 1 (this
book), only SMALLab, the mixed reality space with the large FOV approached virtual
reality. The fact that all conditions made immediate significant learning gains
demonstrates that all six were well designed lessons, even the low embodied ones. In
addition, a decision was made that we did not want students to leave the study with
incorrect mental models. Thus, when participants answered a prompt incorrectly (e.g.,
replying that “a longer string would result in more centripetal force”), participants
were asked to run through the task again and to answer the question again. If they
made the same incorrect conclusion three times in a row, the experimenter explicitly
supplied the correct answer. This corrective guidance assured that the knowledge
needed to show competency on the posttest was voiced at least one time. It is still
worth noting that no one scored 100% on the posttest.
The conditions differed primarily in the amount of kinesthetics and gestural
congruency. Cook, Mitchell, & Goldin-Meadow (2008) report that “gesturing
makes learning last” in the domain of learning a new mathematical concept. We
saw that the condition with the most gesturing and movement via whole body (high
embodied SMALLab) was indeed the condition in which the learning persevered
more robustly. This may be due to the multiple instances of gestural congruency
during encoding and because the high embodied condition elicited more sensori-
motor activity. A greater amount of physical movement should activate complex
motor neuron patterns and these will be associated with the learning signal. In
addition, motor planning recruits resources that have downstream effects that may
affect delayed learning gains. The delayed retention was significantly different
between embodied conditions. We have seen similar delayed results on nutrition
knowledge tests when comparing low and high embodied learning conditions in an
exergame (Johnson-Glenberg, Savio-Ramos, & Henry).
Take Home Message
The take home message from the two studies is that mediated immersive content is
a powerful instructional tool and that each delivery platform comes with its own set
of affordances. Ultimately, what is done with the affordances and the amount of
embodiment included in the lesson may prove to be the strongest predictors of
learning gains and may affect the magnitude of the content retained. The new
handheld controllers with commercial VR packages will make it much easier to
design representional gestures into lessons. It is crucial that the educational design
communities begin to understand and then disseminate principles for design.

11.4 Principles of Design

The chapter ends with a set of design principles that has been gathered and refined
over the 25 years that the author has been designing educational content and
assessment measures. Other chapters in this book describe good and poor uses for
virtual reality (e.g., see Jacobson, Chap. 3). This list begins with content design
principles and then moves on to some guidance for assessment measures:
11 Embodied Education in Mixed and Mediated Realties 209

Be creative with affordances of the technology—For the centripetal force

study, we chose three very different platforms and then spent multiple months
brainstorming how to maximize embodiment in each one. The sessions designing
for the high embodied mixed reality SMALLab condition were especially creative
and engaging because all virtual and physical phenomena were “on the table”. We
practiced physically running around poles with ropes tied to our waists (“you be the
bob!”). We ran in circles with backpacks filled with rocks to increase mass. In the
end, we chose to mimic the classic force concept test item from the CFI inventory
and used the model of spinning a tethered object overhead. Next we had to design
for swinging/spinning motions for the whiteboard which was a vertical surface;
here we were able to engage the shoulder and arm with representational gestures.
Finally, we designed for the horizontal, mouse-driven interface of the
desktop. Spinning the mouse in a circle on the table best approximated the agency
used in the other spin tasks. For each platform the team worked to maximize the
amount of sensori-motor output that could mapped to gesturally congruent
movements.
Being embodied and active—The user should move. The amount of
sensori-motor activation is hypothesized to be predictive of whether the content is
highly embodied or not. A greater amount of physical movement should activate
complex motor neuron patterns and these will be associated with the learning
signal. Cook, Mitchell, and Goldin-Meadow (2008) hypothesize that the significant
delay test results seen in their gesture and gesture/speech groups may be because,
“…expressing information in gesture may produce stronger and more robust
memory traces than expressing information in speech because of the larger motor
movement”. We are now designing for HMD’s that use wireless hand controls, and
believe it is important that meaningful hand gestures be part of the lesson. At the
same time, being in a fully enclosed HMD experience, where the real world is not
viewed, can be dangerous and disorienting and does not lend itself to sustained
locomotion. The cost of adding synchronized co-location to HMD experiences will
continue to drop, but for the next several years it will still be expensive to get
multiple live players in a VR space. If face to face discourse is one of your goals,
you may want to spend more time in the AR and Mixed reality worlds.
It is not necessary to install an expensive CAVE to be able to move the entire
body and openly discourse with others in sightline. Pokemon Go is an augmented
example that encourages locomotion on a ubiquitous device. Klopfer (Chap. 10)
has been a leader in designing for augmented education.
When designing activity into learning games, we have often used a regular
projector and the inexpensive Kinect sensor. For example, Alien Health is a
co-located, dyadic nutrition instruction game that incorporates gesture and virtual
reality components. In the game, players would swipe healthy food choices into an
alien avatar’s mouth. The hypothesis was that the embodied gesture primed the
action of choosing and eating healthier foods in a forced choice task. Players were
then prompted to perform several exercises to “help the alien metabolize the foods”
(Johnson-Glenberg et al., 2014c). Participants in the exercise condition performed
better on a post intervention nutrition test. Again, our research suggests that the
210 M.C. Johnson-Glenberg

platform is not what is critical, more important may be the amount of embodiment
in the lesson.
You may be forced to design for a platform becoming more popular in schools,
e.g., the tablet. How might you make that lesson more embodied? If you were
teaching centripetal force, perhaps users could hold the tablet in outstretched arms
and spin while the accelerometer provided feedback. That would be engaging! On
the other hand, the tablet could easily be dropped. A safer design might be to allow
the user to spin a virtual bob around with the fingertip on a touch surface. That is
active, gesturally congruent, and gives the user agency.
Give a sense of agency—Giving a learner “agency” is a direct consequence of
allowing them to be more active. Agency in the social sciences means that someone
has taken a deliberate action and that forethought was involved. In the mediated
education domain, we extend the definition to mean that the learner controls what
happens on the display device, which in turn affects the feedback received and
future choices. Choices and the activity on screen are not all pre-programmed and
preconceived, i.e., similar to watching a video. As an example, in Scuff’n’Spark, the
learners are in control of how many electrons accumulate on screen because they
control the rate at which they lift their knees. Agency, like embodiment, probably
comes in degrees.
One of the highest degrees of agency would surely be when users can set the start
state for a simulation with parameters of their own choosing. Creating a system that
accepts user-created content is not trivial. A guided user interface (GUI) must be
embedded in the system. That always costs extra funds, but it may well be worth the
expense. As an example, in the gears game referenced earlier, an interface was created
where students and teachers could enter their own coordinates to build a series of hill
heights (slopes) that their peers would then race up on the multi-geared virtual
bicycles (Johnson-Glenberg, Birchfield, Megowan-Romanowicz, & Snow, 2015). It
was a powerful method for combining the construct of physical effort (spinning the
arm around the shoulder to move the virtual bike) and instantiating the concept of
mechanical advantage with gear diameters. See Fig. 11.5 with a dyad playing the
Kinect-driven game called Tour de Force. The player with the top bike is ascending a
hill with a steeper slope: https://www.youtube.com/watch?v=kSsiJZOUKt4.
Be congruent—When picking the representational gesture that will give the
learner agency over the content, think through how congruent the movement is to
the construct to be learned. A hand push forward is not congruent to learning about
circular motion. Mapping the arm as it is spun with a certain velocity is congruent.
In the biking and gears game in Fig. 11.5, the speed and direction of the arm spin
maps directly to the speed and direction of the primary gear turning.
Scaffold components and complexity—As you choose learning objects and
their placement-both in the lesson and on screen, be sensitive to the amount of
content on display. Giving users control over placement of content can quickly lead
to cognitive overload. There is a finite amount of resources learners bring to any
multimedia task (Mayer, 2009; Sweller, van Merrienboer, & Paas, 1998). In our
embodied world, we also expect the learner to master both the kinesthetic and
cognitive requirements of the lesson. The learners must acclimate to a gesture to
11 Embodied Education in Mixed and Mediated Realties 211

Fig. 11.5 The gears Tour de Force game, players spin their arms to drive the bicycles up the hills

control content on the display, as well as adroitly integrate new knowledge into
their pre-existing knowledge structures. If a learner needs to overturn a miscon-
ception or incorrect p-Prim (diSessa, 1988) this will require even more effort. The
timing of the components revealed in the lesson must be appropriate to the com-
plexity of the content (while accommodating varying levels of prior knowledge). In
both experiments described, we were careful to slowly build on foundational
knowledge. The electric field series begins with a simulation to count the number of
electrons in an atom, moves on to vectors and sparks, and ends with the final game
on a lightning strike. The important point is that when learners are in augmented or
virtual environments there are many components vying for attention (Dunleavy &
Dede, 2014; Squire & Klopfer, 2007; Wu, Lee, Chang, & Liang, 2013). We need to
design sensitively and that may mean stripping down the interface for the first few
lessons, or beginning with lower level content as a refresher before revealing the
novel content. As Abrahamson and Lindgren (2014) advise, the
“action-environment couplings should be gradually introduced”.
Encourage collaborative interaction—The education field does not yet have a
cost-effective platform for allowing multiple users to interact in realtime via HMDs,
that day will come though. Until then, we continue to focus on some of the unique
affordances of augmented devices that allow learners to be very collaborative.
Social aspects are important in immersion as discussed in Kraemer’s chapter. One
can also use these devices for jigsaw pedagogy and differentiated role play as
inquiry-based activities (Dunleavy & Dede, 2014; Squire & Klopfer, 2007);
examples are provided in Dede’s and in Klopfer’s chapters. With handheld devices
several learners can huddle around a device and discourse in a natural style. Per
Lindgren and Johnson (2014), designers should be striving to make VR and
212 M.C. Johnson-Glenberg

immersive education more collaborative. SMALLab’s very large floor projection

allows for four learners to be active in the center at once. When SMALLab is in a
school, the rest of the class (those not being tracked in the center) sit around the
perimeter and takes notes, or make predictions, etc. They are kept active and
engaged with relevant tasks. There are benefits to co-located mixed reality envi-
ronments that are not readily apparent with the other platforms.
Be error friendly—Having agency means the learner will also have opportu-
nities to fail. Best to embrace failure. When participants self-constructed and then
ran their models in the final two conditions in the electric field study, they were able
to get it wrong. When that happened learners received immediate feedback. After
two failures, hints were supplied. We speculate there is something special and more
salient about learners creating their own errors (compared watching the animated
errors in the different condition). Best design practices for creating games often
include planning for low stakes failure and when players show they understand a
concept, they are then moved up in levels of difficulty and expertise (Gee, 2007).
Design in opportunities for reflection—Please don’t forget to put in a place for
reflection. We have found ourselves over-designing with bells and whistles and
non-stop action. This may keep eyes on the screen, but it is not always optimal for
comprehension. We now try to build in space where the learner can pause and
reflect on what was learned. A classic technique is to have the learner stop and
speak to a peer about how they understand the content, or perhaps learners could be
prompted to type up a paragraph on how this new knowledge might relate to their
everyday lives.
The Assessment of Learning
Learning gains may show up in unexpected ways—The author comes from a
somewhat traditional experimental psychology training and has too often wielded
the clunky assessment brush of quantitative assessments. The picture will always
remain incomplete if only one type of assessment is used. My best current advice is
to assume that learning gains associated with embodied content will appear in
unexpected places. The effects may show up as retention effects one week later
(Johnson-Glenberg et al., 2016). Perhaps this is because sleep helps to consolidate
the two separate verbal and motor signals as delayed gains. What happens to
memories during sleep is an on-going area of research (Stickgold & Walker, 2007).
The learning gains may never appear if the assessment is not sensitive to them. The
measure should perhaps query for knowledge in a manner isomorphic to how the
knowledge was encoded—called encoding specificity (Tulving & Thomson, 1973).
If the goal is to distinguish the learning difference between low and high embodied
conditions, then you may need to design an assessment that incorporates repre-
sentational gestures (similar to our Wacom Ges-Test) (Johnson-Glenberg &
Megowan-Romanowicz, 2017). Bayesian analyses (Shute, Chap. 5) are well suited
to in-process analyses and non-traditional assessments like learner diaries and the
studio books used by Shaffer et al. (2009) hold promise.
In-game assessments—We have also experimented with in-game or in–process
assessments. The premise is that learning in the game should be designed so that the
11 Embodied Education in Mixed and Mediated Realties 213

player cannot move up a level until the current level has been mastered. If a learner
is not understanding the concept, as evidenced by in-game decisions, then the
learner remains at that level which serves as a proxy to current skill state. For
example, in the Alien Health nutrition exer-game (Johnson-Glenberg, et al., 2014c),
players must deduce that feeding the alien too much high fat food will make him
fall asleep. Players are motivated to learn the rules and move up in levels because it
is much more fun to pilot the ship with gestures, than to watch the alien snooze.
Players are doomed to repeat the same levels over and over again until they feed the
alien the better food choices. We find this to be a more naturalistic way for learners
to demonstrate they understand a concept compared to giving them standard, end of
the unit surveys or multiple-choice tests that “drop from the sky”.
If content is embodied, make assessment match—This has been covered
elsewhere, but deserves its own line.
Don’t stop at lab studies—This is a point that was made in the Lindgren and
Johnson-Glenberg article (2013) called Emboldened by Embodiment, and it bears
repeating. Many great ideas start in the laboratory. However, for the learning sci-
ences to evolve we need field work. Education should be effective for the masses,
and new ideas and technologies should not only be assessed for efficacy in the
laboratory. Content delivery and assessment measures need to be tested in the
trenches as well. As designers and end-use researchers of content, we find it jarring
that so many of the “great ideas” on paper or in our small homogenous-group lab
studies do not play out well in the messy field. Nonetheless, we forge ahead, make
iterative changes in the designs and teacher prep materials, and the process carries
on. This is also a good place to mention that teachers will need specific instructions
and training on how to use the emerging immersive VR and MT technologies in
their classes. Efforts should be expended to create quality teacher preparation
materials.
Above we list some meta-level design principles. Slowly, we see that more
fine-grained design principles for VR are beginning to emerge in journals and
proceedings, e.g., keep locators on horizon, do not allow too many objects to fly
TOWARDS you, be aware if you add a virtual nose or any body parts that skin tone
should match the users’ (unless your goal is one of altering IAT or empathy—see
Slater chapter). It is positive that these tips are being learned and disseminated and
it is an exciting time to be in mediated educational technology.

11.5 Conclusions

This chapter attempts to address several timely and important issues related to
creating mediated embodied educational content. One important issue is that we
need to be using the same language to describe embodiment and we should be more
precise regarding the degree of embodiment in an educational scenario. To that end,
a taxonomy for embodiment in education has been proposed with four degrees of
magnitude based on the presence of three constructs: (a) amount of sensori-motor
214 M.C. Johnson-Glenberg

engagement, (b) how congruent the gestures are to the content to be learned, and
(c) amount of immersion experienced by the user.
The chapter ends with eleven principles:
The Creation of the Content
Be embodied, be active
Give a sense of agency
Be gesturally congruent
Scaffold components and complexity
Encourage collaborative interaction
Be error friendly
Design in opportunities for reflection
The assessment of Learning
Be flexible, learning gains may show up in unexpected ways
Embed in-game assessments
If content is embodied, make assessment match
Don’t stop at lab studies.

Acknowledgements The author wishes to thank David Birchfield, Colleen

Megowan-Romanowicz, James Comstock, and Hue Henry. The majority of funding for the projects
mentioned came from the NSF DRK-12 grant 1020367. Alien Health was funded by the ASU Obesity
Solutions grant. All views in this chapter are those of the author.

References

Abrahamson, D., & Lindgren, R. (2014). Embodiment and embodied design. In R. K. Sawyer
(Ed.), The Cambridge handbook of the learning sciences (2nd edn., pp. 358–376). Cambridge,
UK: Cambridge University Press.
Antle, A. A., Corness, G., & Droumeva, M. (2009). What the body knows: Exploring the benefits
of embodied metaphors in hybrid physical digital environments. Journal of Interactive
Computing, 21(1–2), 66–75.
Barsalou, L. W. (1999). Perceptual symbol systems. Behavioral and Brain Sciences, 22, 577–660.
Barsalou, L. W. (2008). Grounded cognition. Annual Review of Psychology, 59, 617–645.
Biocca, F. (1997). The Cyborg’s Dilemma: Progressive embodiment in virtual environments.
Journal of Computer-Mediated Communication, 3. doi:10.1111/j.1083-6101.1997.tb00070.x.
Campos, J. J., Anderson, D. I., Barbu-Roth, M. A., Hubbard, E. M., Hertenstein, M. J., &
Witherington, D. (2000). Travel broadens the mind. Infancy, 1(2), 149–219. doi:10.1207/
S15327078IN0102_1.
Cook, S. W., Mitchell, Z., & Goldin-Meadow, S. (2008). Gesturing makes learning last.
Cognition, 106(2), 1047–1058.
Coomans, M. K. D., & Timmermans, H. J. P. (1997). Towards a taxonomy of virtual reality user
interfaces. In IEEE Computer Society 17th International Conference on Information
Visualisation, IV, 279. doi:http://doi.ieeecomputersociety.org/10.1109/IV.1997.626531.
Coulter, R., Saland, L., Caudell, T. P., Goldsmith, T. E., & Alverson, D. C. (2007). The effect of
degree of immersion upon learning performance in virtual reality simulations for medical
11 Embodied Education in Mixed and Mediated Realties 215

education. Medicine Meets Virtual Reality 15: In Vivo, in Vitro, in Silico: Designing the Next in
Medicine, 125, 15.
diSessa, A. (1988). Knowledge in pieces. In G. Forman & P. B. Pufall (Eds.), Constructivism in
the computer age. Hillsdale, NJ: Lawrence Erlbaum Associates.
Dunleavy, M., & Dede, C. (2014). Augmented reality teaching and learning. In M. D. M. J. M.
Spector, J. Elen, & M. J. Bishop (Eds.), (Ed.), The Handbook of Research for Educational
Communications and Technology. (4th edn., p. 31). New York: Springer.
Engelkamp, J., & Zimmer, H. D. (1994). Motor similarity in subject-performed tasks.
Psychological Research-Psychologische Forschung, 57(1), 47–53.
Gee, J. P. (2007). Good video games and good learning: Collected essays on video games,
learning and literacy. New York: Peter Lang Publishing.
Glenberg, A. M. (2008). Embodiment for education. In P. Calvo, & A. Gomila (Eds.), Handbook
of Cognitive Science: An Embodied Approach (pp. 355–372). Amsterdam: Elsevier.
Glenberg, A. M., & Kaschak, M. P. (2002). Grounding language in action. Psychonomic Bulletin
& Review, 9(3), 558–565.
Goldin-Meadow, S. (2009). How gesture promotes learning throughout childhood. Child
Development Perspectives, 3, 106–111.
Goldin-Meadow, S. (2011). Learning through gesture. WIREs Cognitive Science, 2, 595–607.
doi:10.1002/wcs.132.
Hauk, O., Johnsrude, I., & Pulvermüller, F. (2004). Somatotopic representation of action words in
human motor and premotor cortex. Neuron, 41(2), 301–307. doi:10.1016/S0896-6273(03)
00838-9.
Hestenes, D. (1987). Toward a modeling theory of physics instruction. American Journal of
Physics, 55(5), 440–454.
Hostetter, A. B., & Alibali, M. W. (2008). Visible embodiment: Gestures as simulated action.
Psychonomic Bulletin and Review, 15, 495–514.
James, K. H., & Swain, S. N. (2011). Only self-generated actions create sensori-motor systems in
the developing brain. Developmental Science, 14, 673–678. doi:10.1111/j.1467-7687.2010.
01011.x.
Johnson-Glenberg, M. C., Birchfield, D., Koziupa, T., & Tolentino, L. (2014a). Collaborative
embodied learning in mixed reality motion-capture environments: Two science studies.
Journal of Educational Psychology, 106(1), 86–104. doi:10.1037/a0034008.
Johnson-Glenberg, M. C., Birchfield, D., Megowan-Romanowicz, M. C., & Snow, E. L. (2015). If
the gear fits, spin it! Embodied education and in-game assessments. International Journal of
Gaming and Computer-based Simulations, 7(4).
Johnson-Glenberg, M. C., & Megowan-Romanowicz, C. (2017). Embodied science and mixed
reality: How gesture and motion capture affect physics education. Cognitive Research:
Practices and Implications, 2, 24.https://cognitiveresearchjournal.springeropen.com/articles/
10.1186/s41235-017-0060-9.
Johnson-Glenberg, M. C., Megowan-Romanowicz, M. C., Birchfield, D., & Savio-Ramos, C.
(2016). Effects of embodied learning and digital platform on the retention of physics content:
Centripetal force. Frontiers in Psychology, 7, 18–19. doi:10.3389/fpsyg.2016.01819.
Johnson-Glenberg, M. C., Savio-Ramos, C., & Henry, H. (2014b). “Alien Health”: A Nutrition
Instruction Exergame Using the Kinect Sensor. Games for Health Journal: Research,
Development, and Clinical Applications, 3(4), 241–251. doi:10.1089/g4h.2013.0094.
Johnson-Glenberg, M. C., Savio-Ramos, C., Perkins, K. K., Moore, E. B., Lindgren, R., Clark, D.,
… Squire, K. (2014c). Science Sims and Games: Best Design Practices and Fave Flops. Paper
presented at the The International Conference of the Learning Sciences (ICLS), Boulder, CO.
Kang, S., & Tversky, B. (2016). From hands to minds: Gestures promote understanding. Cognitive
Research: Principles and Implications., 1(4). doi:10.1186/s41235-016-0004-9.
Koch, S. C., Glawe, S., & Holt, D. V. (2011). Up and down, front and back: Movement and
meaning in the vertical and sagittal axes. Social Psychology, 42(3), 214–224. doi:10.1027/
1864-9335/a000065.
216 M.C. Johnson-Glenberg

Kontra, C., Lyons, D., Fischer, S., & Beilock, S. L. (2015). Physical experience enhances science
learning. Psychological Science, 26, 737–749. doi:10.1177/0956797615569355.
Lakoff, G., & Johnson, M. (1980). Metaphors we live by. Chicago, IL: University of Chicago
Press.
Lakoff, G., & Núñez, R. (2000). Where mathematics comes from. New York: Basic Books.
Lindgren, R., & Johnson-Glenberg, M. C. (2013). Emboldened by Embodiment: Six Precepts
regarding the Future of Embodied Learning and Mixed Reality Technologies. Educational
Researcher, 42(8), 445–452. doi:10.3102/0013189X13511661.
Lindgren, R., & Johnson-Glenberg, M. C. (2014). Emboldened by embodiment: Six precepts
regarding the future of embodied learning and mixed reality technologies. Educational
Researcher, 42(8), 445–452. doi:10.3102/0013189X13511661.
Lindgren, R., Tscholl, M., Wang, S., & Johnson, E. (2016). Enhancing learning and engagement
through embodied interaction within a mixed reality simulation. Computers & Education, 95,
174–187.
Mayer, R. E. (2009). Multimedia learning (2nd edn.). New York: Cambridge University Press.
Miller, H. L., & Bugnariu, N. L. (2016). Cyberpsychology, Behavior and Social Networking, (4),
246–256. doi:10.1089/cyber.2014.0682.
Montessori, M. (1966). The secret of childhood. New York, N.Y.: Ballantine Books.
Nathan, M. J., Walkington, C., Boncoddo, R., Pier, E. L., Williams, C. C., & Alibali, M. W.
(2014). Actions speak louder with words: The roles of action and pedagogical language for
grounding mathematical reasoning. Learning and Instruction, 33, 182–193. doi:10.1016/j.
learninstruc.2014.07.001.
Niedenthal, P. M., Barsalou, L. W., Winkielman, P., Krauth-Gruber, S., & Ric, F. (2005).
Embodiment in attitudes, social perception, and emotion. Personality & Social Psychology
Review, 9, 184–211.
Noice, H., & Noice, T. (2006). What studies of actors and acting can tell us about memory and
cognitive functioning. Current Directions in Psychological Science, 15, 14–18.
Parmar, D., Isaac, J., Babu, S. V., D’ Souza, N., Leonard, A. E., Gundersen, K., & Daily, S. B.
(2016). Programming moves: Design and evaluation of embodied interaction in virtual
environments to enhance computational thinking in middle school students. IEEE Virtual
Reality (VR), 131–140.
Rutten, N., van Joolingen, W. R., & van der Veen, J. T. (2012). The learning effects of computer
simulations in science education. Computers & Education, 58(1), 136–153.
Segal, A. (2011). Do Gestural Interfaces Promote Thinking? Embodied Interaction: Congruent
Gestures and Direct Touch Promote Performance in Math (Dissertation). (ISBN:
978-1-1246-2610-9). from ERIC.
Shaffer, D. W., Hatfield, D. L., Svarovsky, G. N., Nash, P., Nulty, A., Bagley, E., Franke, K.,
Rupp, A. A., & Mislevy, R. (2009). Epistemic network analysis: A prototype for 21st century
assessment of learning. International Journal of Learning and Media.
Slater, M., Spanlang, B., & Corominas, D. (2010a). Simulating virtual environments within virtual
environments as the basis for a psychophysics of presence. ACM Trans. Graph, 29(4), 1–9.
doi:10.1145/1778765.177882.
Slater, M., Spanlang, B., Sanchez-Vives, M. V., & Blanke, O. (2010b). First person experience of
body transfer in virtual reality. PLoS ONE, 5(5). doi:10.1371/journal.pone.0010564.
Slater, M., & Wilbur, S. (1997). A framework for immersive virtual environments (FIVE):
speculations on the role of presence in virtual environments. PRESENCE: Teleoperators and
Virtual Environments, 6, 603–616.
Squire, K., & Klopfer, E. (2007). Augmented reality simulations on handheld computers. Journal
of the Learning Sciences, 16(3), 371–413.
Stickgold, R., & Walker, M. (2007). Sleep-dependent memory consolidation and reconsolidation.
Sleep Medicine, 8(4), 331–343.
Sweller, J., van Merrienboer, J., & Paas, F. (1998). Cognitive architecture and instructional design.
Educational Psychology Review, 10, 251–296.
11 Embodied Education in Mixed and Mediated Realties 217

Tulving, E., & Thomson, D. (1973). Encoding specificity and retrieval processes in episodic
memory. Psychological Review, 80(5), 352–373. doi:10.1037/h0020071.
Wilson, M. (2003). Six views of embodied cognition. Psychonomic Bulletin & Review, 9(4), 625–
636.
Wu, H., Lee, S. W., Chang, H., & Liang, J. (2013). Current status, opportunities and challenges of
augmented reality in education. Computers & Education, 62, 41–49.

Author Biography

Dr. Mina C. Johnson-Glenberg creates embodied games that specialize in teaching STEM
(Science, Technology Engineering and Math) and Health Sciences. She has been a Principle
Investigator on multiple federal and private grants. She has published widely on cognition,
embodied learning in new media including virtual and mixed realities, neural networks, and fragile
X syndrome (the most common form of heritable intellectual disability). She has created several
natural language processing algorithms and is now engaged in creating and assessing the efficacy
of VR lessons and gesture-based knowledge assessment measures. She was recently an Associate
Professor at Radboud University in the Netherlands, but currently works at Arizona State
University, where she also serves as the President of the Embodied Games lab http://www.
embodied-games.com.
Chapter 12
Preparing Students for Future Learning
with Mixed Reality Interfaces

Bertrand Schneider

Abstract In this chapter, I explore how new learning environments, such as mixed
reality interfaces (i.e., interfaces that combine physical and virtual information), can
prepare students for future learning. I describe four controlled experiments that I
have conducted over the years where students learned complex concepts in STEM
and where a Tangible User Interface created a “Time for Telling”. This is followed
by a summary the findings, a discussion of the possible mechanisms for the effect
found in those studies, and a suggestion of design guidelines for creating this type
of constructivist activities. I conclude by discussing the potential of mixed reality
interfaces for preparing students for future learning.

Keywords Tangible user interfaces
Preparation for future learning

Augmented reality

12.1 Introduction

Over the past decades, new advances in Human–Computer Interaction (HCI) have
radically changed the way we interact with computers. Technology has become
more pervasive, ubiquitous and intuitive to use. The possible inputs are now
multi-modal: users can talk, touch, gesture or even use their gaze to control a
computer. The output is no longer limited to a screen; Augmented Reality
(AR) systems can overlay digital information on the perceived physical world, and
Virtual Reality (VR) can immerse users into virtual worlds. The lines between
digital and physical worlds have blurred, which dramatically increases the design
space for creating new types of immersive learning experiences. The scenarios that
students can experience in mixed-reality/virtual worlds are many and can result in

B. Schneider (&)
Harvard University, 13 Appian Way, Cambridge, MA 02138, USA
e-mail: [email protected]
URL: http://www.bertrandschneider.com

© Springer Nature Singapore Pte Ltd. 2017 219

D. Liu et al. (eds.), Virtual, Augmented, and Mixed Realities in Education,
Smart Computing and Intelligence, DOI 10.1007/978-981-10-5490-7_12
220 B. Schneider

effective, efficient and engaging learning. They were not possible before the advent
and maturation of these powerful digital technologies.
It seems logical to assume that those experiences have an untapped potential in
education. It also makes sense that they cannot—and should not—be used to
replace all types of instruction. Rather, we would expect those experiences to be
used strategically to maximize learning. But what are the theories that could inform
when and how those immersive learning experience would benefit learners the
most? In this chapter, I explore one possibility by leveraging a framework called
Preparing for Future Learning (PFL). The PFL framework suggests that particular
kinds of experiences can help students build prior knowledge in specific ways,
which will then help them take advantage of subsequent instruction. The focus of
this chapter is to enhance those experience through new technologies.
In this chapter, I first introduce new types of computer interfaces that offer
interesting potential for education (Natural User Interfaces, and more specifically
Tangible User Interfaces) and describe their affordances for learning. In Sect. 12.3,
I introduce the PFL framework and its implications for designing learning envi-
ronments and new pedagogical scenarios. Section 12.4 is a summary of empirical
findings that highlight the benefits of adopting this constructivist approach. Finally,
I discuss implications of those results for designing innovative learning
environments.

12.2 Natural User Interfaces (NUIs)

If we look back at the first computers in the 80s, it is astonishing to think about the
steepness of their learning curve. The command line interface (CLI) appeared first,
where the keyboard was the only available input. It forced users to memorize
complex mapping between keystrokes and actions: for instance, the VI text editor—
which is 40 years old, but still used today—required users to press the letter h, j, k, l
to move left, down, up, right on the screen, i to insert characters, d to delete them,
and so on. Even though VI is still among the most powerful text editors available to
programmers today, it still takes a massive amount of time and energy for any given
user to gain fluency in its use. The introduction of the graphical user interface
(GUI) has allowed users to point at elements on a screen with a mouse, and reduced
this learning curve by an order of magnitude. Instead of memorizing complex
keystroke sequences, users can merely point and click. Over the past decade, this
learning curve has become almost non-existent. Toddlers, for instance, have no
issues interacting with touch-screen tablets. The emergence of new kinds of
interfaces, called natural user interfaces (NUIs), have transformed the technological
landscape. NUIs are defined as “systems for human-computer interaction that the
user operates through intuitive actions related to natural, everyday human behav-
ior”. In short, there is no need to learn specific mappings between inputs and
outputs on a NUI: the content is the interface (Wigdor & Wixon, 2011). NUIs
include touch screens (e.g., the IPadTM), gesture-based systems (e.g., the KinectTM
12 Preparing Students for Future Learning … 221

sensor), voice-controlled programs (e.g., Cortana/SiriTM), gaze-aware interfaces

(e.g., TobiiXTM), and brain-machine interfaces that read neural signals and use
programs to translate those signals into inputs.
This revolution has dramatically changed the way we interact with computers,
and has implications for designing educational activities as well. Designers can
create rich scenarios for wider audiences who, for instance, could not use standard
interfaces very well—such as very young children, people with physical disabilities
as well as individuals who are less comfortable with technology. This also means
that we can redirect the burden of learning a computer interface toward more useful
cognitive processes. Instead of having students struggle with memorizing mappings
and menus, they can focus their energy on the learning task. Finally, it allows
instructional designers to create multi-modal interactions. Some domains are best
explored through gestures and body movements, while others benefit from col-
laboratively and verbally exploring a subject. New interfaces (gesture-based,
brain-controlled, gaze-aware, voice-controlled, and so on) have particular affor-
dances that could benefit learners in different ways. In this chapter, I focus on one
type of NUI that has interesting properties for educational applications: tangible
user interfaces.

12.2.1 Tangible User Interfaces (TUIs)

One kind of NUIs that holds interesting potential in education are Tangible User
Interfaces (TUIs). TUIs are systems in which users interact with digital information
through the physical world. The most common implementation of a TUI is a
horizontal interactive tabletop that detects the location and orientation of physical
objects (tagged with fiducial markers, which are similar to QR codes), and displays
additional information on top of them usually with a projector. They transform
traditional manipulatives into dynamic objects that respond to users’ actions
through an augmented reality layer. Since anything can be displayed on the virtual
layer, it allows designers to combine physical and digital affordances in a way that
was not possible before. For example, the Reactable (Fig. 12.1, left) is an inter-
active tabletop created to support creativity through musical exploration. In this
environment, each object is associated with a specific musical sound or action (e.g.,
changing the pitch or volume of a tone). Users can easily connect objects together
to create musical compositions in an intuitive and playful way. Another example is
the Sandscape system (Fig. 12.1, right), which allows users to design and under-
stand landscapes using sand. The tangible interface then displays various infor-
mation about the landscape model to show its height, slope, contours, shadows,
drainage or other features of the simulation.
From a pure HCI (Human–Computer Interaction) perspective, there are several
advantages associated with TUIs. They leverage everyday objects or material with
which users are already familiar. This significantly reduces the amount of time
necessary to learn the interface. When well designed, users can just jump in and
222 B. Schneider

Fig. 12.1 On the left, an interactive tabletop for creating music (the Reactable). On the right, a
landscape created with sand and augmented with a digital information (the Sandscape system)

explore the system on their own without any need for tutorials or explanations
because the interface is intuitive and obvious to many users. TUIs also combine the
best of the physical and digital world. 3D physical objects provide essential
information to the users through their shape, weight, texture and colors, while the
2D digital layer can be used to display anything above and between the objects.
Finally, TUIs facilitate bi-manual interactions: because of the tactile feedback, users
naturally know in which orientation and configuration the objects lie in their hands.
In comparison, touch interfaces require users to constantly check if they have
selected a virtual item or if they are dragging it as intended Those advantages allow
users to perform tasks more quickly and more accurately compared to touch
interfaces (Tuddenham, Kirk, & Izadi, 2010).

12.2.2 TUIs in Education

Educational designers are generally enthusiastic about the potential of TUIs because
manipulatives have been used for centuries to study how young children reason
about the world (most notably by Piaget) and to progressively introduce them to
abstract ideas. Friedrich Froebel (1782–1852), for instance, was a German teacher
who created the concept of kindergarten. He designed a set of physical objects
called “Froebel gifts” to help students learn about geometrical shapes and patterns.
In one set, small cubes were used to introduce children to mathematical concepts
such as addition, subtraction, multiplication and division. By explicitly integrating
complex concepts into manipulatives, Froebel was among the first educators to
design various sets of educational toys.
Later, Maria Montessori (1870–1952), an Italian educator, cleverly designed a
different set of manipulatives to teach mathematical concepts in early childhood.
For instance, she used golden beads to help children grasp the idea of large
quantities and help them transition toward abstract numbers by associating stacks of
beads with numbers (Fig. 12.2). Montessori programs are still alive, and it has been
12 Preparing Students for Future Learning … 223

shown that children who were randomly selected to attend a Montessori program
score higher on standardized math tests than children who had not been selected
and attended a traditional program (Lillard & Else-Quest, 2006). Manipulatives
have also been used to support learning of ratios, geometry, physics, and many
other complex ideas in science. TUIs build on those successful applications, but
they do not have to be limited to simple concepts such as addition or multiplication.
Since the digital layer can represent anything, it is possible to design rich and
complex learning scenarios that incorporate simulations, multiple external repre-
sentations, dynamic scaffolding, just-in-time resources, and other mechanisms
known to support learning. TUIs also support exploratory and expressive learning,
they make learning activities more concrete, playful and engaging, they are
well-suited to spatial domains, and have features that make them ideal for collab-
orative learning. Additional benefits of TUIs are summarized in Fig. 12.3 (repro-
duced from Marshall, 2007).
Considering the affordances of TUIs for learning, it is relatively surprising that
there is not a wealth of educational environments leveraging this new technology.
Many TUIs are created for artistic expression (e.g., the Reactable on the left side of
Fig. 12.1) or for merely replacing elements of a traditional Graphical User Interface
such as sliders, buttons or toggle switches (Schmidt et al., 2014). But there have
been some attempts to design educational TUIs. For example, the Youtopia system
(Fig. 12.4, left) allows young children to analyze the relationship between the
economic development of communities and their available resources (renewable
and non-renewable). Learners use physical tokens to “stamp” the landscape to
modify it by building a facility, use resources or check their progress. A study
found that assigning physical stamps to users supported more equitable verbal and
physical participation compared to a control group where any player could use any
stamp (Fan, Antle, Neustaedter, & Wise, 2014).
Another example is the TapaCarp system designed by Cuendet, Bumbacher, and
Dillenbourg (2012; Fig. 12.4, right side). Apprentices in carpentry use physical

Fig. 12.2 On the left, golden beads used in Montessori schools. On the right, tiles used to
facilitate the transition toward numbers
224 B. Schneider

Fig. 12.3 An analytical framework that describe the potential benefits of tangible user interfaces
(TUIs) in educational settings (Marshall, 2007)

Fig. 12.4 Two examples of TUI in education. On the left, the TapaCarp environment allows
apprentices in carpentry to train their spatial skills. On the right, the Youtopia system helps 5th
graders to learn about sustainability in small groups

objects to understand orthographic projections displayed next to them. In one study,

the authors found that tangibles helped students perform better compared to a
virtual representation (it should be mentioned, however, that it did not significantly
increase their learning gains). There are other examples of the benefits, and
sometimes disadvantages, of TUIs for learning (e.g., Schneider, Jermann, Zufferey,
& Dillenbourg, 2011). However, I will not describe them exhaustively here. Suffice
12 Preparing Students for Future Learning … 225

to say that in some situations, tangible interfaces have affordances for learning that
other interfaces do not have.
But what are the best ways to exploit the learning experiences that students have
with TUIs? Below I introduce a framework that could answer this question, and
provide designers with preliminary guidelines for integrating those new experiences
with classroom instruction.

12.3 The “Preparing for Future Learning”

(PFL) Framework

Even though the past decades have witnessed important technological innovations
like NUIs, classrooms and online learning platforms still operate under the same
principles that have existed for centuries. They favor a “tell-and-practice” (T&P)
paradigm, where students are first exposed to a new idea, usually by a teacher
giving lectures, and then given an opportunity to apply this new knowledge by
solving exercises. Learning scientists (e.g., Bransford & Schwartz, 1999), however,
have been criticizing this paradigm, showing that students gain a superficial mastery
of the concepts taught. Instead, they argue that there is a “time for telling”
(Schwartz & Bransford, 1998): “when telling occurs without readiness, the primary
recourse for students is to treat the new information as ends to be memorized rather
than as tools to help them perceive and think”. Our first instinct is to solve this issue
is by doing more telling. Schwartz and Bransford argue that under these conditions,
students often think that they perfectly understand a concept, when in fact, they are
missing the point.
Making sure that students are ready to learn from standard instruction is at the
core of some constructivist frameworks. One of them, in particular, is the Preparing
for Future Learning framework (PFL; Bransford & Schwartz, 1999). The PFL
framework recognizes that more often than not, students come to the lecture hall
without the prior knowledge necessary to understand the lesson. This theory sug-
gests that instead, we should design learning activities where students can build
some prior knowledge, develop curiosity for the domain taught and think critically
before they are being told what the concept, formula or solution is. The PFL
framework was originally developed to target one specific kind of prior knowledge:
perceptual differentiation. The main methodology used to achieve this goal are
contrasting cases. Contrasting cases are carefully designed representations of a
concept, where some representations vary in terms of their surface features (su-
perficial details that are unimportant) and their deep features (variables that are
central to the concept taught). Students’ goal is to analyze those cases to separate
surface features and deep features. This way, when they are listening to a lecture or
reading a textbook, they have a more refined understanding of which information to
focus on (the deep features of a concept) and what to ignore (the surface features).
This relates to the concept of authenticity discussed in Jacobson’s chapter.
226 B. Schneider

In various studies, researchers have found the PFL framework to yield positive
results on students’ learning. Schwartz, Chase, Oppezzo, and Chin (2011), for
instance, taught adolescents about density using a set of contrasting cases
(CCs) featuring buses of various sizes filled with clowns. The surface features were
the types of clowns and buses. The deep features were the number of clowns and
the size of the buses. Two experimental groups did the same activity, but in dif-
ferent orders. The T&P (“tell and practice”) group was told the formula for density
and then asked to practice this formula on the CCs. The constructivist group
invented a formula for density using the CCs first, and were formally taught about
density afterward. The authors found that even though both groups did equally well
on standard word problems, the second group transferred the idea of density to
semantically unrelated topics much more frequently. In a different study, Schwartz
and Martin (2004) showed similar results where students in statistics had to create a
reliability index for baseball pitching machines—in other words, they had to invent
the formula for the concept of standard deviation from a set of CCs. They found that
students in the PFL condition did better on a test that included additional learning
resources compared to students who followed a T&P instruction.
In the next section, I explore how TUIs could be used as a preparation for future
learning. While CCs focus on having students perceive details that they might
otherwise miss, I suggest that TUIs have particular affordances that could also be
used in a PFL fashion.

12.3.1 Tangible User Interfaces as a Preparation for Future

Learning

This section describes learning environments that I have designed and/or evaluated
in collaboration with others using cutting-edge technology for building
mixed-reality interfaces (Fig. 12.5). This preliminary design cycle has explored
domains as varied as neuroscience, math and probability, neuro-acoustics and
logistics. In those examples, students learn about a complex system or a scientific
phenomenon in a constructionist fashion by re-constructing, de-constructing or re-
assembling its physical elements. The augmented reality layer displays digital
information based on students’ actions. For instance, it can project a simulation, a
connection between two pieces, additional information or display hints. This kind
of learning environment allows the system to dynamically adapt to users’ actions
and provide (to some extent) personalized learning experiences. This type of “just
in time” feedback is especially useful for scaffolding students’ learning. Figure 12.5
describes four TUIs designed for educational purposes.
On the top left of Fig. 12.5, the Brain Explorer system (Schneider et al., 2013)
allows students to learn about concepts in neuroscience by interacting with a
small-scale brain. They first take apart the physical brain regions, and the aug-
mented reality layer displays visual pathways between the tangibles. Students can
12 Preparing Students for Future Learning … 227

Fig. 12.5 Four examples of tangible interfaces in education: Brain Explorer (top left),
Combinatorix (top right), Ear Explorer (bottom left) and the Tinker Table (bottom right)

then disrupt those pathways using an infra-red pen (arrow #1 on the picture), and
the system displays the impact of those lesions on the visual field of this brain
(arrow #2 on the picture). In this example, Mayer’s loop is disrupted on the left side
of the brain, which means that this subject loses the top right quadrant of his visual
field. By repetitively iterating through different scenarios, students can start to build
a basic understanding of how the brain separates visual information into four
quadrants.
On the top right corner of Fig. 12.5, the Combinatorix system allows students to
explore basics concepts in probability. By recombining the letters A, B, C, D and E,
they build an intuitive understanding of how many combinations can be formed
with those letters through the various visualizations that are displayed above them
(e.g., a probability tree). The system then offers additional challenges, where var-
ious constraints are added. For instance, how many letters can be formed when E
has to be the second letter in the sequence? Or how many letters can be formed
when A has to be before B? By progressing through the various challenges and by
analyzing the visualization displayed above the tangibles, students start to build an
intuition about how the different formulas in combinatorics are structured.
On the bottom right corner of Fig. 12.5, The Ear Explorer interface allows
students to rebuild the human hearing system from scratch. The goal is to recreate
228 B. Schneider

the pathway between the outer ear and the auditory cortex by connecting 3D-printed
replicas of the organs of the hearing system. Students can then generate sound
waves at different frequencies to test their construction and see which waves reach
the brain. An important feature of the interface is the ability of students to make
mistakes and build dysfunctional structures. They can then correct those errors by
using the information box (i.e., the circle on the bottom left corner) where they
place tangible items to access hints and can learn additional information about each
organ.
Finally, on the bottom right corner of Fig. 12.5, the Tinker Table is a tangible
interface for apprentices in logistics. Students learn about good design principles for
organizing a warehouse by building a small-scale model, that they then analyze
using more abstract representations such as graphs. This allows the teacher to
provide concrete examples of interesting pedagogical situations, and to progres-
sively move toward more formalized representations of warehouses’ efficiency
(e.g., equations).
This first wave of systems provided enthusiastic feedback from user and
promising directions to follow. We learned one main lesson from building and
testing those systems: using them as a stand-alone learning activity—where would
students to learn everything about a phenomenon—is an extremely challenging
task. We realized that using those systems for mastery learning was potentially
misguided, and prevented us from using TUIs to their full potential. Instead, we
observed that students were more likely to be intellectually and emotionally
engaged about the underlying mechanisms of a phenomenon when interacting with
a TUI. They became more curious, started to ask critical questions and engaged
their peers into conceptual discussions. Those behaviors are important in their own
rights, because they can also prepare students for future learning.
The main design guideline for those activities was to target the situation
(mentioned by Bransford and Schwartz above) where over-telling students pushes
them to “think that they know” because they have memorized information from a
lecture when, in fact, they have large gaps in their knowledge. The learning
activities on the TUI have the main function of helping students explore the domain
taught and realize that there are subtle points that are more difficult to understand
than expected. One side-effect of this intervention is to raise their curiosity: we also
expect them to have more questions about the topic taught after having interacted
with the TUI.

12.3.2 Empirical Findings

A series of controlled experiments combined the PFL framework with some of the
TUIs shown in Fig. 12.5. The experimental design was a replication from Schwartz
et al. (2011). College-level students interacted with a TUI either before or after
following a standard kind of instruction (i.e., reading a textbook chapter or
watching a video lecture).
12 Preparing Students for Future Learning … 229

In a first study (Schneider et al., 2013), we found that individuals who used
Brain Explorer before reading an excerpt of a textbook outperformed students who
used the TUI after reading the text on a learning test (Fig. 12.6, top left). Those
results suggest that TUIs can be used to prepare students for future learning, and
that using this kind of interface in a T&P kind of instruction is less effective.
Additionally, we found differences in the quality of students’ elaborations when
thinking aloud. Students in the PFL group made more high-level comments (such as
defining a rule based on a set of observations), which suggests that they tried to
formulate their mini-theory of how the brain processed visual information. In the
T&P condition, students made more simple observations (such as describing the
effect of one lesion on the visual field of the brain) and repeating information from

6
5
4
3
2
1
0
pre-test post-test
TUI->Instr. Instr.->TUI

20
14
18
16 12
14
10
12
10 8
8 6
6
4 4
2 2
0
CC TUI 0
Pre-test Middle-test Post-test
T&P Invent
Discover Listen

Fig. 12.6 Results of the 4 empirical studies where the PFL framework was applied to TUIs. The
top figures compared the “tell and practice” paradigm (red line, also labelled as “Instruction !
TUI”) with the PFL sequence (“TUI ! Instruction”). The bottom left graph replicates those results
and compares TUIs with a set of Contrasting Cases (CC). The bottom right graph compares two
PFL sequences and asks students to freely rebuild a complex system (discover) or follow step by
step instructions (listen)
230 B. Schneider

the text. Overall, we found that the quality of students’ elaborations was a signif-
icant mediator for their learning gains. It suggests that the PFL sequence increased
learning gains by having students construct their own mental model of the inner
workings of the human brain—which they could then use to make sense of the text.
Based on Bransford and Schwartz (1999)’s framework, we hypothesize that stu-
dents in the T&P did not feel the same need to construct their own mini-theory: they
likely believed that they knew how the brain worked, because they had the chance
to memorize this information from the text. In other words, the text might have
made them overconfident in their ability to understand the inner workings of the
human brain.
In a second study (Schneider & Blikstein, 2015a, b, c), we replicated those
results in a different domain (probability) using a pre-recorded video lecture where
dyads of students interacted with the Combinatorix system (Fig. 12.5, top right).
Again, students in the PFL condition outperformed students in the “tell and prac-
tice” group (Fig. 12.6, top right) on the learning test. We found that students in the
PFL group explored the interface to a greater extent (e.g., they did more actions and
accessed the visualizations more often) and spent more time discussing the concepts
taught. On the other hand, students in the “tell & practice” group spent more time
trying to remember information, such as formulas from the video lecture. This
suggests that the T&P sequence pushed students to memorize, recall and apply
information—at the cost of having in-depth conceptual discussions.
In the third study (Schneider & Blikstein, 2015a, b, c), we used a 2  2
experimental design to replicate those findings and compared the TUI (an updated
version of Brain Explorer) with a set of Contrasting Cases (CC). CC are state of the
art discovery learning activities, which is what was used in the PFL studies men-
tioned above. Additionally, we included two questionnaires between the activities.
One questionnaire captured students’ curiosity by asking them to write all the
questions that they would like to see answered about the human visual system after
the first activity. A second questionnaire measured the quality of their mental model
by asking them to draw a diagram that summarized everything that they had learned
so far. Again, we found that the students in the PFL condition outperformed stu-
dents in the “tell and practice” group on a pre/post questionnaire measuring their
learning gains. We also found that students who first interacted with the TUI built
more complex mental models (as shown by the drawings they made between the
two activities; see examples on Fig. 12.7) which was significantly correlated with
their learning gains. They also became more curious (as expressed by the number of
questions they asked themselves half-way through the study), which was signifi-
cantly correlated with the quality of their mental model. This shows how the PFL
sequence increased students’ curiosity, which had a positive impact on the quality
of their mental models. Better mental models, in turn, were associated with higher
learning gains.
We also observed that learning gains were higher compared to a state of the art
discovery-learning activity (Contrasting Cases). This does not mean that TUIs are
better than CC for preparing students for future learning: it only means that for this
specific domain, this specific TUI and CC, the interactive system yielded higher
12 Preparing Students for Future Learning … 231

Fig. 12.7 Three categories of models drawn by the participants of study 3. The model on the top
left has no or little useful information; the one on the top right has some useful information, mostly
about the terminology; and the one on the bottom contains clear signs of conceptual understanding.
Students in the PFL condition drew models that were more similar to the last two, while students in
the T&P condition were more likely to draw models similar to the first two

learning gains for those students. This might be caused by a variety of factors: a
higher engagement due to the novelty of the interface, the fact that complex systems
might be a particularly good fit for TUIs, or because it is easier to collaborate
around physical objects than a sheet of paper. Exploring this difference should be
the focus of additional studies.
Finally, a last experiment (Schneider et al., 2015) refined the experimental
design and clarified this difference. More specifically, we wondered if this effect
was merely caused by the fact that students in the PFL sequence started the activity
by physically rebuilding a system, which might have increased their engagement
and carried over to the second activity. Similarly, it is possible that completing the
standard instruction in the T&P sequence bored the participants and contaminated
232 B. Schneider

the second activity (i.e., when they interacted with the TUI). In summary, are the
increased learning gains merely caused by a motivational effect? To answer this
question, we used the Ear Explorer system (Fig. 12.5, bottom right) and asked
dyads of students to rebuild the human hearing system from scratch. In one con-
dition, they did so freely. In a different condition, a video of a teacher was displayed
on the bottom right corner of the table, explaining the steps to follow to rebuild the
system. Both groups then read an excerpt from a textbook which described how the
human hearing system works. We found that students in the first group scored
higher on a learning test, as shown on the fourth graph of Fig. 12.6 (bottom right).
They also accessed the information box more often, which was significantly cor-
related with learning gains. It suggests that, when given the chance, students who
could freely discover a domain were more likely to take advantage of additional
resources. Finally, those results demonstrate that combining the PFL framework
with TUIs is not about a quick boost in engagement that carries over the standard
instruction: it is about having students actively build mental models of a concept, so
that they can then use this prior knowledge to make sense of a lecture or textbook
chapter.
In summary, those four studies provide evidence that using TUIs in a PFL
fashion can significantly increase learning gains compared to a traditional T&P
approach. More specifically, there are evidences that students in the PFL group
were more likely to take advantage of the TUI, became more curious, made
higher-level reflections, had more conceptual conversations and were able to build
more complex mental models. In comparison, students in the T&P were more likely
to spend their time recalling and applying information from a standard instruction
(e.g., when watching a video lecture or reading a textbook chapter) and might have
been overconfident in their ability to understand the concepts taught.

12.4 Preliminary Design Principles

Even though more research is needed to explore the effect described above, I
suggest here a few preliminary design guidelines for creating technology-enhanced
PFL activities. Those guidelines are based on my experience designing and eval-
uating those systems, and thus are not always supported by empirical findings.
1) Target prior knowledge: As mentioned above, the best use of Tangible
Interfaces might not be to directly teach concepts to students, but to prepare
them for future learning. The activity should be designed to help students build
some prior knowledge, raise their curiosity, push them to ask critical questions
and highlight potential misconceptions so that they can ground a teacher’s
explanations into a concrete, accessible experience—and not just refer to an
abstract, formalized representation of the concept taught (i.e., equations or
rules).
12 Preparing Students for Future Learning … 233

2) When dealing with complex systems: The studies above suggest that having
students either physically deconstruct, reconstruct or recombine elements of a
complex system is a potentially useful way to prepare them for future learning.
It allows them to build a mini-theory of a concept as they are physically
interacting with its different parts. Gardner’s chapter provides some examples of
this.
3) Design coherent mappings to physical objects: When designing a tangible
system, a crucial decision is to choose what the physical objects will represent.
Ideally, the tangibles should (1) make intuitive sense to the learners (for
instance, it is clear what a brain represents; but it is less clear which idea a cube
is supposed to embody), (2) activate their prior knowledge (e.g., we found that
the shelves of the Tinker Table helped apprentices in logistics activate knowl-
edge from their everyday workplace), (3) be central to the activity to support
students’ learning (e.g., by helping them explore a domain by quickly trying out
combinations), and (4) propose synergies between the physical and digital layer
(e.g., each representation should help the learners make sense of the other
representation: a physical configuration should help students understand a
simulation or a graph projected on the augmented reality layer). For more on this
topic, the interested reader should feel free to consult Mina Johnson’s chapter in
this book.
4) Foster curiosity and engagement: Virtually all learning theories recognize that
engagement is a necessary pre-requisite for learning. When designed well,
tangible interfaces provide students with engaging ways to think about hard
concepts in STEM, because they can represent and embody those ideas in a
playful way (Marshall, 2007). This should be a central aspect to be kept in mind
when designing an educational TUI.
5) Making learning social: Another advantage of TUIs is that they support col-
laborative learning in small groups, by making it easy to own and share physical
objects. Social learning has been recognized as one of the most powerful ways
to foster constructivist learning in the learning sciences and can be supported in
tangible environments, as discussed in Kraemer’s chapter Interactive tabletops,
because of their size and shape, are natural environment for multiple users. It’s a
shared workspace where students are fully aware of each other’s actions, which
helps them externalize and share their thinking process. In a similar way, it
serves as a “group working memory” where the set of objects represents the
current state of the problem. Finally, collaboration can be facilitated by
assigning roles (Schneider & Blikstein, 2015a, b, c) or tangibles (e.g., Fan,
Antle, Neustaedter, & Wise, 2014) to students, which promotes engagement and
participation from each member of the group. For a more exhaustive description
of the benefits of interactive tabletops for collaborative learning, the interested
reader can consult the review by Dillenbourg and Evans (2011).
234 B. Schneider

12.5 Discussion and Conclusion

In this chapter, I have suggested that mixed-reality interfaces allow us to design

new immersive learning experiences for teaching complex concepts in STEM.
I have also described a framework in the learning sciences, called Preparing for
Future Learning (PFL), which suggests ways to leverage those new learning
experiences to enhance standard instruction. More specifically, empirical results
show that having students reconstruct, deconstruct or reassemble a complex system
on a TUI can have a positive impact on their learning, when this activity was used
before a standard instruction (compared to a traditional T&P sequence).
Study 1 showed that neuroscience is a domain that could be particularly
well-suited to a tangible implementation. When students used BrainExplorer in a
PFL fashion, they were more likely to elaborate their own theory of the inner
workings of the brain. In study 2, students who followed a T&P sequence tended to
adopt a mechanistic behavior typical of classroom environments: they spent most of
their time trying to memorize, recall and apply information from the lecture. They
were also less likely to take advantage of the scaffoldings offered by the TUI. In
study 3, there was evidence that students became more curious and built more
complex mental models in the PFL sequence, and that the quality of their model
was predictive of their learning gains. This really emphasizes the importance of
helping students build prior knowledge that could be leveraged to understand
standard instruction. The study also suggests that TUIs can be as good—or
sometimes better—than state of the art PFL activities (i.e., Contrasting Cases).
Finally, a fourth study clarified this effect. It showed that students did not learn
merely because they could be physically active, or because of a novelty effect of the
TUI: it showed that students who can actively build knowledge (via trial and errors,
building hypotheses, testing them, and by freely exploring a domain) tended to
learn more compared to the exact same activity where they are being told how
things work. Additional studies should further refine this effect to design specific
design guidelines for creating learning experiences that could be used in a PFL
fashion.
In summary, the PFL sequence seemed to be helpful to students because it
helped them become more curious. Their curiosity, in turn, supported constructivist
learning. It became easier for them to create complex mental models that they could
use to make sense of a standard instruction. The T&P sequence, on the other hand,
seemed to promote a more mechanistic behavior: students tended to memorize and
recall information, and they spent less time having conceptual reflections.
Those findings can be somewhat counter-intuitive because students and teacher
tend to believe that the T&P sequence is usually more efficient. During informal
interviews of study 1, students in the PFL group strongly believed that being
instructed first would have helped them do a better job. Being immersed in an
environment where they had to make mistakes and figure things out felt uncom-
fortable to them compared to the familiar T&P sequence. But the data shows that
their intuition was wrong: they actually learned more by following a PFL
12 Preparing Students for Future Learning … 235

progression. Additionally, when discussing with teachers during the Tinker

Table project, they expressed reluctance to let students follow a PFL approach. For
them, they had to teach students what they needed to know first. They did not feel
comfortable letting students explore ideas on their own without being told what to
do. Those observations reflect some deeply ingrained cultural beliefs about what
counts as effective instruction.
Finally, there are a few obvious limitations that need to be mentioned. First, the
activities described in this chapter were used to introduce students to concepts that
were new to them. It is possible that the results would be different for students who
already hold misconceptions about the concepts taught. Second, we found TUIs to
be well-suited to spatial domains. It would likely be much more challenging to
teach purely abstract concepts using tangibles. Third, the interventions were rela-
tively short (20–30 min. for each activity). It is unclear if the findings above would
hold for longer lessons. Finally, each study was conducted as a controlled exper-
iment, which does not take the complexity of a classroom into account. Future work
should study how this effect can be transferred to formal and informal learning
environments.

Acknowledgements I would like to thank my numerous collaborators: Roy Pea, Paulo Blikstein,
Dan Schwartz, Pierre Dillenbourg, Engin Bumbacher, Guillaume Zufferey, Sebastien Cuendet, and
many others. Part of this work has been funded through the NSF grants #0835854 and #1055130,
as well as the Leading House Technologies for Vocation Education, funded by the Swiss State
Secretariat for Education, Research and Innovation.

References

Bransford, J. D., & Schwartz, D. L. (1999). Rethinking transfer: A simple proposal with multiple
implications. Review of Research in Education, 24(1), 61–100.
Cuendet, S., Bumbacher, E., & Dillenbourg, P. (2012, October). Tangible vs. virtual represen-
tations: When tangibles benefit the training of spatial skills. In Proceedings of the 7th Nordic
Conference on Human-Computer Interaction: Making Sense Through Design (pp. 99–108).
ACM.
Dillenbourg, P., & Evans, M. (2011). Interactive tabletops in education. International Journal of
Computer-Supported Collaborative Learning, 6(4), 491–514.
Fan, M., Antle, A. N., Neustaedter, C., & Wise, A. F. (2014). Exploring how a co-dependent
tangible tool design supports collaboration in a tabletop activity. Paper presented at the
International Conference on Supporting Groupwork, Sanibel Island, FL.
Lillard, A., & Else-Quest, N. (2006). The early years: Evaluating Montessori education. Science,
313(5795), 1893–1894.
Marshall, P. (2007). Do tangible interfaces enhance learning? In Proceedings of the 1st
international conference on Tangible and embedded interaction (pp. 163–170). Baton Rouge,
Louisiana: ACM.
Schmidt, D., Ramakers, R., Pedersen, E. W., Jasper, J., Köhler, S., Pohl, A., Baudisch, P. (2014).
Kickables: Tangibles for Feet. In Proceedings of the SIGCHI Conference on Human Factors in
Computing Systems, (pp. 3143–3152). New York, NY, USA: ACM.
236 B. Schneider

Schneider, B., & Blikstein, P. (2015). Flipping the lipped classroom: A study of the effectiveness
of video lectures versus constructivist exploration using tangible user interfaces. IEEE
Transactions on Learning Technologies, 9(1), 5–17.
Schneider, B., & Blikstein, P. (2015a). Unraveling students’ interaction around a tangible interface
using multimodal learning analytics. International Journal of Educational Data Mining, 7(3),
89–116.
Schneider, B., & Blikstein, P. (2015b). Flipping the flipped classroom: A study of the effectiveness
of video lectures versus constructivist exploration using tangible user interfaces. IEEE
Transactions on Learning Technologies, 9(1), 5–17.
Schneider, B., & Blikstein, P. (2015c). Comparing the benefits of a tangible user interface and
contrasting cases as a preparation for future learning. In Proceedings of the 11th International
Conference on Computer Supported Collaborative Learning—Volume 1 (pp. 134–141).
International Society of the Learning Sciences.
Schneider, B., Jermann, P., Zufferey, G., & Dillenbourg, P. (2011). Benefits of a tangible interface
for collaborative learning and interaction. Learning Technologies, IEEE Transactions on, 4(3),
222–232.
Schneider B., Wallace J., Pea, R. & Blikstein P. (2013). Preparing for future learning with a
tangible user interface: the case of Neuroscience. IEEE Transactions on Learning
Technologies, 6(2), 117–129.
Schwartz, D. L., & Bransford, J. D. (1998). A time for telling. Cognition & Instruction, 16, 475–
522.
Schwartz, D. L., Chase, C. C., Oppezzo, M. A., & Chin, D. B. (2011). Practicing versus inventing
with contrasting cases: The effects of telling first on learning and transfer. Journal of
Educational Psychology, 103(4), 759–775.
Schwartz, D. L., & Martin, T. (2004). Inventing to prepare for future learning: The hidden
efficiency of encouraging original student production in statistics instruction. C&I, 22(2), 129–
184.
Tuddenham, P., Kirk, D., & Izadi, S. (2010). Graspables revisited: Multi-touch vs. tangible input
for tabletop displays in acquisition and manipulation tasks. In Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems (pp. 2223–2232). New York, NY, USA:
ACM.
Wigdor, D., & Wixon, D. (2011). Brave NUI world: designing natural user interfaces for touch
and gesture. Elsevier.

Author Biography

Bertrand Schneider obtained a Ph.D. in the Learning Sciences and Technology Design Program
at Stanford University’s Graduate School of Education. In January 2017, he has joined the faculty
of the Harvard Graduate School of Education as an assistant professor. His research interests
include the development of new and original interfaces for collaborative learning in formal and
informal learning environments such as maker spaces and fabrication labs.
Chapter 13
Conclusion—Strategic Planning for R&D
on Immersive Learning

Christopher J. Dede and John Richards

Abstract This chapter articulates a Grand Challenge of applied, collective

research. In our judgment, we must establish educational research laboratories with
access to the next generations of tools and complementary research agendas to
rapidly accumulate knowledge. Immersive technologies cannot be examined in the
hope of changing one parameter and observing the effects. Rather, immersion,
particularly VR and MUVEs, by their very nature alter perspective, context, and
even the participants’ sense of self, while immersion in the form of MR and AR
alters our sense of participation in the external environment. Design strategies for
this scholarly work include applied, collective research; studies on what works
when for whom, it what context; and research balanced between design and
evaluation. Our illustrative research agenda includes thematic areas: authenticity,
representation of the self, social immersion, and technological support and infras-
tructure. We also suggest developing implementation testbeds that provide
authentic contexts to judge the educational value of immersive learning
experiences.

Keywords Strategic planning
Immersive learning

Research and development

C.J. Dede (&)
J. Richards

Harvard Graduate School of Education, 336 Longfellow Hall, 13 Appian Way,
Cambridge, MA 02138, USA
e-mail: [email protected]
URL: https://www.gse.harvard.edu/faculty/christopher-dede
J. Richards
e-mail: [email protected]
URL: http://www.cs4ed.com
J. Richards
Consulting Services for Education, Inc., 22 Floral Street, Newton, MA 02461, USA

© Springer Nature Singapore Pte Ltd. 2017 237

D. Liu et al. (eds.), Virtual, Augmented, and Mixed Realities in Education,
Smart Computing and Intelligence, DOI 10.1007/978-981-10-5490-7_13
238 C.J. Dede and J. Richards

13.1 Next Steps for the Research Community

As the book chapters illustrate, researchers have found that immersive media have
great potential to promote enriched curricular content, active forms of learning,
performance assessments with high validity, and the application of academic
learning to real world situations. However, much work remains to be done before
these educational technologies will be truly practical, affordable, and scalable. That
goal will require much greater, more targeted, and more sustained support for
research and development than the piecemeal funding that exists today, offered
mainly through a handful of intermittent federal and corporate programs. At pre-
sent, the most important priorities for scholars are (1) to design and study
high-quality educational environments that promote situated learning and transfer in
areas where immersion can make a big difference in student outcomes, and (2) to
help education stakeholders grasp the value of these learning media, understand
metrics for judging quality, and increase investments in this approach.

13.2 Important Dimensions for Research Design

Applied, collective research: At present, an important priority is for studies that

produce “usable knowledge” about technology-enhanced instruction, motivated
primarily not by intellectual curiosity, but instead out of a desire to address per-
sistent problems in practice and policy (Dede, 2011). This is not to disparage basic
research or theoretical work, which is required particularly for types of immersive
learning that have no precedent in prior educational research (e.g., mixed realities,
transformed social interaction), as discussed later. However, in immersive learning
at present, greater investments in applied research are warranted. The process of
creating and sharing such usable knowledge is best accomplished not by scholars
working in isolation, but instead by a community of researchers, practitioners, and
policymakers. While individually developed “outlier” ideas can be valuable and
should be some part of the research agenda. However, if the goal is to develop and
invest in immersive media that promote greater learning, then what is required is a
multi-dimensional perspective, touching on the whole range of academic,
inter-personal, and intra-personal capacities that have been shown to matter to
young people’s long-term success. That sort of “grand challenge” necessitates a
group effort, combining various research methods and integrating knowledge from
several fields, from cognitive psychology to teacher capacity building to curriculum
studies to educational technology.
Fully understanding a complex educational intervention involving immersive
media that is effective across a wide range of contexts may require multiple studies
along its various dimensions, each scholarly endeavor led by a group that spe-
cializes in the methods best suited to answering research questions along that
dimension. Using such a distributed research strategy among collaborating
13 Conclusion—Strategic Planning for R&D on Immersive Learning 239

investigators, funders could create portfolios in which various studies cover dif-
ferent portions of this sophisticated scholarly territory, with complementary
research outcomes enabling full coverage and collective theory-building. Further,
once the efficacy of an intervention is determined via exploratory research, a single
large effectiveness study with a complex treatment is of greater value for research
than multiple small studies of individual simple interventions, none of which has
the statistical power to determine the nuanced interaction effects described next.
Research on what works for whom, in what context: Numerous studies
document that no single pedagogy is optimal for all subject matter and every
student (Dede, 2008). Education developers too often assume that innovations
cannot be brought to scale unless they can be replicated precisely and implemented
with fidelity. However, experience has shown that the successful implementation,
use, and spread of learning technologies often depends on the adaptations to a
particular setting that they undergo at the local level. Therefore, the best way to
invest in new technologies for immersive learning is to acknowledge that context
matters; these media must be designed for customization to serve a range of edu-
cational settings, their teachers and students, their curriculum, and their culture. In
brief, such tools should be designed with local adaptations in mind, and profes-
sional development should include building educators’ capacity to make these
adjustments. Therefore, statistical power in research studies about efficacy is
important because, rather than assuming that an educational technology is “effective
for all learners” in some universal manner, research and development should focus
on what works for whom, in what contexts (Means, 2006).
Research balanced between design and evaluation: A research agenda should
respond both to stakeholders’ needs for evaluative studies and the designers’ needs
for research that informs design and theory. Too many studies now privilege the
former over the latter. A “blended” empirical research model designed not only to
answer questions about whether a program design works well, but also to provide
evidence to explain why it works well seems a reasonable and effective alternative
to the evaluation-centric approach now prevalent. Such a research strategy also
mitigates researcher-practitioner tensions: On the one hand, the need for
evaluation-based marketing to potential adopters of immersive media is a ‘neces-
sary evil’ for many scholars. On the other hand, the more theoretical work of
explaining why and to what extent design interventions work is at best the ‘icing on
the cake’ to developers and vendors of educational games and simulations, who
know that ultimate sustainability depends on creating an evidence-based com-
mercial product.
Thus, a research agenda for immersive learning should focus on applied, col-
lective research; adaptation of learning environments based on what works for
whom, in what context; and a balance between design-based studies and evalua-
tions of effectiveness.
240 C.J. Dede and J. Richards

13.3 Illustrative Research Questions Suggested

by the Chapters in This Volume

Below are illustrative research questions suggested by the chapters in this volume.
The intent is neither to display a complete list of possibilities nor to claim that these
constitute the “best” research agenda, but instead to start a dialogue about what
such an agenda might include and how it might be formulated.
Authenticity
• To what extent can immersive media replicate via simulation various types of
authentic practices learners can master and transfer to the real world?
• For various types of objectives for an immersive learning experience, which
types of authenticity are most important, and what design heuristics can be used
to create these? (Jacobson’s analysis of authenticity provides guidance for the
level of detail required in different contexts.)
Representation of the Self
• For various types of objectives for an immersive learning experience, when is
embodied learning important, and what design heuristics can be used to realize
this? (Johnson-Glenberg’s exploration of embodiment describes ways to bridge
the distance between the immersive experience and learning educational
content.)
• To what extent do having experiences that convey an alternative perspective
embodied in a body not your own result in implicit changes in attitude and
behavior? (Slater’s research on racism and on access to mental resources
illustrates this question.)
• Do pedagogical agents that resemble oneself result in more effective learning?
(Kraemer discusses various design strategies for pedagogical agents and for
transformed social interaction.)
Social Immersion
• How can VR technologies foster learning via optimizing social interactions
between teacher and learner, or between learners in collaborative learning
environments? (Gardner describes various design strategies to enhance learner
collaboration and interaction.)
• For various types of objectives for an immersive learning experience, which
types of social immersion are most important, and what design heuristics can be
used to create these? (Klopfer proposes that massively multiplayer games will
provide a level of engagement—in his terms, investment–that supports social
learning and exploration.)
Technical Support and Infrastructure
• For various types of objectives for an immersive learning experience, what
heuristics can determine which type of immersive medium (e.g., virtual reality,
13 Conclusion—Strategic Planning for R&D on Immersive Learning 241

MUVE, mixed reality) will be most effective? What heuristics can determine
when transmedia design (blending these) adds substantial value for learning?
(Dede discusses transmedia narratives as a means of enhancing learning.)
• What design heuristics can reduce issues of comfort and of registration in
immersive learning experiences?
• What infrastructure is required for immersive technologies to be successfully
brought to scale in the school environment? (Richards articulates both the
technological requirements and the design requirements necessary for broad
acceptance.)
Research
• What frameworks for terminology and taxonomy can best aid communication
among scholars in immersive learning? (Liu and Huang propose frameworks
that could address this need.)
• For various types of objectives for an immersive learning experience, which
multi-model dimensions add most value in understanding effectiveness?
(Schneider describes frameworks for interrelating these dimensions.)
• What design heuristics can maximize transfer by manipulating presence,
authenticity, embodiment, saliency, and identity?
• Through which types of instructional design can immersive media help learners
to develop affective capabilities, such as self-efficacy, growth mindset, tenacity,
initiative, flexibility, and conscientiousness?
• Through which types of instructional design can immersive media be scalable
via adaptation to local settings?
• For students who need direct instructional supports embedded in immersive
learning experiences, what are effective models for accomplishing this without
undercutting engagement and flow? (Shute discusses the use of stealth assess-
ments to aid learning and teaching.)
• What capabilities do teachers need to use immersive media effectively in formal
educational settings? What types of professional development and peer-based
learning will most aid teachers?
This is an incomplete list of high-level research questions—it illustrates a
method for generating a research agenda by aggregating and categorizing the
research foci of various scholars studying immersive learning. This initial list was
derived based on the assumptions articulated above: usable knowledge, collective
research, adaptation to specific situations, and a balance of design and evaluation
are all represented as cross-cutting themes.

13.4 Developing Implementation Testbeds

Integrating research into educational practice involves three steps (Carlile, 2004):
Transfer from research to practice, Translation of research into practice, and
Transformation of practice based on research. The initial stage, Transfer, is
242 C.J. Dede and J. Richards

insufficient because it reflects the traditional one-size-fits-all scale-up model that has
not led to sustainable improvements. Translation and Transformation involve major
shifts in policies and practices as mutual adaptation between local contexts and
research-based innovations. In this process, risk-taking is essential. For example,
it’s crucial to see experiments that are “informative failures” as a success in
advancing knowledge.
Researchers typically struggle to find even small numbers of classrooms and
schools, practitioners and policymakers, willing to engage in this type of risk-taking
and experimentation. Therefore, educational testbeds are needed that provide
authentic contexts for Transfer, Translation, and Transformation based on immer-
sive learning. This requires substantial numbers of classrooms and schools willing
to move beyond the traditional instructional and assessment models of practitioners,
the conventional ideological and political standards of policymakers, and the usual
rigorous practices and evidence standards of researchers. Finding venues willing to
undertake immersive learning at scale is a substantial barrier to implementing the
research agenda described in this volume.

13.5 Concluding Thoughts

For immersive media, the prices are cycling down, and the experiences that are
available now on high-end devices will be cost appropriate for schools in the next
few years. Anticipating these developments, we have articulated a Grand Challenge
of applied, collective research. In our judgment, we must establish educational
research laboratories with access to the next generations of tools and comple-
mentary research agendas to rapidly accumulate knowledge.
The research agenda we have outlined presents intricate challenges, perhaps best
summed up by Marshall McLuhan, “There is no ceteris paribus [other things being
equal] in the world of media and technology. Every extension or acceleration effects
new configurations in the over-all situation at once” (1964, p. 167). Immersive
technologies cannot be examined in the hope of changing one parameter and
observing the effects. Rather, immersion, particularly VR and MUVEs, by their
very nature alter perspective, context, and even the participants’ sense of self, while
immersion in the form of MR and AR alters our sense of participation in the
external environment. This book documents the most advanced work completed to
date in order to understand the applications of immersive technologies to learning
and education, but also demonstrates how early stage our current understandings
are. We have only scratched the surface of understanding the educational impli-
cations of these technologies.
13 Conclusion—Strategic Planning for R&D on Immersive Learning 243

References

Carlile, P. R. (2004). Transferring, translating, and transforming: An integrative framework for

managing knowledge across boundaries. Organizational Science, 15(5), 555–568.
Dede, C. (2008). Theoretical perspectives influencing the use of information technology in
teaching and learning. In J. Voogt & G. Knezek (Eds.), International handbook of information
technology in primary and secondary education (pp. 43–62). New York: Springer.
Dede, C. (2011). Developing a research agenda for educational games and simulations. In S.
Tobias & J. D. Fletcher (Eds.), Computer games and instruction (pp. 233–250). Hershey, PA:
Information Age Publishers.
McLuhan, M. (1964). Understanding media (2nd edn.). New York: New American Library.
Means, B. (2006). Prospects for transforming schools with technology-supported assessment. In R.
K. Sawyer (Ed.), Cambridge handbook of the learning sciences (pp. 505–520). Cambridge:
Cambridge University Press.

Author Biographies

Christopher J. Dede is the Timothy E. Wirth Professor in Learning Technologies at Harvard’s

Graduate School of Education (HGSE). His fields of scholarship include emerging technologies,
policy, and leadership. From 2001 to 2004, he was Chair of the HGSE department of Teaching and
Learning. In 2007, he was honored by Harvard University as an outstanding teacher, and in 2011
he was named a Fellow of the American Educational Research Association. From 2014 to 2015, he
was a Visiting Expert at NSF, Directorate of Education and Human Resources. Chris has served as
a member of the National Academy of Sciences Committee on Foundations of Educational and
Psychological Assessment, a member of the U.S. Department of Education’s Expert Panel on
Technology, and a member of the 2010 National Educational Technology Plan Technical Working
Group. In 2013, he co-convened a NSF workshop on new technology-based models of
postsecondary learning; and in 2015 he led two NSF workshops on data-intensive research in the
sciences, engineering, and education. His edited books include: Scaling Up Success: Lessons
Learned from Technology-based Educational Improvement, Digital Teaching Platforms:
Customizing Classroom Learning for Each Student, and Teacher Learning in the Digital Age:
Online Professional Development in STEM Education.

John Richards Ph.D., is an Instructor at the Harvard Graduate School of Education teaching
courses in Education and Entrepreneurship. He is founder and President of Consulting Services for
Education, Inc. (CS4Ed). CS4Ed works with publishers, developers, and educational organiza-
tions, as they negotiate the rapidly changing education marketplace to improve business–planning
processes, and to develop, evaluate, and refine products and services.
John was President of the JASON Foundation, and GM of Turner Learning—the educational
arm of CNN and the Turner Broadcasting System. Over the years, John has served on boards for a
variety of education groups including NECC; Cable in the Classroom; Software Information
Industry Association (SIIA), Education Market section; and the Association of Educational
Publishers (AEP). John’s projects have won him numerous awards including two Golden Lamps
and several CODIEs, as well as several EMMY nominations. He is a respected keynote speaker,
has been responsible for the publication of over 1000 educational products, and is the author/editor
of over 100 chapters and articles, and four books, including Digital Teaching Platforms, Teacher’s
College Press (with Chris Dede). He is the primary author of the Software and Information
Industry Association’s annual U.S. Educational Technology Market: Pre K–12 report.
Glossary of Terms Related to Immersive
Learning

Forms of Immersion
Sensory Immersion The experience of having your senses mostly or totally
surrounded by a three-dimensional “view” of a virtual world. For example a
head-mounted display (HMD) or a planetarium dome shows you the virtual
environment in (almost) any direction you look. A good pair of stereo head-
phones can immerse your hearing in an unreal soundscape.
Psychological Immersion The mental state of being completely absorbed or
engaged with something. Because psychological immersion is a state of mind, it
could be brought about by a variety of situations (such as reading a book or
watching a movie).
Immersive Media Media that use sensory immersion to induce psychological
immersion in the viewer or participant. These include Virtual Reality (VR),
Augmented Reality (AR), all manner of Mixed Reality (MR), and to a lesser
extent other large-format media such as movie theaters or planetariums.
Virtual Environment (VE) The digital world which the user occupies. For
example, a person wearing an HMD will see the VE all around him or her, while
a user in a MUVE or a single-user VR will see his/her avatar at a specific place
in the VE. The term does not apply to any mixed or augmented reality; a virtual
environment is digital and takes the user (figuratively) somewhere else.
Multi-user Virtual Environment (MUVE) A virtual world accessed by one or
more people, usually many people. Each person is represented by a (usually)
humanoid character that s/he controls (an avatar). Technically, MUVEs are not
immersive media, because they are usually accessed through a standard com-
puter or mobile interface. However, they achieve many of the same psycho-
logical effects.
Virtual Presence (Place Illusion) A particular form of psychological immersion,
the feeling that you are at a location in the virtual world. For example, using an
head-mounted display to see a virtual room in every direction you look, makes

© Springer Nature Singapore Pte Ltd. 2017 245

D. Liu et al. (eds.), Virtual, Augmented, and Mixed Realities in Education,
Smart Computing and Intelligence, DOI 10.1007/978-981-10-5490-7
246 Glossary of Terms Related to Immersive Learning

you feel like you are in that room. MUVEs achieve a similar form of presence,
though the user’s emotional investment in their avatar.
Actional Immersion Empowering the participant in an experience to initiate
actions that have novel, intriguing consequences. For example, when a baby is
learning to walk, the degree of concentration this activity creates in the child is
extraordinary. Discovering new capabilities to shape one’s environment is
highly motivating and sharply focuses attention.
Symbolic/Narrative Immersion Triggering powerful semantic associations via
the content of an experience. As an illustration, reading a horror novel at mid-
night in a strange house builds a mounting sense of terror, even though one’s
physical context is unchanging and rationally safe. Narrative is an important
motivational and intellectual component of all forms of learning. Invoking
intellectual, emotional, and normative archetypes deepens the experience by
imposing a complex overlay of associative mental models.

Interfaces

Augmented Reality A form of mixed reality: Real world situations enhanced for
learning by overlays with virtual information and experiences, presented on
mobile devices (e.g., looking at a statue and seeing history about the person
superimposed through the camera view of a tablet).
Haptics Using touch and pressure in a sensory experience (e.g., learning surgery
on a virtual patient with sensory feedback on the incisions you make).
Tangible Interfaces A form of mixed reality: Interactive experiences manipulating
objects that have both real and virtual components (e.g., manipulating physical
blocks that, as you move them around, provide virtual overlays about geometry).

Cognitive Science

Situated Learning “Situated” learning takes place in the same or a similar context
to that in which it is later applied, and the setting itself fosters tacit skills through
experience and modeling. For example, in a medical internship, both the con-
figuration and the coordinated team activities in a hospital surgical operating
room provide embedded knowledge.
Transfer Transfer is the application of knowledge learned in one situation to
another situation, demonstrated if instruction on a learning task leads to
improved performance on a transfer task, typically a skilled performance in a
real-world setting. For example, statistical reasoning learned in a classroom can
potentially aid with purchasing insurance, or with gambling.
Glossary of Terms Related to Immersive Learning 247

Plan/Act/Reflect (PAR) Students first prepare for an experience by doing some-

thing they want to master, then they attempt that performance, and finally they
assess what went well, what did not, why, and what they need to learn in order to
execute a more successful repetition of the cycle.
Embodied Cognition An instructional strategy that posits retrieving a concept
from memory and reasoning about it is enhanced by creating a mental perceptual
simulation of it.