Computers in Human Behavior 81 (2018) 250e264

Contents lists available at ScienceDirect

Computers in Human Behavior

journal homepage: www.elsevier.com/locate/comphumbeh

Full length article

Breakdowns in children's interactions with a robotic tutor: A

longitudinal study
Sofia Serholt 1
Department of Applied Information Technology, University of Gothenburg, Forskningsgången 6, 412 96, Gothenburg, Sweden

a r t i c l e i n f o a b s t r a c t

Article history: In recent years, there has been a growing research interest towards exploring the benefit of ChildeRobot
Available online 22 December 2017 Interaction for educational purposes through the use of social robotics. Despite the label, such robots are
typically only social within scripted activities. The current study takes a critical look at the case of a
Keywords: robotic tutor which was implemented in an elementary school for 3.5 months, where children repeatedly
Childerobot interaction took turns interacting with the robot individually as well as in pairs. The aim of the study was to explore
Education
what caused breakdowns in children's interactions with the robotic tutor. In this qualitative study, over
Robotic tutor
14 h of video recordings of children's interaction sessions were analyzed in-depth through interaction
Breakdowns
Interaction analysis
analysis and thematic analysis. The results comprise four themes to explain why children's interactions
Thematic analysis with the robotic tutor break down: (1) the robot's inability to evoke initial engagement and identify
misunderstandings, (2) confusing scaffolding, (3) lack of consistency and fairness, and finally, (4)
controller problems. The implications of these breakdowns for the educational use of robots are dis-
cussed, and it is concluded that several challenges need to be rigorously addressed in order for robotic
tutors to be able to feature in education.
© 2017 Elsevier Ltd. All rights reserved.

1. Introduction DeMeester, 2013; Lee, Lee, Kye, & Ko, 2008; Teo, 2011).
As Selwyn (2008) argues, research on educational technology
In recent years, there has been a growing research interest to- tends to focus on what should or could happen once technology
wards exploring the benefit of ChildeRobot Interaction (CRI) for moves into the classroom (i.e., the state of the art), leading to a focus
educational purposes through the use of social robotics (Benitti, on the positive aspects of educational technology. Yet he argues
2012; Mubin, Stevens, Shahid, Mahmud, & Dong, 2013). As part of that research needs to be equally concerned with the state of the
this effort, having robots feature as tutors is considered a promising actual, i.e., “questions concerning what is actually taking place
approach (Castellano et al., 2013), argued to offer a number of when technology meets classroom” (Selwyn, 2008, p. 83). Although
benefits for education, such as to personalize education to indi- robotic tutors can be considered state of the art-technology, they
vidual children's needs (Leyzberg, Spaulding, & Scassellati, 2014), can still be placed as-is in authentic educational settings, where
support learning (Kory Westlund et al., 2017), and alleviate teach- negative aspects of children's interactions with them can be
ers' workload (Movellan, Tanaka, Fortenberry, & Aisaka, 2005). As brought to the forefront in the research process. Thus, in order to
teachers agree, robots and other educational technologies should study how the abovementioned visions for robotic tutors play out
not be overbearing in relation to their professional workload in practice, this paper takes a critical look at children's interactions
(Serholt, Barendregt, et al., 2014), while they also need to be useful, with a robotic tutor in an educational setting. The robotic tutor was
and able to support children's learning (Fridin & Belokopytov, 2014; developed as part of an interdisciplinary EU-project called EMOTE,
Kennedy, Lemaignan, & Belpaeme, 2016; Kim, Kim, Lee, Spector, & and was implemented in a school in Sweden. The robot was
designed to tutor students in activities based on the syllabi for
geography and social studies for elementary education, seeking to
Abbreviations: CRI, ChildeRobot Interaction; HCI, HumaneComputer Interac- offer educational value to participating schools, in addition to the
tion; HHI, HumaneHuman Interaction. scientific value of studying CRI (Serholt, Barendregt, et al., 2014).
E-mail address: sofi[email protected]. Robots can be said to present both differences as well as simi-
1
Present address: Department of Computer Science and Engineering, Chalmers larities in relation to other educational technologies such as
University of Technology, Ho€ rselgången 4, 412 96 Gothenburg, Sweden.

https://doi.org/10.1016/j.chb.2017.12.030
0747-5632/© 2017 Elsevier Ltd. All rights reserved.
 S. Serholt / Computers in Human Behavior 81 (2018) 250e264 251

computers and tablets. On the one hand, children may have that are traditionally held by other humans” (p. 628). There are a
distinctive expectations of robots because of their appearance and number of robot capabilities that are thought to facilitate a positive
behavior (Belpaeme et al., 2013). Robots can share and move about interaction between children and robots, such as empathy
within the same physical world as people, and they can resemble (Castellano et al., 2013), non-verbal immediacy (Kennedy, Baxter, &
humans in appearance and behavior (Duffy, 2003; Fong, Belpaeme, 2017), social support (Leite, Castellano, Pereira,
Nourbakhsh, & Dautenhahn, 2003). This entails that robots are Martinho, & Paiva, 2012), personalization (Gordon et al., 2016;
more likely to be approached as (humanlike) artifacts to interact Leyzberg et al., 2014), and various levels of social behaviors
with, rather than to be used as tools for something else (Ho € flich, (Kennedy et al., 2015).
2013; Zhao, 2006; van Oost & Reed, 2011). Indeed, when the ro- Robots can also take on more instrumental roles in education, in
botic tutor under study was implemented in a set of schools in which case social capabilities become less relevant. For instance,
Europe, it was evident that children were prone to interact with it following Papert's notion of constructionism (Papert, 1980), robots
socially (Serholt & Barendregt, 2016), or to perceive it as a friend (or robotic kits) are used as hands-on tools in order to explore their
(Alves-Oliveira, Sequeira, & Paiva, 2016); findings that have been potential for facilitating students' computational thinking and
reported in other CRI studies, as well (Fior, Nugent, Beran, Ramirez- learning of skills in subjects relating to science, technology, engi-
Serrano, & Kuzyk, 2010; Hyun, Yoon, & Son, 2010; Kahn, Friedman, neering and math (STEM). This can be done by, e.g., practicing the
Perez-Granados, & Freier, 2004; Kanda, Hirano, Eaton, & Ishiguro, actual programming of robots (Nugent, Barker, & Grandgenett,
2004; Kennedy, Baxter, & Belpaeme, 2015; Tanaka, Cicourel, & 2012) or through robot assembly (Vandevelde, Wyffels, Ciocci,
Movellan, 2007). Vanderborght, & Saldien, 2015). Furthermore, robots can also be
On the other hand, robots are similar to other digital learning used as proxies when joint presence is not possible; e.g., if teachers
applications in the way that such tasks are usually structured. need to conduct lessons away from actual classrooms, they can
Indeed, although robotic tutors are intended to function in a social control, and thus communicate through a robot remotely (Yun
context, there are still substantive technical constraints when it et al., 2011).
comes to robots perceiving the natural (social) world around them Mubin et al. (2013) employ five dimensions to classify research
(Belpaeme et al., 2013), making it much easier to design a robotic on educational robots. These are: the embodiment or type of robot
tutor with specific capabilities for one or more structured educa- used, the roles or behaviors of the robot, the pedagogical theories
tional activities. There may be clear trajectories within the actual underpinning the research, the location of the learning activity
activities, such as correct versus incorrect answers. Also, there may (e.g., formal or informal education), as well as the domain or sub-
be a specific way in which the activities can be carried out, both in ject of the learning activity (Mubin et al., 2013). Following the scope
terms of the social interaction modalities that the robot can of the current study, the embodiments of social robots in education
perceive and respond to, as well as the ways in which answers can typically take some sort of humanoid appearance (such as NAO or
be provided or moves within the activity can be carried out. In the Robovie), or zoomorphic form (such as iCat or AIBO). When it
current study, a map reading activity (Hall et al., 2016), as well as a comes to roles and behaviors, robots can feature as, e.g., teachable
game on sustainable energy consumption (Alves-Oliveira et al., agents (Lemaignan et al., 2016; Tanaka & Matsuzoe, 2012), learning
2016), could be carried out alongside the robot on an interactive companions (Castellano, Pereira, Leite, Paiva, & McOwan, 2009;
touchtable. Kanda, Sato, Saiwaki, & Ishiguro, 2007), or tutors (Kennedy,
A robotic tutor's primary purpose is to instruct and guide chil- Baxter, Senft, & Belpaeme, 2016; Leyzberg et al., 2014). Concern-
dren within specific learning activities. If this purpose is jeopar- ing pedagogical underpinnings, the prote ge
 effect, i.e., the idea that
dized for whatever reason, problems may arise that lead to children put more effort into learning for others than for them-
breakdowns in interaction (Iacovides, Cox, McAndrew, Aczel, & selves (Chase, Chin, Oppezzo, & Schwartz, 2009), is considered to
Scanlon, 2015; Ryan & Siegel, 2009), where children, e.g., grow motivate the use of robots as agents that can be taught, whereas
disengaged or unable to progress in the task (Plurkowski, Chu, & Vygotsky's zone of proximal development (Vygotsky, 1930) and
Vinkhuyzen, 2011). The aim of this paper is to explore the chal- principles of scaffolding (Wood & Wood, 1996) are common un-
lenges that currently exist when moving robotic tutors into actual derpinnings for robots that can tutor children. Social robots can be
classrooms by focusing specifically on breakdowns in children's used in either formal or informal educational settings, such as
interactions with a robotic tutor at their school. Video recordings of classrooms or children's after school venues, but this is typically
such instances are analyzed in-depth through qualitative methods, dependent on the subject content. Subject content and learning
guided by the following research question: What causes break- activities also vary (e.g., in the current study, the focus is on ge-
downs in children's interactions with a robotic tutor, and what con- ography and sustainable development). Yet, as mentioned previ-
sequences do such breakdowns pose for the educational use of robots? ously, when robots are designed to feature in tutoring roles, they
usually tutor participants in structured activities. Examples of such
2. Related work activities include nonogram puzzles (Leyzberg et al., 2014), chess
(Leite, Martinho, Pereira, & Paiva, 2009), prime number identifi-
In this section, the use of robots in education is presented in cation games (Kennedy et al., 2017), wooden block games aiming to
brief with a particular focus on social robotics. Then, previous teach children to count in a foreign language (Vogt, de Haas, de
research findings relating to breakdowns in HCI is discussed. Jong, Baxter, & Krahmer, 2017), and foreign word learning games
(Gordon et al., 2016).
2.1. Social robots in education
2.2. Breakdowns
Social robots are physical, autonomous artifacts that interact
and communicate with humans through human social mecha- As Bødker (1995) points out, “[a]n artifact works well in our
nisms, such as natural speech and social cues (Breazeal, 2003; activity if it allows us to focus our attention on the real object and
Edwards, Edwards, Spence, Harris, & Gambino, 2016). As badly if it does not” (p. 148). When something happens that dis-
explained by Edwards et al. (2016), “social robots overlap in form rupts the flow of the task, there is a risk that the interaction breaks
and function with human beings to the extent that their locally down as a result. In HCI, breakdowns can occur when a person's
controlled performances occupy social roles and fulfill relationships process of using a computer application becomes interrupted by
252 S. Serholt / Computers in Human Behavior 81 (2018) 250e264

something occurring within the application, e.g., if a tool behaves more experienced other. In a CRI setting, a robotic tutor is thought
unexpectedly (Bødker, 1995). Suddenly, the person becomes aware to fulfill such a function. Also, provided that breakdowns are rela-
of the tool itself rather than the task that he/she was initially doing tively quickly overcome, Iacovides et al. (2015) conclude that they
(Urquijo, Scrivener, & Palme n, 1993). In some cases, this can be are not detrimental for the player's involvement in the game.
resolved swiftly through repair strategies, in which case it can be Nevertheless, they also refute the proposition that all breakdowns
regarded as temporary trouble (Jordan & Henderson, 1995; lead to learning; instead, they argue that breakdowns lead to
Plurkowski et al., 2011). In other cases, the problems remain un- learning provided there is subsequent breakthrough in under-
resolved, leading to “the breakdown of interaction and participants' standing, which is not always the case (Iacovides et al., 2015). Even
disengagement” (Plurkowski et al., 2011, p. 61), which is a sign of if players progress in the game, it will not be as satisfying if they do
usability failure (Urquijo et al., 1993). not understand why (Iacovides et al., 2015).
According to Sharp, Rogers, and Preece (2007), there are several The current study draws on this previous research in order to
aspects to a usable technology. First, its effectiveness, i.e., how well identify and interpret the indications and causes of breakdowns in
it fulfills what it is supposed to do. Hence, a robotic tutor should be children's interactions with a robotic tutor, so that their implica-
good at tutoring. Second, its efficiency, i.e., the way in which the tions can be discussed in relation to the educational use of robots.
robot is able to support children in accomplishing their tasks However, given the social nature of interaction with robots, in
without having to carry out time-consuming and arbitrary action addition to the fact that learning is not necessarily an efficient
sequences unrelated to the task at hand. Third, its safety, such that enterprise in the traditional sense of the word, it is not straight-
it does not subject children to danger or undesirable situations. This forward how usability and breakdowns within the context of HCI is
includes designing the interface in a way that reduces the risk of applicable for CRI. It is possible that the HCI literature is primarily
serious error, and also provides the opportunity to recover from relevant for what occurs within the learning activities themselves,
mistakes made. Fourth, its utility, i.e., the extent to which the robot whereas breakdowns between the child and the robot may have to
(and application) is able to provide the functionalities necessary to be viewed in a slightly different light. The importance here may lie
accomplish the tasks. Fifth, its learnability, i.e., the ease of learning in how well the robot is able to support children in avoiding
how to interact with the robot and carry out the tasks. And finally, breakdowns, or alternatively, to use breakdowns as teaching
the memorability, i.e., the ease of remembering how to interact moments.
with the robot and use the touchtable once it has been learned
(Sharp et al., 2007). 3. Method
To recognize the occurrence of breakdowns, a variety of in-
dicators can be considered. In their multiple case study of break- In order to study CRI in an authentic setting, a field trial was
downs and breakthroughs in the context of adults playing a variety conducted at a primary school in Sweden, where the educational
of console games, Iacovides et al. (2015) considered breakdowns as robot setup was implemented for 6 months, of which 13 weeks
belonging to different categories comprising different indicators; were active experimental days. The setup was placed in close vi-
breakdowns in player actions, player understanding, or player cinity to one of the classrooms, in a small ‘group-room’ that the
involvement. In terms of actions, these could occur if the player students were familiar with using for other educational activities
failed to carry out an action successfully, e.g., if he/she pressed the such as individual study or group work. The students took turns in
wrong button. For breakdowns in understanding, these could occur carrying out repeated interaction sessions with the robot both
if the player was unsure about what to do. In terms of involvement, individually and in pairs. A researcher was present at the school for
negative affective states such as frustration or boredom were the whole duration of the study.
considered to be clear signals that the player's engagement had
been disrupted. Further, in their study of mobile learning applica- 3.1. Robot and educational applications
tion called Myartspace, Vavoula and Sharples (2009) considered
moments when the student either seemed to be struggling with the The technical setup was developed as part of the EU FP7 EMOTE
technology, requested help, or was working under a clear misun- project (EMbOdied perceptive Tutors for Empathy-based learning),2
derstanding to be indications of breakdowns. with the overall research aim to study whether tutoring robots
Previous research suggests that there are a number of reasons with perceptive and empathic capabilities can facilitate social
for breakdowns to occur within the context of playing games, e.g., if connections between students and robots, and subsequently facil-
the game does not meet the player's expectations (Iacovides et al., itate learning (Castellano et al., 2013). As can be seen in Fig. 1, the
2015; Ryan & Siegel, 2009), or if an action carried out by the player technical setup consisted of a robotic tutor (NAO T14 from Soft-
results in negative consequences (Iacovides et al., 2015). Break- bank), and an interactive touchtable for the educational activities
downs have also been shown to occur if players attempt to carry (55” touch-sensitive interactive display from MultiTaction). Also a
out strategies that are not supported within the game (Barr, 2017). Microsoft Kinect 2.0, as well as a web camera connected to OKAO
A lack of initial engagement in the narrative or game mechanics, vision software were used for analyzing the emotional states of the
can result in further breakdowns in actions and understanding students. Two video cameras (a camcorder and a GoPRO) as well as
(Iacovides et al., 2015). Moreover, Iacovides et al. (2015) found that a Zoom H4N with a connected external overhead microphone were
the player's sense of agency is important for maintaining involve- used to record the sessions.
ment. If players feel that there are recurring problems with game The robot could speak through a text-to-speech engine (TTS),3
controllers, or if outcomes within the game are considered unfair point to the screen, produce gestures with its arms, gaze around,
and inconsistent, players may lose their sense of agency and sub- and display different eye colors (Greczek, Swift-Spong, & Mataric,
sequently their desire to continue playing the game. 2011). In terms of perception, the robot was equipped with face
Yet breakdowns do not necessarily have to be viewed in a tracking, facial expression recognition to determine emotional
negative light. According to Ryan and Siegel (2009), breakdowns states, as well as functionality to register and interpret students'
are also teaching moments where a mentor or tutor can engage
with the student and discuss their misunderstandings and pre-
conceived expectations. Here, the student can develop new stra- 2
www.emote-project.eu.
3
tegies to work through the problem in other ways with the help of a An artificial child's voice from www.acapela-group.com.
 S. Serholt / Computers in Human Behavior 81 (2018) 250e264 253

reading tools was available within the task at students' disposal: a

compass, a map key and a measuring tool. The robot's tutoring
strategy consisted of posing and repeating questions (Graesser,
Wiemer-Hastings, Wiemer-Hastings, & Kreuz, 1999), hints
(Graesser et al., 1999), keywords (Anghileri, 2006; Parson, 1998),
providing elaborations and tutorials on difficult concepts (Graesser
et al., 1999), and different forms of positive and constructive
feedback (Hattie & Timperley, 2007). Negative feedback was
omitted in order to avoid lowering students' intrinsic motivation
(Deci, Vallerand, Pelletier, & Ryan, 1991), in addition to the UCD
process revealing that teachers never conveyed strictly negative
feedback when tutoring students on a paper-based version of the
map task. Finally, if all forms of scaffolding failed, the robot could
reveal/splice in the correct answer and move on to the next step.
Each step instruction in the trail was delivered verbally by the
robot while also being visible on the screen until the step was
completed. A step instruction contained three elements (map
symbol, cardinal direction, and distance). For each step, only one
map symbol corresponded to all of these, whereas all other sym-
bols were incorrect in at least one of the constraints. A task step was
Fig. 1. Technical setup.
designed to heed to a specific trajectory (see Table 1 for a fictive
example).
input on the interactive touchtable. As the task progressed, the difficulty level increased. Several
Two activities could be carried out with the robot, one of which different trails with accompanying backstories were developed to
was an individual map reading scenario, and the other was a sus- support a longitudinal study. These were of varying difficulty levels
tainability scenario that students played in pairs. Both activities (Jones et al., 2015), and selected based on students' earlier perfor-
were designed through a User-Centered Design (UCD) approach mances. The easy, medium, and difficult trails contained 10, 13, or
with teachers and students (Jones et al., 2015), utilizing mock-up 16 steps, respectively.
prototypes (Alves-Oliveira et al., 2014), Wizard of Oz-studies At specific steps in the trail, a pop-up window appeared, which
(Sequeira et al., 2016), and drawing on principles of scaffolding displayed information in the form of clues that the student was to
(Wood & Wood, 1996). While the sustainability scenario was based use at the end of the trail to find a hidden location. Three clues in
on an existing single-player game where the goal was to build a total could be found within each trail. These could e.g. be: (1) The
sustainable city, the map reading scenario was developed within treasure is west of a museum, (2) The treasure is 100 m from a parking
the EMOTE project. The system architecture comprised a set of lot, and (3) The treasure is buried in a forest. The clues were saved in a
modules which constituted the basis for the interaction capabilities scrapbook (not visible in Fig. 2, but placed to the right of the
of the robot within the two scenarios. For a technical elaboration on measuring tool). During this phase, the robot reminded the student
the modules used, see Hall et al. (2016) regarding the map reading to read the clues carefully, which the student needed to combine in
activity, and Alves-Oliveira et al. (2016) for the sustainability game. order to figure out the correct location. To provide an answer, the
student was to drag and drop a star symbol at the correct place. A
3.1.1. Map reading scenario (S1) pop-up window then appeared which requested confirmation
The goal in the map reading scenario (henceforth referred to as before the robot delivered feedback (either e.g., “Well done!” or “I
S1) was to progress along a trail on a local city map by selecting can see how you are thinking here, but I suggest you check the clues
correct map symbols (see Fig. 2 for an illustration). A set of map again.”).

3.1.2. Sustainability scenario (S2)

The sustainability scenario (henceforth referred to as S2) was a
collaborative game where two students and the robot worked
together in order to construct a sustainable city (see Fig. 3 for the
game interface). The players took turns in either building a con-
struction, performing upgrades on existing constructions (to make
them more energy efficient, for example), or implementing a policy
for their city. Personal menus appeared to indicate whose turn it
was, while the robot sometimes announced whose turn it was
(either its own or a particular student's). All players, including the
robot, could also skip a turn if resources were running out.
To build a construction, the player made a selection in their
menu, selected a construction slot, and pressed a ‘build’ button that
appeared. To perform an upgrade on an existing construction, the
player had to press the specific building in its construction slot,
whereby a set of buttons appeared above the active player's per-
sonal menu with grey buttons. Once selected, the construction was
upgraded and the respective button turned green to indicate this.
Three upgrades could be performed on a given turn. To implement
Fig. 2. Prototype of Scenario 1 interface with an example trail (Note that the black text a policy, the city hall (which was always placed in the city center at
has only been added for illustrative purposes). the beginning of the game) needed to be pressed, whereby a series
254 S. Serholt / Computers in Human Behavior 81 (2018) 250e264

Table 1
Trajectory design of Scenario 1 (fictive example).

Robot Student

“We need to take the bus. Walk to the bus stop 500 m north of here.”
Student opens the map key to find out what a bus stop looks like.
The student selects a bus stop on the map. (Incorrect)
“Is there a tool that you can use to figure out the direction?”
The student activates the compass tool, and looks at it.
“We were supposed to go north.”
The student selects a bus stop to the north, 100 m away. (Incorrect)
“Okay, it's time to try out the measuring tool. Can you press the measuring tool button?” (.) “Good job! The student opens the measuring tool and tries to measure as the
Now, to measure a distance, tap once on the map where you want to measure from, and tap again robot is speaking.
on the place you would like to measure to. To measure further, just keep extending the measuring
line by tapping again. The distance will appear in bold numbers next to the line.” (.) “Just press the
measuring tool button again when you want to close it.”
The student selects the correct symbol, whereby a green circle
appears over the symbol to mark that it is correct.
“Very good job!”

3.2. Procedure

Prior to the study, written parental consent, students' informed

consent, as well as ethical approval were obtained. The researcher
visited each class in advance to explain the aim of the study and
answer any questions. The students were informed that they would
be working on geography and sustainable development with a
robot named Nao, but that they could opt out at any time should
they not wish to continue with the study. They were also informed
that the robot would not be able to understand speech, but that it
would try to help them based on other perceptive capabilities, i.e.,
how they worked on the task and their facial expressions. Finally,
the researcher informed them that the robot would only stay at
their school for a limited time.
Interaction sessions always took place during ordinary lessons
in social studies, geography, or Swedish language lessons, during
Fig. 3. Scenario 2 interface with example of a city in the process of being created. which time the researcher invited individual (S1) or pairs of stu-
dents (S2) to carry out their sessions. The pairs were decided by the
students' teachers, and these were kept consistent across all S2
of buttons appeared above the active player's personal menu. Like sessions. Before each session, students were informed that they
the construction upgrades, each policy turned green when should try to interact with the robot on their own, but that they
implemented. could signal to the nearby researcher if they experienced technical
All players could build energy plants, as well as residential areas and/or other difficulties that they could not solve on their own. The
which increased the population of the city, leading to new levels researcher always initiated the task from inside the room to make
within the game. Each player had a specific role with additional sure that everything was working before stepping out into the
constructions associated to it: the economist was the only one who classroom in order to not disrupt the session. After the sessions,
could build businesses and industries, whereas the environmen- brief informal interviews were held with the students about their
talist could build parks and woods. The robot always featured as the experience.
mayor, able to build recreational facilities, and seeking to balance The study started with S1, which continued for 9 weeks.
the scores within the game. Thereafter, S2 was carried out for 4 weeks. Each S1 session varied in
The scores related to economic, environmental and social length, ranging from 10 to 40 min, depending on how much time
wellbeing. The constructions within the city impacted the amount the student needed to accomplish the task. The S2 sessions were
of money, environmental state, social wellbeing, as well as the limited to 20 min, but the game could end prematurely due to the
natural resources available. The playing time was set to 20 min, but depletion of non-renewable resources (i.e., oil). Each student con-
the game could end earlier if non-renewable natural resources ducted 3e4 sessions of S1, and 1e4 sessions of S2. The time be-
were exhausted. tween each student's sessions could range from two weeks to a
In S2 the robot tried to facilitate discussion between the two couple of days depending on convenience. Factors such as the
students. For example, the robot could explain that it was impor- teacher's preference about which student should work with the
tant to create balance between the economy, the environment and robot at any given time, as well as an attempt to establish a fair
the population's wellbeing when creating a sustainable city. This distribution of turns (i.e., to follow a list) played a role. Student
was expressed in the form of a personal opinion in order to illus- absence from school and students' personal preferences also
trate for students that there may be conflicting perspectives when affected the number of sessions conducted.
it comes to creating a sustainable society (Antle, Warren, May, Fan, Condensed illustrations of interaction sessions are presented in
& Wise, 2014; Gough & Scott, 2003). Fig. 4 (S1) and Fig. 5 (S2).
 S. Serholt / Computers in Human Behavior 81 (2018) 250e264 255

The robot greets the student The robot asks the student The student uses the compass The student uses the
and explains the task. to open and look at the map (overlaid on map center). measuring tool.
key.

The robot asks if the student The student selects a The robot shows enthusiasm The robot explains the final
can find the gas station while correct answer, whereby as a clue appears on the task as a star symbol appears
pointing to the map. the robot says “Good job!” screen. in the center of the map.

Fig. 4. A condensed example of a Scenario 1 session.

The robot greets the Student 2 builds a The robot shows The robot builds a
students and explains the construction enthusiasm about reaching a construction
game. new level.

Student 1 builds a construction. The robot is talking about the roles in the The robot announces that the game
game while the students are interacting has ended because they ran out of
with each other. oil.

Fig. 5. A condensed example of a Scenario 2 session.

3.3. Participants Basedow, Barendregt, & Obaid, 2014). To acquire a broad spread,
the 3 students who scored the highest (most negative), and the 3
All students at the school between the ages of 10e12 (grades who scored the lowest (least negative), on the questionnaire
4e6) were invited to participate in the study, amounting to a total following the study, were included in the analysis. Where there was
of 46 students. To study interactions in-depth over time, a sample a tie in scores, the NARS administered prior to the study was also
of 6 students was selected for analysis in this paper. The selection considered. Here, the students who had altered their attitudes the
criteria were guided by the hypothesis that students' attitudes to- most between the pre- and post-questionnaire were included in
wards robots after the study were affected by the existence and the analysis. The participants were aged 10e12 (M ¼ 10.8,
causes of breakdowns in the interaction sessions. Thus, the sample SD ¼ 0.98, 1 girl and 5 boys). Table 2 details the participants'
was selected based on the students' scores on the Negative Atti- pseudonyms, NARS scores, as well as the duration of video data
tudes Towards Robots Scale (NARS) (Nomura, Kanda, & Suzuki, gathered for each participant. In addition to the sample, each stu-
2006), which has previously been adapted for children (Serholt, dent had a partnering student from their class in S2 who was also
256 S. Serholt / Computers in Human Behavior 81 (2018) 250e264

Table 2
Participant pseudonyms, with NARS score as participant selection criteria (min ¼ 14 [least negative], max ¼ 70 [most negative]).

Participant pseudonym NARS score No. of sessions S1 Video S1 No. of sessions S2 Video S2 Video total

Evan 17 3 1h 20 m 44 s 4 1h 46 m 23 s 3h 07 m 07 s
Katie 19 3 1h 28 m 44 s 4 2h 01 m 31 s 3h 30 m 15 s
James 19 4 1h 50 m 31 s 1 0h 21 m 30 s 2h 12 m 01 s
Finn 40 3 1h 02 m 08 s 4 1h 17 m 09 s 2h 19 m 17 s
Oliver 40 3 1h 18 m 03 s 2 0h 29 m 53 s 1h 47 m 56 s
Felix 48 4 1h 06 m 17 s 2 0h 36 m 27 s 1h 42 m 44 s

part of the analysis for those particular sessions, amounting to a children's interactions with a scripted robotic tutor, and discuss the
total of 12 students for S2 (M ¼ 11, SD ¼ 0.85, 1 girl and 11 boys). consequences that this poses for the educational use of robots. In
this study, a total of 41 breakdowns in 6 different children's inter-
3.4. Data collection and analysis action sessions with a robotic tutor were analyzed. During these
breakdowns, a total of 50 breakdown indicators, described in the
Video recordings of the interaction sessions were the primary previous section, were observed. How often the different indicators
source of data in this study, with a specific focus on breakdowns. occurred is summarized in Table 4.
While the video data also contained less severe occurrences of Four main breakdown themes were derived from the thematic
trouble that the students were able to solve on their own, these are analysis (the number of observed breakdowns are provided in
only discussed in relation to the primary analysis of breakdowns. percentages for each theme). These are: (1) the robot's inability to
The video analysis process drew on qualitative methods, namely evoke initial engagement and identify misunderstandings (27%),
thematic analysis (Braun & Clarke, 2006) and interaction analysis (2) confusing scaffolding (10%), (3) lack of consistency and fairness
(Jordan & Henderson, 1995). (20%), and finally, (4) controller problems (44%). While themes 1
First, all interaction videos specified in Table 2 were looked and 2 primarily lead to breakdowns in understanding, themes 3
through in their entirety at least twice for each student. Second, all and 4 mainly lead to breakdowns in the participants' sense of
video segments that were indicative of breakdowns in interaction agency and control. In the following subsections, each theme is
were selected for in-depth analysis. To locate these events, a presented in turn, some of which contain a number of subthemes
number of breakdown indicators was considered, derived from the (indicated by subheadings). Each theme is described and some-
first step in the analysis process. For S2, there was of course an times illustrated using excerpts. The excerpts are produced through
additional student to consider, in which case breakdown indicators verbatim transcription, where (.) is used to indicate short pauses.
could present themselves in either student. The following break- The follow-up interviews are referred to when applicable, and
down indicators were examined: participant pseudonyms (EvaneFelix) are used to denote individual
students.
1. Situations where the student(s) exhibit adverse emotional
states for a duration of more than 5 s, including confusion, 4.1. The robot's inability to evoke initial engagement and identify
frustration, anger, or sadness (verbally or non-verbally). misunderstandings
2. Situations where the student(s) become inactive, or engage in
off-task activities such as playing around (for a duration of more At the beginning of the two scenarios, the robot presented the
than 10 s, and not related to social interaction with the robot). backstory and goals of the activities. It also explained the me-
3. Situations where the student(s) seek external assistance from chanics, including tools, options and menus that the students could
the researcher. use. However, breakdowns could occur if the robot did not catch
the students' attention, or if it simply omitted certain pieces of
Each of the breakdown indicators could occur in either isolation information; hence an inability to evoke the students' engagement.
or combination. Typically, a sequence of these different breakdown In S1, there may have been too much information for the students
indicators was present, such that e.g., an upset student eventually to process, especially given the novelty of the situation, in addition
called on the researcher. to the fact that the students had a hard time understanding what
The selected segments were then extracted, beginning at the the robot said over the noise produced by the motors in its
last step instruction for S1, while segment beginnings could be gesturing arms. When not knowing how to start the task, some
more flexible for S2. The segments were imported into NVivo 11 students tried to address the robot verbally or exhibit clear non-
software for qualitative analysis. Each breakdown segment was verbal signals such as pointing to the screen or shrugging their
coded in terms of its breakdown indicator(s), and analyzed using shoulders, followed by gazing at the robot, expecting it to recip-
interaction analysis (Jordan & Henderson, 1995). The breakdown rocate. James became completely inactive for 12 min, only touching
causes were organized according to themes, where participants' the screen when the screen saver went on. Oliver became very
subjective accounts of the interaction were taken into account by distressed about not knowing what to do (see Table 5 for excerpt).
including the short follow-up interviews at the end of each session Here, it is evident that Oliver did not understand what to do in
in the analysis (see Table 3 for a specification on the number, the task. He exhibited a need to alert the researcher by moving over
duration and distribution of breakdowns and interviews across to the door, but changed his mind quickly. He then decided to call
sessions for each participant). To avoid bias during the analysis for the researcher anyway and opt to ask a very specific question
process, observations and findings put forth by the author were about the measuring tool (rather than admitting that he had no
continuously discussed with two colleagues familiar to the field. idea what to do). When the researcher then left and Oliver was on
his own again, he became distraught with the situation since the
4. Results initial breakdown was never resolved in the first place. The robot
perceived Oliver's emotional state and tried to remedy the situation
The aim of this study was to explore what causes breakdowns in by engaging him in small talk. However, this only seemed to make
 S. Serholt / Computers in Human Behavior 81 (2018) 250e264 257

Table 3
Number, distribution and duration of breakdowns across scenarios (S1/S2), and sessions (1e4) for each participant (- ¼ N/A).

Number of breakdowns per session Length of extracted video segments

Participant pseudonym Scenario 1 Scenario 2 Total Breakdowns (in % of total time) Interviews (in % of total time)

1 2 3 4 1 2 3 4

Evan 6 1 0 e 7 1 1 1 42 m 33 s 79% 23%

Katie 2 0 0 e 5 0 1 0 19 m 20 s 40% 57%
James 3 0 0 0 0 e e e 17 m 55 s 79% 18%
Finn 0 1 0 e 0 0 0 0 12 m 29 s 16% 75%
Oliver 4 0 1 e 0 1 e e 12 m 51 s 43% 56%
Felix 0 0 2 0 2 2 e e 15 m 18 s 42% 56%

Table 4
Breakdown indicators observed during video analysis with a specification of the number of students exhibiting each indicator, as well as the number of occurrences in total.

Breakdown Indicators Specification No. of participants (N ¼ 6) No. of occurrences in whole data corpus

Indicator 1 5 16
Anger 3 5
Confusion 3 4
Sadness 1 2
Frustration 3 5
Indicator 2 3 7
Inactivity 2 3
Playing around 2 4
Indicator 3 6 27
Request for researcher assistance 6 25
Researcher intervenes without request 2 2
Total 6 50

matters worse, and Oliver subsequently decided to call for the had performed one upgrade.
researcher again.4 Taken together, these situations illustrate that These issues were enhanced when the robot was unable to
the robot was unable to evoke the students' engagement, but more evoke students' engagement when providing instructions. For
importantly, it was unable to identify the misunderstandings that example, during the first S2 session for Evan (and Hugo), the
the students needed help with as a result. excerpt in Table 6 illustrates how lack of initial engagement led to
Such breakdowns at the start of the activity could also be misunderstanding.
observed in S2. As mentioned in the method section, there were Here, it is evident that the robot is unable to engage the students
four actions that the players could perform. They could: (1) build a at the beginning of the activity, which led to a series of breakdowns
construction (in their personal menus), (2) perform three upgrades throughout the session. It illustrates the robot's incapacity to direct
(by pressing on existing constructions, followed by buttons that the student's attention towards its instruction (i.e., they were
appeared above their personal menus), (3) implement a policy (by occupied playing around with the interface instead), coupled with
pressing on the city hall, followed by buttons that appeared above its inability to recognize that the students did not pay attention and
their personal menus), and (4) skip a turn e.g., if they needed to understand its instruction (i.e., when Hugo said “I wasn't listening”).
save money (by pressing on a skip-button located below their However, it is also clear that when Evan spotted his personal menu,
personal menus). The robot often said that the students could and was about to say something about it, he was interrupted by the
choose to either build a construction, perform upgrades or imple- robot's statement, making him abstain from further investigation of
ment a policy on their turn (i.e., “We can choose to build a con- the menu. A series of breakdowns in understanding followed this
struction, perform upgrades, or implement a policy. Remember that we initial breakdown, with the session transpiring for quite some time
can only do one of these things at once.”). However, the robot did not without the students knowing how to build a construction, or that
mention exactly how to go about doing so. Breakdowns occurred they could perform three upgrades. Instead, they were busy playing
when the students did not know how to perform these actions. around by zooming in on the application, performing one upgrade
First, when an upgrade had been performed, the button denoting at a time on the constructions built by the robot, followed by
that particular upgrade turned green, whereas the others remained pressing the skip-button. Throughout the session, confusion was
grey. The students generally seemed to believe that the green color verbalized when the robot, e.g., said that it was Evan's turn to play,
signified the possibility to upgrade, whereas the grey signified that or that they were good at building a sustainable city. Here, they
the choice to upgrade was unavailable. This caused them to press responded by mocking the robot, or stating out loud, e.g., “Oh yeah
persistently on the green options, not understanding why nothing I'll play. Oh wait, no I can't!” (Evan). Finally, when all upgrades had
happened. Second, the students were not informed that three up- been performed on the constructions built by the robot, they
grades were possible for each turn, making students wonder requested assistance from the researcher, explaining that they did
whether they were supposed to press the skip-button once they not understand what they were supposed to do. The researcher
then explained the aim, as well as the mechanics of the game.

4
4.2. Confusing scaffolding
The researcher then comforted Oliver for a while until he confirmed that he was
ready to resume the session. The researcher then sat down in a chair in the room
and monitored the session until Oliver confirmed that it was okay for her to leave The robot could also deliver irrelevant and confusing scaffolding
(after 10 min). in relation to both the subject content, as well as the mechanics of
258 S. Serholt / Computers in Human Behavior 81 (2018) 250e264

Table 5
Excerpt 1 illustrating the robot's inability to evoke initial engagement and identify misunderstandings.

Actor Verbal Nonverbal S1 application

Robot: (The robot reads the first step in S1) An orange circle appears to
Follow the road 120 m south to the tourist center. indicate the start position.
Oliver: Oliver moves around and gazes intermittently at the screen
and surrounding room. He shifts back and forth towards the
door and the table.
Robot: I suggest you check the map key again. Robot waves its arm towards the tool buttons.
Oliver: Oliver moves over to the tools.
Oliver extends finger towards one of the buttons, but draws
it back quickly.
Oliver moves over to the door and looks into the classroom
briefly.
Oliver returns and presses the compass button. The compass button turns
grey, and a compass is
displayed on the map.
(Reads the step instruction quietly to himself) Oliver looks at the compass and then to the step instruction.
Robot: Can you see this symbol?
Oliver: Oliver moves over to the tool buttons and presses the The measuring tool button
measuring tool. turns grey.
Oliver deactivates the compass button and the measuring Buttons return to default, and
tool button. compass disappears.
Oliver presses the measuring tool again. The measuring tool button
turns grey.
Oliver deactivates the measuring tool again. The measuring tool button
returns to default.
Oliver signals to the researcher by knocking on the door.
Robot: Is there really a tourist center there?
Researcher: Enters room.
Oliver: How does the measuring tool work? Looks at the researcher while pointing to the tool buttons.
Researcher: (Briefly explains how to use the measuring tool.) Researcher demonstrates the use of the measuring tool on The measuring tool is
the table. displayed.
Researcher leaves the room.
Oliver: Oliver looks at the screen and around the room for
approximately 30 s, growing increasingly stressed.
Oliver sits down in armchair in the room, and begins to cry.
Robot: (Perceives the emotional distress and tries to engage Oliver in Oliver looks at the robot, but continues to cry.
small talk)
What is your favorite subject? (.) Mathematics is a good subject
for robots to learn since it's based on special rules. I only know
geography though.
Oliver: Oliver gets up from the armchair and knocks on the door for
the researcher.
I can't do it! (Sniveling) Oliver places his hands over his mouth, and looks towards
the door.
Researcher: Enters the room.
Hey, what's wrong? Approaches Oliver.
Oliver: I can't do it.
Researcher: Oh. The researcher tries to comfort Oliver, who continues to sob
for a few seconds.
Robot: What does a tourist center look like on a map?

the interface. In S1, for instance, the robot sometimes pointed in the in close succession). Thus, the robot began an utterance, aborted it,
wrong direction in relation to where it was referring. The robot began a new utterance, aborted it, and so on and so forth. Even-
could, e.g., say that the student should look at the map reading tools tually, the robot broke down due to an overload of information, and
while pointing to the instructions instead, or hint at where the the students needed to request assistance from the researcher.
correct answer was but wave its arm over a completely different Furthermore, breakdowns occurred during the final task in S1
area. In the first few sessions, students tried to follow these in- when students were to combine the three clues that they had
structions; in time, however, they learned to disregard these gathered during the trail and place a star at the correct location. The
movements from the robot. Although this did not always result in robot's continuous speech seemed stressful for the students in the
breakdowns (i.e., when the confusion exhibited was only brief), sense that they did not have the opportunity to think through the
there were some situations related to confusing scaffolding that problem properly without interruption. Sometimes, the students
did. asked the robot to wait or slow down. For example, during Katie's
For example, when students found a step in the S1 task difficult, first session, she needed to seek external assistance from the
they often times tried to get the answer correct without making use researcher in order to solve the task. While the video data showed
of the map reading tools. Instead, they pressed many incorrect map that she was engaging in private speech, on her way to solving the
symbols in close succession seemingly at random. While the robot task, the robot was not perceptive to this. Instead, it kept inter-
was designed to perceive and respond to random answer attempts, rupting her progression towards solving the task by telling her to
the video data showed that it did not succeed in doing so. Instead, it open the scrapbook. She abided by this, confusing her to such an
tried to guide the students based on each answer attempt that they extent that she needed the researchers' support. In the follow-up
made (which were many, since they were pressing several symbols interview, she stated that the task was “a bit tricky”.
 S. Serholt / Computers in Human Behavior 81 (2018) 250e264 259

Table 6
Excerpt 2 illustrating the robot's inability to evoke initial engagement and identify misunderstandings.

Actor Verbal Nonverbal S2 application

Robot: (Explaining the goal of and mechanics of the game) The students are playing with the interface, shuffling A red circle flashes over
… The buildings are going to affect the different resources in the city, which you it around, zooming in and out. the score meter.
can see there at the bottom of the screen. (.) Yes, I mean the scores within the
bubbles below. Every score shows the current state of the city's resources.
The robot builds a suburb. A suburb appears on the
screen.
Evan: Oh! Evan places his hand over his mouth in a surprised
gesture.
Hugo: Zooms in on the suburb just built.
Robot: To have built a suburb creates balance in this city. Evan's personal menu
appears.
Evan: Presses repeatedly on a construction slot. Zooms out.
Robot: We have a talent for building a well-balanced city.
Evan: (Laughing) Evan imitates the robot's arm movement, dancing
around.
Hugo: (Laughing) Hugo smacks/shoves Evan in a playful manner.
Evan: Ouch! (laughingly) Evan stops dancing, but quickly starts dancing again.
Hugo: Hugo threatens to hit Evan again.
Evan: Ok ok! Evan stops dancing and guards himself against
another blow.
Alright, so what are we supposed to do now then? Both students are moving the screen around.
Hugo: I wasn't listening. (.) Presses on the score meter.
Wait, look.
Evan: Evan gazes at his personal menu, and points to it, but
is interrupted by the robot.
Robot: Remember that we don't have any roles in this game. We are all responsible for
the development of the city.
Both students are pressing different places on the
screen without effect.
Evan: Everyone is feeling well so far. (.) Leans forward and gazes at the robot.
How do you do this? (i.e., play the game) (.) Makes a grimacing expression.
Nao? Hello! Hellooooo!
Hugo: He can't communicate with sound! Places his hand over Evan's mouth and shoves him
(.) Oh wait, he can see. back.
Smiles like he is embarrassed.
Evan: Nao, hey hey! Evan jumps from side to side, waving at the robot.
Hugo: Hey hey! Hugo waves at the robot, too.
Evan: Nao? Evan moves around in front of the robot.
Hugo: I don't understand anything. The students go on to pressing on the screen until an Upgrade buttons appear
upgrade menu finally appears. They perform an above Evan's personal
upgrade. menu.

In relation to this, the robot did not perceive how long the perform actions, and the students did not notice the personal menu
students might have seriously struggled with the final task. For appearing on Eric's turn. Thus, the score meter was mistaken as the
example, when Evan was not able to figure out the correct answer, personal menu.
he was growing notably tired and impatient (i.e., his face reddened,
he started sighing and grunting, and rubbing his face and hair).
When the robot then said, “I suggest you read one clue at a time, nice 4.3. Lack of consistency and fairness
and slowly,” the student expressed frustration towards the robot; he
pressed his hands against his face, walked towards the door as if he Several breakdowns occurred when something happened
wanted to leave the room, but turned back around towards the within the scenarios that was either inconsistent with what the
robot and exclaimed, “Nice and slowly?!” with an angry expression students had encountered earlier, or that the students perceived as
on his face. Eventually, the student requested assistance from the unfair. Here, their sense of agency and control over the situation
researcher, saying, “I can never find it!” The researcher proceeded to was compromised. In S1, this could occur if there were clues
stay and guide him in solving the task. Nevertheless, he was missing from the scrapbook. For example, when Oliver was trying
exclusively positive about the experience in the follow-up to figure out the task, there was only one clue in the scrapbook. The
interview. robot continuously told him to be patient and read the clues, which
In S2, confusing scaffolding was mostly related to the mechanics caused frustration and ultimately required external assistance from
of the game. In the first session for Felix (and Eric), the students the researcher. Furthermore, there was a discrepancy between one
were attentive to the robot's instructions, shifting their gaze to the of the mentioned map symbols in the clue and the name of the
buttons and menus to which the robot was referring. The robot symbol in the map key for the first session (i.e., cultural heritage vs.
then performed its turn, followed by the following statement aimed cultural site [in Swedish: kulturarv vs. kulturminne]). The students
at Eric, “Eric, it's your turn”. Eric feigned surprised fear at the robot often requested assistance from the researcher to help with this.
addressing him by name, followed by the two students laughing In S2, the breakdowns were associated with an inability to keep
hysterically at the robot. Eric then went on to verbalize that he did the session going for the whole 20 min because the robot failed to
not know what to do, and proceeded to press the score meter. explain a critical feature within the game (i.e., the limited oil sup-
Hence, the problem was that the robot was not clear about how to ply). Indeed, an important learning goal in S2 was to understand
that certain natural resources are non-renewable, such as the oil
260 S. Serholt / Computers in Human Behavior 81 (2018) 250e264

reserve. The robot did not explain this from a sustainability Felix pleaded, “Okay Nao, don't be mean now.” However, the robot
perspective, but rather from a game perspective, where scores built a suburb which caused the game to end. Upon realizing this,
expressed in numerals were emphasized as needing to be balanced the students were notably upset:
for the city to prosper. Thus, when the game ended because the oil Eric: “Yeah, it was your fault, Nao!”
ran out, all student teams were at least surprised by this, while in Felix: “You're kidding?! (.) Nao. He is so … Nao! He is so bad at
one case, frustration could be observed as a result (see Table 7 for this!”
excerpt). Here, the students may have felt a loss of control over the sit-
In this excerpt, it is evident that both students were surprised by uation, considered it unfair for the game to end outside their con-
the game ending. In the follow-up interview, the students com- trol, and subsequently directed this frustration at the robot. In the
plained about the game ending, explaining that they did not know follow-up interview, the students explained that they had 76 in
that the oil could run out. Then, the students expressed uncertainty oil prior to the game ending and that they had only built con-
regarding their skills at the game. When asked if they would like to structions that were not supposed to cost any oil (indicated by a
play the game again, both students responded affirmatively, saying minus score next to each construction).
that it was a fun experience. However, Felix said that they were not Other students developed strategies to overrule or influence the
very good at it, while Eric expressed disagreement with this robot's decision-making. Some students reached quickly across the
statement. The researcher explained that, “You can't really be good table and pressed the robot's skip-button upon its turn; others tried
or bad at this sort of game,” which seemed difficult for Felix to accept to verbalize clearly articulated instructions for the robot into the
since the game had in fact ended. microphone equipment in the room. Neither of these strategies
Breakdowns could also occur when the robot interfered with the actually worked to positively influence the robot's decision-
students' strategies for building a sustainable city. In the follow-up making; instead, they typically resulted in breakdowns.
interview for the first S2 session with Felix (and Eric), the students
criticized the robot's lack of cooperation skills, saying, “Nao. (.) He
4.4. Controller problems
doesn't really want to cooperate,” (Felix), whereby Eric filled in,
stating, “He didn't really do what we wanted him to do”. After some
The majority of breakdowns occurred as a result of technical
probing from the researcher, the students explained that Nao only
problems with the controllers, where either the touch-interface
built suburbs (except for one park [which was in fact a market
was difficult to use, or the robot broke down. These subthemes
place]), as well as a few upgrades. The students explained that they
are described in turn in the following subsections.
would have liked Nao to focus more on saving money.
Later on when the students became familiar with the S2 game,
they developed strategies for, on the one hand, maximizing the 4.4.1. Problems with the touch-interface and false signifiers
playing time and, on the other hand, making sure that the scores A substantial amount of trouble in S1 related to the children's
did not drop to a critical level. These strategies were negotiated actions on the table that the application (and hence, neither the
verbally during, and prior to the game. For example, the students robot) acknowledged, causing a substantial amount of uncertainty
could agree that they would skip their turns in order to accumulate about what answers were correct versus incorrect. For example, at
money in the game. Thus, when the robot made a decision upon its times when the students answered a step instruction correctly, the
turn, which either impacted the scores negatively, or expended the robot continued to provide scaffolding to the students, making
oil supply, thereby ending the game, children were notably upset by them second guess their performances, and forcing them to search
this, both during the actual interaction but also expressed in the for alternative solutions. Often times, the students returned to their
follow-up interviews. In Felix's and Eric's second session, the stu- original answers after all other attempts were exhausted (unless
dents recognized that the oil was running out. They therefore the robot spliced in the correct answer), and realized that they had
agreed to skip their turns, but the robot was of course not been correct all along, but the robot did not repair its initial
perceptive to this strategy negotiation. Upon the robot's final turn, misperception in any way. Although this did not necessarily lead to
breakdowns, some students expressed discontent with this in the

Table 7
Excerpt 3 illustrating lack of consistency and fairness.

Actor Verbal Nonverbal S2 application

Eric: Eric builds a forest. A forest is built.

Felix: We should maybe put some over there, and then they can … Felix waves his hand over the forest being built (discusses what they
should build next).
Eric: Yeah.
Game ends and moves to
scoreboard.
The students gaze in surprise at the screen.
Robot: Our game is over, because our city ran out of oil.
Eric: What? (Sniffles and sighs) Eric leans forward on the table frame and places his hand on his
forehead.
Felix: What?! Felix places his hand on his forehead.
Robot: The depletion of natural resources, such as oil, makes the city
unsustainable.
Eric: Whaaat?
Felix: Oil? I didn't know there was any oil. Felix shakes his head
Eric: But we had a bunch of oil!
(.)
The students gaze at each other and start giggling.
What should we … ?
Felix: We bought too much of the oil. (.) What should we do now?
 S. Serholt / Computers in Human Behavior 81 (2018) 250e264 261

follow-up interviews, e.g.: “I pressed several times in this one spot, a scared expression on his face. The whole situation took 40 s before
but he [the robot] said ‘no’, and then I tried pressing others, but then it the researcher intervened. After the session, Oliver opted out of
turned out that it was the one I had pressed first” (Felix). further participation in the study.
After some time, the students grew more assertive in their an-
swers, pressing several times on the symbols they thought were 4.4.2. Problems with the robot
correct. While this helped them to compensate somewhat for the The robot often malfunctioned during the scenarios, which
technical issues, it was of course not a fruitful strategy in the cases almost always required assistance from the researcher to solve.
when the students were persistent about an incorrect answer. The However, the students also tried to get the robot's attention both
opposite situation could also occur, where the application regis- verbally and non-verbally before seeking outside help. For example,
tered incorrect answers as correct, and the robot delivered positive when Finn carried out his second session for S1, he selected the
feedback. correct symbol on the map. A green circle appeared on top of the
Another example of when the touch-function did not work symbol to indicate that it was correct. After a longer-than-usual
correctly became evident when the students tried to use the pause, the robot said, “Good job, Finn”, but it did not move on to
measuring tool or move the star in the final task of S1. Here, the the next step. Finn tried to get the robot's attention and get it to
application did not respond to their touch in an acceptable manner, move on with the activity, by saying “Mm!” and waving his hand in
making such tasks extremely tedious and frustrating. Such situa- front of its face. When this did not work, he called for the
tions did not always result in interaction breakdowns, but e.g., researcher.
during Felix's session, when he had fervently attempted to move In S2, the robot sometimes malfunction such that it stopped
the star symbol for 50 s, and the robot delivered irrelevant scaf- speaking as well as playing its turn. Here, the students were more
folding, he became frustrated and finally resorted to using a com- verbally expressive towards the robot in order to get it to start
puter mouse (mainly used by the researcher) which was connected speaking. For example, in their first session, Katie (and Nathan)
to the table. In the follow-up interview, Felix explained somewhat tried communicating with the robot for about 50 s when it mal-
annoyed that he had been trying to move the star for 10 min before functioned before alerting the researcher. They said, e.g., “Hey Nao!
using the mouse. When asked whether he felt that the robot had It's your turn!” Nathan also tried dancing in front of the robot, as
helped him figure out a task step during the session, he responded, well as feign an angry posture and facial expression, stating, “Let's
“No.” see if he reacts”. If the robot malfunctioned yet again, the students
Issues with the touch-function rarely occurred in S2 since a decided to solve this on their own by pressing the robot's skip-
more sensitive touch-interface software was used for this scenario. button instead of alerting the researcher.
However, there was a non-identified technical problem which Breakdowns could also occur if the robot did not deliver any
caused the turn to sometimes automatically move to the next feedback or closing statement after the activities. Here, the stu-
player even when the skip-button was not pressed. In the second dents were unsure about whether they had accomplished the task
session with Felix (and Eric), this problem occurred three times. (S1), or confused about why a scoreboard was suddenly displayed
Initially, the students considered that they must have pressed the in the game (S2). In one case, a breakdown was researcher induced,
button accidentally, but later on, the students became upset by this where she accidentally started the wrong account for Felix. Here,
and alerted the researcher. the robot addressed the student by the wrong name, and the
Moreover, breakdowns also occurred in relation to the ‘safe’ researcher had to struggle for some time to stop the robot.
usability of the interface, where buttons were placed in a risky
location. For example, when the highest level had been reached in 5. Discussion
S2, the players could still keep playing and improving their city
within the time frame of 20 min. In all cases, students were keen on The results of this study indicate that CRI is not always fric-
playing S2 for as long as possible. Yet, a button appeared at the tionless by any means, perhaps particularly so when a robot is
bottom of the screen when Evan (and Hugo) reached the highest implemented in an actual educational setting. Confined to a sample
level, which read: Quit game. Evan seemed enticed by this and of six participants, 41 breakdowns occurred during the children's
decided to press it while Hugo was in the middle of performing his interactions with the robotic tutor. In more than half of these cases,
turn. This action ended the game, causing Hugo to become frus- the researcher needed to intervene. In addition to the breakdowns
trated at Evan, grabbing a hold of his arm while shouting “No! Ugh!” presented in this paper, there were many occasions in the video
Evan tried to remedy the situation by pressing additional buttons to data where children encountered trouble that did not lead to
return to the game. Unfortunately, it was not possible to do so, and breakdowns. Since this paper only focuses on breakdowns, this
Evan ended up apologizing for this mistake. After some time, they means that there is in fact substantial trouble in children's in-
called for the researcher. When the researcher entered, Hugo teractions with a robotic tutor due to its lack in perception of its
complained about Evan pressing the button when they still had half surrounding environment and the social situation at hand, despite
a session to go. The researcher was surprised that there was an the substantial design and technical work that went into creating
option to quit the game. the EMOTE setup.
In spite of the touch-interface being easier to use in S2 overall, The breakdowns were associated with a variety of causes, some
breakdowns could occur because it was too sensitive. In fact, the of which were similar to breakdowns identified in previous
touch-interface in S2 could respond to hovering movements, research. For example, to explain the mechanics and rules of the
whereas this was not possible for S1. As in the second session for educational activities, the robot needed to devote a substantial
Oliver (and Victor), the students were discussing their strategy, amount of time to elaborate explanations mostly at the beginning
while Oliver was pointing towards the score meter. Victor was of the activity. The crux was that the children were not prone to
occupied listening to Oliver, but had placed his hand close to his listen actively to long instructions conveyed by the robot, which is
skip-button. This hovering movement caused the skip-button to not very surprising given the robot's monotone voice, the age group
activate, whereby the turn went over to the robot. Upon realizing in question, and the novelty of the situation, but they also had
this, Victor became very angry at Oliver and accused him of skip- trouble actually hearing what the robot said because its noisy
ping his turn, which Oliver denied. It was clear that Oliver felt motors were obstructing its speech. Additionally, the robot some-
threatened by the situation as he backed away from Victor and had times failed to explain important information, such as the fact that
262 S. Serholt / Computers in Human Behavior 81 (2018) 250e264

the non-renewable resources in S2 could run out, how to perform application, such as taking control over the robot's personal menu,
upgrades, etc. In other words, the robot was unable to engage the which resulted in additional breakdowns. In terms of safety, the S2
children initially because the robot either explained something interface did not allow children to recover from mistakes, such as if
poorly, or failed to explain it at all; further, the robot was unable to they accidentally pressed their skip-buttons, or pressed the Quit
identify the children's misunderstandings. When this occurred, this game-button (which was clearly a false signifier/design flaw).
led to breakdowns in understanding of the task objectives, as well Taking into account the fact that the current study dealt with
as the line of action required to exercise those objectives (Iacovides two scenarios; both a one childeone robot scenario (S1), and a two
et al., 2015). childreneone robot scenario (S2), the question is how these differ.
Despite the longitudinal nature of the study, none of the The former was more structure-focused whereas the latter was
breakdown indicators could be traced back to signs of boredom more collaboration-focused. This, of course, influenced the social
with the educational activities. From a usability perspective, the dynamics, where S2 allowed for a higher level of peer support in
educational applications posed no issues in terms of learnability the face of trouble. Yet, this also introduced several challenges. For
and memorability. Once children learned the different features, example, the children tended to interact differently with the robot,
these were readily used in upcoming sessions. The problem was largely ignoring it in S2, or even mocking it. They exhibited a higher
that the robotic tutor was not very effective in teaching the children level of acceptance for the robot's guidance when alone, and
the basic features. Thus, these problems were associated with the became adversarial when with a peer. Further, the video data
robot's ineffectiveness, i.e., the robot's inability to explain, perceive, revealed that there was sometimes an unequal power balance be-
and respond to the children's difficulties in an effective way. For tween the two children. In these cases, one child could take charge
example, when inactivity occurred in S1, this was always associated over both children's turns, pushing the other child's hand away
with students not understanding what they were supposed to do. from his/her menu or otherwise obstructing the other child's in-
Here, the robot sometimes delivered confusing scaffolding, which fluence in the decision-making process. These sorts of social dy-
led to further breakdowns in understanding and involvement. The namics turned out to be impossible for the robot to perceive. This
child could, e.g., engage in random tapping on the screen to quickly can be discussed from a long-term perspective, i.e., if robotic tutors
arrive at the correct answer without any understanding. Such would be implemented in classrooms in the future without a
strategies on the part of the child are hindrances, rather than fa- researcher (or teacher) ready to intervene. In an interview study
cilitators, for learning the subject at hand. conducted with teachers during the design process of the robot,
Furthermore, while the video data revealed that the robot often one teacher voiced a concern about this very issue, stating that “one
delivered relevant scaffolding, e.g., by reminding the child to check student can take over and be very controlling if it's a game for example,
the compass tool if he/she selected the correct symbol in the wrong and be very dominant. In those cases it's possible for me as a teacher to
direction, it did not perceive the more complex difficulties related intervene and even this out. Is that something that the robot will be
to, e.g., distance measuring. When the task increased in difficulty, able to do?” (Serholt, Barendregt, et al., 2014). Thus, there is a clear
the distances were expressed in kilometers rather than meters. discrepancy between teachers' expectations of the state of the art,
Since the measuring tool only displayed the measured distance in on the one hand, and what Selwyn (2008) refers to as the state of
meters, this required the child to convert kilometers into meters. the actual, on the other, and this implies several consequences for
This was not made clear by the robot, which led to certain diffi- the educational use of robots.
culties in determining the correct distance. The robot was not
perceptive to this, and rather than explaining how meters relate to 5.1. Limitations
kilometers, the robot provided the correct answer to the child.
As argued by Iacovides et al. (2015) in relation to game-playing, There were several limitations to this study. First, the study was
progression is only satisfying if the player understands why they confined to one specific school in Sweden, with the cultural and
progressed. From a teaching and learning perspective, such diffi- social background inherent there. It is likely that different sites
culties are important teaching moments, where a tutor can engage could have provided different results. Also, the participant sample
with a child to sort out their misunderstandings and formulate new size was relatively small, including only six students. However, the
strategies (Ryan & Siegel, 2009). However, it is clear that the robot aim of the study was not to generalize across the population as
only perceived the fact that the child answered incorrectly, but not such, but rather to provide an in-depth perspective on the break-
why, which is a central competence for a skilled teacher. Thus, the downs that may occur in CRI. Here, it is possible that such things as
robot never dealt with this learning opportunity directly, which the participants' prior knowledge and academic proficiency played
makes it plausible to assume that this was not very satisfying for a role. Furthermore, there was an uneven distribution of gender in
the children. More importantly, these were learning moments that the sample, and it is possible that if there had been more female
were either lost completely or inflicted on the researcher to explain participants, additional breakdowns could have been found. How-
at a later time (provided the children actually asked the researcher ever, it should be mentioned that the selection criteria (i.e., nega-
about it). tive attitudes towards robots) which yielded more boys in the
Breakdowns that jeopardized children's sense of agency were sample, does not imply that the girls necessarily had a more neutral
frequent in the material (Iacovides et al., 2015). These could be attitude towards robots. Indeed, an independent-samples t-test
related to situations where there was a perceived lack of consis- was conducted to compare NARS scores for boys and girls, which
tency and fairness in the activities, or where there were controller showed that there was not a significant difference in the scores for
problems; either the touch-interface did not respond, or seemed to males (M ¼ 29.00, SD ¼ 8.6) and females (M ¼ 29.78, SD ¼ 7.5); t
have a ‘mind of its own’, the robot malfunctioned in the middle of (25) ¼ 0.231, p ¼ .820.
the task, or children pressed buttons on the interface either
intentionally or accidentally which led to negative consequences. In 6. Conclusion
support of previous research, breakdowns occurred when chil-
dren's strategies were not supported by the applications (Barr, There are many visions for the educational benefit of CRI, such
2017), e.g., if they had a vision for building a sustainable city that as robots holding the ability to interact with children in the real
they could not communicate to the robot in a meaningful way. This physical world (Castellano et al., 2013), facilitating learning (Kory
often led to frustration and strategies not supported by the Westlund et al., 2017), personalizing education (Leyzberg et al.,
 S. Serholt / Computers in Human Behavior 81 (2018) 250e264 263

2014), and so on and so forth. Belpaeme et al. (2013) even assert Acknowledgements
that there are technical ways of circumventing a robot's lack in
perception such that children can be fooled into believing that a 1 I thank all the students, teachers and staff at the participating
robot actually understands them similar to how a human interac- school for taking part in the study. I also thank my collaborating
tant would. However, when looking at the state of the actual, it is partners and the technical team in the EMOTE project for making
clear that CRI in education is not as straightforward. Fooling chil- this study possible. Thanks to Johan Lundin and Wolmet Barendregt
dren into believing that a robot understands them does not offer for providing valuable input during the analysis, as well as your
much consolation when there is a structured task to be carried out, comments on earlier drafts of this paper.
or a fruitful collaboration to be upheld. Like in HCI (Iacovides et al.,
2015; Ryan & Siegel, 2009), CRI breaks down when expectations go
unmet, i.e., that robots should have humanlike perception and References
communication abilities. Of course, structured tasks common to CRI
Alves-Oliveira, P., Janarthanam, S., Candelas, A., Deshmukh, A., Ribeiro, T., Hastie, H.,
scenarios are helpful for increasing the prospect of robots et al. (2014). Towards dialogue dimensions for a robotic tutor in collaborative
perceiving and responding socially, but even then, it is evidently learning scenarios. In Proceedings of the 23rd international symposium on robot
not enough. and human interactive communication (RO-MAN'14) (pp. 862e867).
Alves-Oliveira, P., Sequeira, P., & Paiva, A. (2016). The role that an educational robot
Trouble in interaction is not unique for CRI or HRI. It also occurs plays. In Proceedings of the 25th IEEE international symposium on robot and
in HHI all the time. The difference is that HHI (or teacherestudent human interactive communication (RO-MAN) (pp. 817e822).
interaction, if you will) is not scripted; instead, interaction between Anghileri, J. (2006). Scaffolding practices that enhance mathematics learning.
Journal of Mathematics Teacher Education, 9(1), 33e52.
human subjects is eloquently negotiated. Repair work is a funda- Antle, A. N., Warren, J. L., May, A., Fan, M., & Wise, A. F. (2014). Emergent Dialogue:
mental part of this process (Nakamura, 2008). The challenge in Eliciting values during Children's collaboration with a tabletop game for
robot design is that robots feature somewhere on the border of change. In Proceedings of the interaction design and children conference, Aarhus,
Denmark (pp. 37e46).
what is, on the one hand, a computer interface, and on the other, a Barr, P. (2017). Video game values: Play as human-computer interaction. Wellington,
human actor (Ho € flich, 2013; Zhao, 2006; van Oost & Reed, 2011). New Zealand: Victoria University of Wellington.
Given the robot as a proficient participator in a learning activity, its Belpaeme, T., Baxter, P., de Greeff, J., Kennedy, J., Read, R., Looije, R., et al. (2013).
Child-robot Interaction: Perspectives and challenges. In G. Herrmann,
lack of social interaction skills, non-existent cooperation skills, M. Pearson, A. Lenz, P. Bremner, A. Spiers, & U. Leonards (Eds.), Social robotics
while its humanoid appearance provided indications of the con- (Vol. 8239, pp. 452e459). Springer International Publishing.
trary, the whole interaction situation became paradoxical. As this Benitti, F. B. V. (2012). Exploring the educational potential of robotics in schools: A
systematic review. Computers & Education, 58(3), 978e988.
study demonstrates, children expected the robot to be able to
Bødker, S. (1995). Applying activity theory to video analysis: How to make sense of
interpret their intentions, much like human teachers do. When the video data in human-computer interaction. In B. A. Nardi (Ed.), Context and
robot was unable to do so, children either tried to compensate for consciousness (pp. 147e174). Massachusetts Institute of Technology.
this themselves, or the interaction broke down. Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative
Research in Psychology, 3(2), 77e101.
Looking ahead, it is possible that children will get used to the Breazeal, C. (2003). Toward sociable robots. Robotics and Autonomous Systems,
idea that robots are simply something else, neither human nor 42(3e4), 167e175.
computer, but machines interacting on their own terms. But the Castellano, G., Paiva, A., Kappas, A., Aylett, R., Hastie, H., Barendregt, W., et al. (2013).
Towards empathic virtual and robotic tutors. In H. C. Lane, K. Yacef, J. Mostow, &
question is why we should give children the wrong impression in P. Pavlik (Eds.), Artificial intelligence in education (Vol. 7926, pp. 733e736).
the first place? One clear argument for this is that robots create Springer Berlin Heidelberg.
better learning effects than, say, virtual agents (Leyzberg, Castellano, G., Pereira, A., Leite, I., Paiva, A., & McOwan, P. W. (2009). Detecting user
engagement with a robot companion using task and social interaction-based
Spaulding, Toneva, & Scassellati, 2012). However, these effects are features. In Proceedings of the international conference on multimodal interfaces
within the limitations of a scripted task. Indeed, if the goal is to (ICMI-MLMI’09), Cambridge, MA, USA (pp. 119e126).
avoid breakdowns in children's interactions with robots as long as Chase, C. C., Chin, D. B., Oppezzo, M. A., & Schwartz, D. L. (2009). Teachable agents
and the Prote ge
 Effect: Increasing the effort towards learning. Journal of Science
robots are unable to use breakdowns as teaching moments, then a
Education and Technology, 18(4), 334e352.
robot should only be able to tutor students on simple and Deci, E. L., Vallerand, R. J., Pelletier, L. G., & Ryan, R. M. (1991). Motivation and
straightforward tasks. However, it is questionable whether this Education: The self-determination perspective. Educational Psychologist, 26(3 &
4), 325e346.
poses any true benefit from a pedagogical perspective. If, instead,
Duffy, B. R. (2003). Anthropomorphism and the social robot. Robotics and Autono-
one chooses to design a complex task like in the current study, mous Systems, 42(3e4), 177e190.
technical limitations in current technology in terms of perception Edwards, A., Edwards, C., Spence, P. R., Harris, C., & Gambino, A. (2016). Robots in
will likely lead to a robot not being able to provide the guidance the classroom: Differences in students' perceptions of credibility and learning
between “teacher as robot” and “robot as teacher”. Computers in Human
necessary to facilitate learning. This invariably presents an impasse, Behavior, 65, 627e634.
where the only solution is to wait and hope that robots someday Fior, M., Nugent, S., Beran, T. N., Ramirez-Serrano, A., & Kuzyk, R. (2010). Children's
reach an adequate level of intelligence to play such a role (if that is relationships with Robots: Robot is Child's new friend. Journal of Physical Agents,
4(3), 9e17.
what education needs). In the current state of the actual, they Fong, T., Nourbakhsh, I., & Dautenhahn, K. (2003). A survey of socially interactive
certainly do not. robots. Robotics and Autonomous Systems, 42, 143e166.
Fridin, M., & Belokopytov, M. (2014). Acceptance of socially assistive humanoid
robot by preschool and elementary school teachers. Computers in Human
Funding Behavior, 33, 23e31.
Gordon, G., Spaulding, S., Kory Westlund, J., Lee, J. J., Plummer, L., Martinez, M., et al.
This work was supported by the European Commission [grant (2016). Affective personalization of a social robot tutor for Children's second
€stra Go€taland language skills. In Proceedings of the 30th AAAI conference on artificial intelligence
number EU FP7 ICT-317923]; and the Region Va (pp. 3951e3957).
[project DigitaL]. The author is solely responsible for the content of Gough, S., & Scott, W. (2003). Sustainable development and Learning: Framing the
this publication. It does not represent the opinion of the funding issues. London, UK: RoutledgeFalmer.
Graesser, A. C., Wiemer-Hastings, K., Wiemer-Hastings, P., & Kreuz, R. (1999).
bodies, and the funding bodies are not responsible for any use that
AutoTutor: A simulation of a human tutor. Cognitive Systems Research, 1(1),
might be made of data appearing therein. 35e51.
Greczek, J., Swift-Spong, K., & Mataric, M. J. (2011). In Using eye shape to improve
Declarations of interest affect recognition on a humanoid robot with limited expression. University of
Southern California, Comp. Sci. Department.
Hall, L., Hume, C., Tazzyman, S., Deshmukh, A., Janarthanam, S., Hastie, H., et al.
The author declares no conflict of interest. (2016). Map reading with an empathic robot tutor. In Paper presented at the
264 S. Serholt / Computers in Human Behavior 81 (2018) 250e264

International Conference on Human-Robot Interaction (Extended abstracts), interaction. Language Teaching Research, 12(2), 265e283.
Christchurch, New Zealand. Nomura, T., Kanda, T., & Suzuki, T. (2006). Experimental investigation into influence
Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational of negative attitudes toward robots on humanerobot interaction. AI & Society,
Research, 77(1), 81e112. 20(2), 138e150.
Ho€ flich, J. R. (2013). Relationships to social Robots: Towards a triadic analysis of Nugent, G., Barker, B. S., & Grandgenett, N. (2012). The impact of educational ro-
media-oriented behavior. Intervalla: Platform for Intellectual Exchange, 1, 35e48. botics on student STEM learning, attitudes, and workplace skills. In B. S. Barker,
Hyun, E., Yoon, H., & Son, S. (2010). Relationships between user experiences and G. Nugent, N. Grandgenett, & V. I. Adamchuk (Eds.), Robots in K-12 education: A
children's perceptions of the education robot. In Proceedings of the 5th ACM/IEEE new technology for learning (pp. 186e203). IGI Global.
international conference on Human-robot interaction, Osaka, Japan (pp. van Oost, E., & Reed, D. (2011). Towards a sociological understanding of robots as
199e200). companions. In M. Lamers, & F. Verbeek (Eds.), Human-robot personal re-
Iacovides, I., Cox, A. L., McAndrew, P., Aczel, J., & Scanlon, E. (2015). Game-play lationships (Vol. 59, pp. 11e18). Springer Berlin Heidelberg.
breakdowns and Breakthroughs: Exploring the relationship between action, Papert, S. (1980). Mindstorms: Children, computers, and powerful ideas. Basic Books,
understanding, and involvement. Human-Computer Interaction, 30(3e4), Inc.
202e231. Parson, M. L. (1998). Focus student attention with verbal cues. Strategies, 11(3),
Jones, A., Küster, D., Basedow, C., Alves-Oliveira, P., Serholt, S., Hastie, H., et al. 30e33.
(2015). Empathic robotic tutors for personalised learning: A multidisciplinary Plurkowski, L., Chu, M., & Vinkhuyzen, E. (2011). The implications of interactional
approach. In Proceedings of the 7th international conference on social robotics, "repair" for human-robot interaction design. In Proceedings of the IEEE/WIC/ACM
Paris, France (pp. 285e295). international conferences on web intelligence and intelligent agent technology (pp.
Jordan, B., & Henderson, A. (1995). Interaction Analysis: Foundations and practice. 61e65).
The Journal of the Learning Sciences, 4(1), 39e103. Ryan, W., & Siegel, M. A. (2009). Evaluating interactive entertainment using
Kahn, P. H., Friedman, B., Perez-Granados, D. R., & Freier, N. G. (2004). Robotic pets in breakdown: Understanding embodied learning in video games. In Paper pre-
the lives of preschool children. Paper presented at the CHI '04 extended abstracts on sented at the breaking new ground: The fourth digital games research association
human factors in computing systems. Vienna, Austria. conference, London, UK.
Kanda, T., Hirano, T., Eaton, D., & Ishiguro, H. (2004). Interactive robots as social Selwyn, N. (2008). From state-of-the-art to state-of-the-actual? Introduction to a
partners and peer tutors for children: A field trial. Human-Computer Interaction, special issue. Technology, Pedagogy and Education, 17(2), 83e87.
19, 61e84. Serholt, S., & Barendregt, W. (2016). Robots tutoring children: Longitudinal evalu-
Kanda, T., Sato, R., Saiwaki, N., & Ishiguro, H. (2007). A two-month field trial in an ation of social engagement in child-robot interaction. In Proceedings of the 9th
elementary school for long-term human-robot interaction. IEEE Transactions on Nordic Conference on Human-Computer Interaction (NordiCHI'16), Gothenburg,
Robotics, 23(5), 962e971. Sweden.
Kennedy, J., Baxter, P., & Belpaeme, T. (2015). The robot who tried too Hard: Social Serholt, S., Barendregt, W., Leite, I., Hastie, H., Jones, A., Paiva, A., Vasalou, A., et al.
behaviour of a robot tutor can negatively affect child learning. In Proceedings of (2014). Teachers' views on the use of empathic robotic tutors in the classroom.
the 10th annual ACM/IEEE international conference on human-robot interaction, In Proceedings of the 23rd IEEE international symposium on robot and human
Portland, Oregon, USA (pp. 67e74). interactive communication (RO-MAN'14), Edinburgh, UK (pp. 955e960).
Kennedy, J., Baxter, P., & Belpaeme, T. (2017). The impact of robot tutor nonverbal Serholt, S., Basedow, C., Barendregt, W., & Obaid, M. (2014). Comparing a humanoid
social behavior on child learning. Frontiers in ICT, 4(6). tutor to a human tutor delivering an instructional task to children. In Pro-
Kennedy, J., Baxter, P., Senft, E., & Belpaeme, T. (2016). Social robot tutoring for child ceedings of the 2014 IEEE-RAS international conference on humanoid robots,
second language learning. In Paper presented at the 11th annual ACM/IEEE in- Madrid, Spain (pp. 1134e1141).
ternational conference on human-robot interaction. Sequeira, P., Alves-Oliveira, P., Ribeiro, T., Tullio, E. D., Petisca, S., Melo, F. S., et al.
Kennedy, J., Lemaignan, S., & Belpaeme, T. (2016). The cautious attitude of teachers (2016). Discovering social interaction strategies for robots from restricted-
towards social robots in schools. In Paper presented at the 11th ACM/IEEE in- perception wizard-of-Oz studies. In Proceedings of the 11th ACM/IEEE Interna-
ternational symposium on robot and human interactive Communication: Robots 4 tional Conference on Human Robot Interaction, Christchurch, New Zealand (pp.
learning workshop New York, USA. 197e204).
Kim, C., Kim, M. K., Lee, C., Spector, J. M., & DeMeester, K. (2013). Teacher beliefs and Sharp, H., Rogers, Y., & Preece, J. (2007). Interaction Design: Beyond human-computer
technology integration. Teaching and Teacher Education, 29(0), 76e85. interaction (2nd ed.). John Wiley & Sons.
Kory Westlund, J. M., Dickens, L., Jeong, S., Harris, P. L., DeSteno, D., & Breazeal, C. L. Tanaka, F., Cicourel, A., & Movellan, J. R. (2007). Socialization between toddlers and
(2017). Children use non-verbal cues to learn new words from robots as well as robots at an early childhood education center. Proceedings of the National
people. International Journal of Child-Computer Interaction, 13, 1e9. Academy of Sciences, 104(46), 17954e17958.
Lee, E., Lee, Y., Kye, B., & Ko, B. (2008). Elementary and middle school teachers', Tanaka, F., & Matsuzoe, S. (2012). Children teach a care-receiving robot to promote
students' and parents' perception of robot-aided education in Korea. In Pro- their Learning: Field experiments in a classroom for vocabulary learning.
ceedings of the conference on educational multimedia, hypermedia and telecom- Journal of Human-Robot Interaction, 1(1), 78e95.
munications (pp. 175e183). Teo, T. (2011). Factors influencing teachers' intention to use technology: Model
Leite, I., Castellano, G., Pereira, A., Martinho, C., & Paiva, A. (2012). Long-term in- development and test. Computers & Education, 57(4), 2432e2440.
teractions with empathic Robots: Evaluating perceived support in children. In Urquijo, S. P., Scrivener, S. A. R., & Palme n, H. K. (1993). The use of breakdown
S. Ge, O. Khatib, J.-J. Cabibihan, R. Simmons, & M.-A. Williams (Eds.), Social analysis in synchronous CSCW system design. In G. de Michelis, C. Simone, &
robotics (Vol. 7621, pp. 298e307). Berlin Heidelberg: Springer. K. Schmidt (Eds.), Proceedings of the third European conference on computer-
Leite, I., Martinho, C., Pereira, A., & Paiva, A. (2009, Sept. 27 ). As Time goes by: Long- supported cooperative work 13e17 September 1993, Milan, Italy ECSCW ’93 (pp.
term evaluation of social presence in robotic companions. In Proceedings of the 281e293). Dordrecht: Springer Netherlands.
RO-MAN 2009-the 18th IEEE international symposium on robot and human Vandevelde, C., Wyffels, F., Ciocci, C., Vanderborght, B., & Saldien, J. (2015). Design
interactive communication (pp. 669e674). and evaluation of a DIY construction system for educational robot kits. Inter-
Lemaignan, S., Jacq, A. D., Hood, D., Garcia, F., Paiva, A., & Dillenbourg, P. (2016). national Journal of Technology and Design Education, 1e20.
Learning by teaching a Robot: The case of handwriting. Robotics and automation Vavoula, G., & Sharples, M. (2009). Meeting the challenges in evaluating mobile
magazine. learning: A 3-level evaluation framework. International Journal of Mobile and
Leyzberg, D., Spaulding, S., & Scassellati, B. (2014). Personalizing robot tutors to Blended Learning, 1, 54e75.
individuals' learning differences. In Proceedings of the ACM/IEEE international Vogt, P., de Haas, M., de Jong, C., Baxter, P., & Krahmer, E. (2017). Child-robot in-
conference on human-robot interaction, Bielefeld, Germany (pp. 423e430). teractions for second language tutoring to preschool children. Frontiers in Hu-
Leyzberg, D., Spaulding, S., Toneva, M., & Scassellati, B. (2012). The physical presence man Neuroscience, 11(73).
of a robot tutor increases cognitive learning gains. In Paper presented at the 34th Vygotsky, L. (1930). Mind and society. Harvard University Press.
annual conference of the cognitive science society. Wood, D., & Wood, H. (1996). Vygotsky, tutoring and learning. Oxford Review of
Movellan, J. R., Tanaka, F., Fortenberry, B., & Aisaka, K. (2005). The RUBI/QRIO Education, 22(1), 5e16.
Project: Origins, principles, and first steps. In In proceedings of the 4th inter- Yun, S., Shin, J., Kim, D., Kim, C. G., Kim, M., & Choi, M.-T. (2011). Engkey: Tele-
national conference on development and learning (pp. 80e86). education robot. In B. Mutlu, C. Bartneck, J. Ham, V. Evers, & T. Kanda (Eds.),
Mubin, O., Stevens, C. J., Shahid, S., Mahmud, A. A., & Dong, J.-J. (2013). A review of Social Robotics: Proceedings of the 3rd international conference on social robotics
the applicability of robots in education. Technology for Education and Learning, (ICSR), Amsterdam, The Netherlands (pp. 142e152). Berlin, Heidelberg: Springer
1e7. Berlin Heidelberg.
Nakamura, I. (2008). Understanding how teacher and student talk with each other: Zhao, S. (2006). Humanoid social robots as a medium of communication. New Media
An exploration of how `repair' displays the co-management of talk-in- & Society, 8(3), 401e419.