Social Robots and Teaching Music to Autistic

Children: Myth or Reality?

Alireza Taheri1,4, Ali Meghdari1 ✉ , Minoo Alemi1,2, Hamidreza Pouretemad3,4,

( )

Pegah Poorgoldooz4, and Maryam Roohbakhsh4

1
Social & Cognitive Robotics Laboratory, Center of Excellence in Design, Robotics,
and Automation (CEDRA), Sharif University of Technology, Tehran, Iran
[email protected]
2
Islamic Azad University, Tehran-west Branch, Tehran, Iran
3
Institute for Cognitive and Brain Sciences (ICBS), Shahid Beheshti University, Tehran, Iran
4
Center for Treatment of Autistic Disorders, Tehran, Iran

Abstract. Music-based therapy is an appropriate tool to facilitate multisystem

development in children with autism. The focus of this study is to implement a
systematic and hierarchical music-based scenario in order to teach the funda‐
mentals of music to children with autism through a social robot. To this end, we
have programmed a NAO robot to play the xylophone and the drum. After running
our designed robot-assisted clinical interventions on three high-functioning and
one low functioning autistic children, fairly promising results have been observed.
We indicated that the high-functioning participants have learned how to play the
musical notes, short sentences, and simple rhythms. Moreover, the program
affected positively on autism severity, fine movement and communication skills
of the autistic subjects. The initial results observed indicate promising potentials
for involving social robots in music-based autism therapy.

Keywords: Music-based therapy · Xylophone · Autism spectrum disorders

(ASD) · Humanoid social robot · Social and cognitive skills · Imitation

1 Introduction

Music greatly influences humans and in particularly children’s emotions, moods, and
feelings. Teaching music can help develop new or improve existing social, verbal and
non-verbal communication skills in children [1, 2].
Children with autism have stereotyped behaviors and limited verbal communication
skills [3]. Music and rhythms are effective methods to involve them in rhythmic and
non-verbal communication. Nowadays, at least 12 % of all treatment of individuals with
autism consist of music-based therapies [4].
Music has often been used in therapeutic sessions with children with mental and
behavioral disabilities [5, 6]. In particular, there is ample evidence that shows either
playing music during therapy sessions or teaching music to children with autism spec‐
trum disorders (ASD) can significantly increase the impact of therapy sessions [7]. In
such studies or therapy sessions an instrument is either played by a human or recorded

© Springer International Publishing AG 2016

A. Agah et al. (Eds.): ICSR 2016, LNAI 9979, pp. 541–550, 2016.
DOI: 10.1007/978-3-319-47437-3_53
542 A. Taheri et al.

music is played back in individual and group intervention sessions [4, 8]. The effects of
music-based therapy in improving social skills of children with autism (i.e. eye contact
and initiating social behaviors) have been reported in [9]. Kim et al. [2] showed
improvement in joint attention, turn taking and eye contact of children with autism in
active music-making interventions. In [10, 11], the studies showed a decrease in ster‐
eotyped behaviors and self-injuries in children with autism after running music-based
interventions. Music therapy interventions have been used to increase social [12] and
emotional [13] skills, verbal and gestural communication [14], and behaviors [15] of
individuals with autism in individual and group modes. It should be noted that the lack
of studies on improving gross and fine motor skills of autistic children through music-
based interventions is still a gap in this area [4].
Recently, we have designed a comprehensive robot-assisted music-based interven‐
tion scenario to improve perceptuo-motor, social, and cognitive skills of ASDs and
conducted it in a single subject design study. The purpose of this educational-therapy
program is to teach the fundamental concepts of how to play drum and xylophone using
a NAO humanoid robot as a teacher’s assistant to children with autism. Our goal is to
find scientific answers for the following research questions: (1) Does a humanoid social
robot have the ability to teach music (i.e. notes and rhythms) to children with autism?
(b) Can a humanoid robot improve social and cognitive skills in children with autism
through music education?
To this end, a drum/xylophone playing humanoid robot in addition to other musical
instruments, both of which are loved by children, were used. Although the use of robotics
technology in different aspects of education and treatment is increasing [16–25], to the
best of our knowledge utilizing a humanoid robot to systematically teaching music to
children with autism is still an interesting topic. Tapus [22] has used a social robot in a
music-therapy program on individuals with cognitive impairments; however the robot
did not play any musical instruments in her study. It should be noted that some
researchers have also used music instruments like a drum played by a robot [23] as
reinforcement tools (and not necessarily as an education tool) in autism treatment.
This paper presents the results and observations of a robot-assisted therapy in a single
subject design study on three high-functioning and one low-functioning children with
autism. The study was conducted in eleven music-based intervention sessions in Iran in
order to explore the potentials of music-based games on ASDs.

2 Research Methodology

2.1 Participants

Three children with high-functioning and one child with low-functioning autism
enrolled in this robot-assisted research study. All of the participants were 6 years old
males without any previous music background. The children’s details are describe in
Table 1.
 Social Robots and Teaching Music to Autistic Children 543

Table 1. Our participants’ details

# Abbreviation Autism severity
1 P1 High-Functioning autism, with hyperactivity
2 P2 High-Functioning autism
3 P3 High-Functioning autism, with verbal deficits
4 P4 Low-Functioning autism with poor verbal skills

2.2 Humanoid Robot

The humanoid robot used in this research is the NAO H-21 robot made by Aldebaran
Company [26]. The capabilities of NAO as well as the suitable programming interface
of this robot make it a commonly used commercial robot for autism research [27–29].
We have renamed the robot to the Iranian boy’s name, “Nima”, during our studies.

2.3 Musical Instruments

Having noticed that when the instrument sound is simpler and more pleasant the patient
will be deeper involved and the interventions will be more effective, we have selected
a drum and a xylophone for the robot to play in our intervention sessions.

2.4 Technical Design of the Games

We have designed two general music games to involve children in interventions: (a)
playing a real drum/xylophone in Robot-Child or Robot-Child-Therapist/Parent imita‐
tion turn taking games and (b) playing a Kinect based virtual xylophone on the screen.

Play the Drum/Xylophone: Our robot has been programmed to be able to play the
drum/xylophone. A configurable user friendly GUI1 as well as some rhythm patterns by
Choregraphe [26] software have been developed in order to enable the robot playing the
instruments either manually by operators/psychologists or automatically in a real-time
situation. The robot is able to play different rhythms and notes with its right/left arm.

Virtual Xylophone: We have developed a Kinect based virtual xylophone containing

8 colored bars programmed by C# WPF. In this game, the player can see the music bars
on the screen. The participant can hit the bars using colored mallets or the palm of their
hands. The sounds of one octave from C4 (261.6 Hz) to C5 (523.2 Hz) are heard when
hitting these bars. We have used the Toub.Sound.Midi Library in order to generate music
notes with the computer.

1
Graphical User Interface .
544 A. Taheri et al.

2.5 Experimental Setup

Our study was conducted in the Social & Cognitive Robotics Laboratory at Sharif
University of Technology with four autistic children during eleven sessions in the pres‐
ence of a human therapist (and sometimes their parents), a humanoid robot, and a robot
operator. Time duration of each session was 20–30 min.
The games’ instructions were described by the robot and/or the human therapist
before each game. The Wizard of OZ (WOZ) style robot control has been used in this
study, and most of the time the robot operator sent appropriate real-time voice/motor
commands to the robot after seeing the child’s performance. Our single subject design
study contains a Baseline, Pre-Test, Post-Test, and Follow-up Test (four weeks after the
last session) all in the absence of the Nima robot. Because no control group was utilized
in this paper, our focus was on comparing each participant’s skills/behaviors with his
previous behaviors based on the assessment tools (which will be introduced in
Subsect. 2.7).

2.6 Interventions Protocol

The purpose of designing a comprehensive music-based robot-assisted intervention

scenario is teaching the fundamentals of music, decreasing impairments and improving/
generalizing social and cognitive skills as well as fine/gross movement skills of children
with autism spectrum disorders. Our music games are based on active music-based
therapies which included imitating the robot in playing the drum/xylophone, rhythm
perception, working memory games, teaching the notes, involving in turn-taking group
games, reading/playing music sentences, and finally generalizing the learned knowledge
to another instrument (i.e. the virtual xylophone). The designed therapeutic games have
the potentials to improve auditory perception, perceptuo-motor activities, vision skills,
mental development, and social and communication skills.

2.7 Assessment Tools

To answer the research questions of this study, two measuring instruments have been
used each for four times (on Baseline, Pre-, Post-, and Follow up tests).
(1) Stambak’s Rhythmic Structures Reproduction test [30] which is a test containing
21 (easy to hard level) rhythmic tasks that the participant should reproduce the
patterns through a drum after hearing (and not seeing) them performed by the
therapist.
(2) Gilliam Autism Rating Scale (GARS) [31], a questionnaire for estimating autism
severity, with 56 questions which covers four subscales: Stereotyped Behaviors,
Communication, Social Interactions, and Developmental Disturbances.
 Social Robots and Teaching Music to Autistic Children 545

3 Results and Discussion

At the first session, the Nima robot was introduced to the participants. The main purpose
of this session was to familiarization/desensitization the participants to the class envi‐
ronment as well as observe the child’s tendency to start/keep communication with
Nima. It should be noted that music was not taught during the first session.
After that, following the designed educational protocol, music was taught step-by-
step by the robot during the rest of the sessions. The selected snapshot of the intervention
sessions is presented in Fig. 1.

Fig. 1. Snapshot of the robot-assisted music-based intervention sessions

Music was a happy and enjoyable activity for all of the participants. The existence
of the robot itself considerably increased the motivation of the children to use their
capabilities to involve their sensorimotor mechanisms. A short description of observa‐
tions for the participants are presented in the following.

P1: The psychologists reported noteworthy improvement in his social skills, attention,
and the ability to learn. The music learning process occurred for Benyamin and
the presence of the robot was the reason for this observation
P2: P2’s performance was very similar to typically developing children. Although P2
is very resistance to education classes in his real life, he did not show any malad‐
aptive behaviors in our course. It seemed the child felt secure and relaxed during
the sessions
P3: Improvement in Radvin’s verbal skills was reported by his mother. In comparison
to the other two high-functioning children, Radvin’s weakness at the first sessions
546 A. Taheri et al.

was his lack of expressing/identifying some colors; fortunately, this deficit was
resolved by the last sessions
P4: He understood none of the instructions at the beginning of the program. Improve‐
ment in his instruction perception, attention, and understanding what happened in
the class was the positive note occurred for P4 over time. He usually played
randomly on the real xylophone bars instead of correctly imitating the robot/parent
during the turn-taking games; however he was able to acceptably imitate the robot
switching hands in the last four sessions. We observed that P4’s stereotyped
behaviors (especially his fluttering fingers) decreased and his verbal skills
increased. Music is very effective in decreasing the stereotyped behaviors of chil‐
dren with autism [10, 11].

3.1 Stambak Rhythmic Structures Reproduction Test

To investigate the participants’ improvement in rhythm perception during this time, the
Stambak’s Rhythm Reproduction Test has been run on the autistic subjects. The results
of this test in the Pre- and Post-Test are presented in Table 2. We indicated that all of
the three high-functioning participants show improvement in playing rhythm which
means music learning has occurred for these subjects. In [32], the Stambak rhythm test
has been used to show the music improvement of typically developing subjects after
running a human-based music-therapy program.

Table 2. Stambak’s Rhythm Test results in Pre-Test and Post-Test

# Abbreviation Scores of the Stambak’s Rhythm Test
Pre-Test Post Test
1 P1 18 19
2 P2 6 13
3 P3 11 14
4 P4 0 4

Meanwhile, P4 was unable to do any of the rhythm tasks in Pre-Test; however, in

the Post-Test he was able to play tasks #1 to #4, each in his second try with the help of
counting (the number of played hits) and saying “Baam-Baam”.
For the high-functioning autistic children, our main goal was observing the trend of
their music learning pace. Rhythm perception for the three high functioning subject have
been improved. From our observations, we hypothesized that there is no obvious differ‐
ence between the music learning process/progress rate of high-functioning ASDs and
TDs2; they can read/play the notes and simple musical sentences and the progress rate
for the spent three months was really promising. Moreover, we observed noticeable
progress in all four children’s fine hands imitation as well as using two hands consecu‐
tively, which they all had problems with in the first sessions.

2
Typically Developing.
 Social Robots and Teaching Music to Autistic Children 547

One of the promising observations from the high-functioning subjects’ performance

in this study was their following, curiosity, and questioning increased in the presence
of the robot. The children used as many opportunities as they could to push Nima to
speak and ask him questions.

3.2 GARS Questionnaire

The results of GARS are presented in Fig. 2. We observed a decrease in the autism
severity of all four participants from Pre-Test to Post-Test. A detailed assessment of the
GARS questionnaire showed us improvement occurred in 3 subscales: stereotyped
behaviors, communication, and social interaction for two of the participants, P1 and P4
from Pre-Test to Post-Test. P3 progressed in stereotyped behaviors and communication,
while P2 showed improvement in communication and social interaction. The interesting
point of the GARS results is the progress of all of the subjects in subscale “communi‐
cation”. This finding is in line with our therapists’ reports during the intervention
sessions. The existence of Nima, as an attractive communication tool, positively affected
the communication skills of the children. Comparison of GARS overall scores between
the Post-Test and Follow up-Test shows the retention as well as the stability of the
sessions’ impact on autistic children. Similarly, improvement of autistic children in
different social and communication skills after music-based therapies or robot-assisted
clinical interventions have also been confirmed in [12–15, 20, 21, 29].

Fig. 2. GARS overall scores (autism severity) of the participants in Baseline, Pre-Test, Post-Test,
and Follow up Tests.
548 A. Taheri et al.

3.3 Limitations and Future Works

In order to check for the effectiveness of the treatment, a larger sample size is needed,
ideally with male and female children and a control group. Moreover, one of the greatest
limitations in these kinds of studies is accessing valid tools which have the potential to
accurately measure/assess children’s behavior. Unfortunately, because of small number
of the autistic participants in this study, no scientific statistical analysis could be applied
on the data. To generalize the mentioned findings in robot-assisted music-based therapy,
further research with more autistic subjects is needed.

4 Conclusion

Through the designed music-based scenario package, we wanted to explore the potential
of music-based intervention on ASD’s improvement in motor, communication skills as
well as learning music. The robot does have the ability to teach the fundamentals of
music to children with autism. We also saw improvement in fine hands imitation, using
both hands in order, and rhythm identification for all of the participants. The high-func‐
tioning subjects can now read/play the notes and simple musical sentences and their
progress was quite acceptable during the three months interventions. In the case of the
low-functioning subject, improvement in verbal skills and a decrease in stereotyped
behaviors have been indicated. Additionally, the GARS showed that the autism severity
of all of the participants were reduced after the robot-assisted intervention sessions.

Acknowledgement. Our sincere appreciation is extended to the “Cognitive Sciences and

Technologies Council of Iran” for their financial support through research grant # 103. We also
acknowledge the “Center for the Treatment of Autistic Disorders (CTAD)” and its psychologists
for their technical support and cooperation in participating in the clinical interventions with the
children with autism spectrum disorders.

References

1. Wan, C.Y., Bazen, L., Baars, R., Libenson, A., Zipse, L., Zuk, J., Schlaug, G.: Auditory-
motor mapping training as an intervention to facilitate speech output in non-verbal children
with autism: a proof of concept study. PLoS ONE 6(9), e25505 (2011)
2. Kim, J., Wigram, T., Gold, C.: The effects of improvisational music therapy on joint attention
behaviors in autistic children: a randomized controlled study. J. Autism Dev. Disord. 38(9),
1758–1766 (2008)
3. Pouretemad, H.: “Diagnosis and treatment of joint attention in autistic children”, (in Persian).
Arjmand Book, Tehran (2011)
4. Srinivasan, S.M., Bhat, A.N.: A review of “music and movement” therapies for children with
autism: embodied interventions for multisystem development. Frontiers Integr. Neurosci. 7,
22 (2013)
5. Roper, N.: Melodic intonation therapy with young children with apraxia. Bridges 1(8), 1–7
(2003)
 Social Robots and Teaching Music to Autistic Children 549

6. Boso, M., Emanuele, E., Minazzi, V., Abbamonte, M., Politi, P.: Effect of long-term
interactive music therapy on behavior profile and musical skills in young adults with severe
autism. J. Altern. Complement. Med. 13(7), 709–712 (2007)
7. Lim, H.A., Draper, E.: The effects of music therapy incorporated with applied behavior
analysis verbal behavior approach for children with autism spectrum disorders. J. Music Ther.
48(4), 532 (2011)
8. Corbett, B.A., Shickman, K., Ferrer, E.: Brief report: the effects of Tomatis sound therapy on
language in children with autism. J. Autism Dev. Disord. 38(3), 562–566 (2008)
9. Stephens, C.E.: Spontaneous imitation by children with autism during a repetitive musical
play routine. Autism 12(6), 645–671 (2008)
10. Lanovaz, M.J., Fletcher, S.E., Rapp, J.T.: Identifying stimuli that alter immediate and
subsequent levels of vocal stereotypy a further analysis of functionally matched stimulation.
Behav. Modif. 33(5), 682–704 (2009)
11. Wood, S.: A study of the effects of music on attending behavior of children with autistic-like
syndrome (1991)
12. Pasiali, V.: The use of prescriptive therapeutic songs in a home-based environment to promote
social skills acquisition by children with autism: three case studies. Music Ther. Perspect.
22(1), 11–20 (2004)
13. Katagiri, J.: The effect of background music and song texts on the emotional understanding
of children with autism. J. Music Ther. 46(1), 15–31 (2009)
14. Simpson, K., Keen, D.: Music interventions for children with autism: narrative review of the
literature. J. Autism Dev. Disord. 41(11), 1507–1514 (2011)
15. Carnahan, C., Basham, J., Musti-Rao, S.: A low-technology strategy for increasing
engagement of students with autism and significant learning needs. Exceptionality 17(2), 76–
87 (2009)
16. Taheri, A.R., Alemi, M., Meghdari, A., PourEtemad, H.R., Basiri, N.M.: Social robots as
assistants for autism therapy in Iran: Research in progress. In: 2014 Second RSI/ISM
International Conference on Robotics and Mechatronics (ICRoM), pp. 760-766. IEEE,
October 2014
17. Boccanfuso, L., O’Kane, J.M.: CHARLIE: an adaptive robot design with hand and face
tracking for use in autism therapy. Int. J. Soc. Robot. 3(4), 337–347 (2011)
18. Meghdari, A., Alemi, M., Ghazisaedy, M., Taheri, A. R., Karimian, A., Zandvakili, M.:
Applying robots as teaching assistant in EFL classes at Iranian middle-schools. In: The 2013
International Conference on Education and Modern Educational Technologies
(EMET-2013), Venice, Italy, September 2013. http://www.europement.com. Accessed
19. Alemi, M., Meghdari, A., Ghazisaedy, M.: The impact of social robotics on L2 learners’
anxiety and attitude in English vocabulary acquisition. Int. J. Soc. Robot. 7(4), 523–535
(2015)
20. Scassellati, B., Admoni, H., Mataric, M.: Robots for use in autism research. Annu. Rev.
Biomed. Eng. 14, 275–294 (2012)
21. Kajopoulos, J., Wong, A.H.Y., Yuen, A.W.C., Dung, T.A., Kee, T.Y., Wykowska, A.: Robot-
assisted training of joint attention skills in children diagnosed with autism. In: Tapus, A.,
André, E., Martin, J-.C., Ferland, F., Ammi, M.(eds.) ICSR 2015. LNCS, vol. 9388, pp. 296–
305. Springer, Heidelberg (2015)
22. Tapus, A.: The role of the physical embodiment of a music therapist robot for individuals
with cognitive impairments: longitudinal study. In: Virtual Rehabilitation International
Conference, 2009, pp. 203–203. IEEE, June 2009
550 A. Taheri et al.

23. Robins, B., Dautenhahn, K., Dickerson, P.: From isolation to communication: a case study
evaluation of robot assisted play for children with autism with a minimally expressive
humanoid robot. In: Second International Conferences on Advances in Computer-Human
Interactions, 2009, ACHI 2009, pp. 205–211. IEEE, February 2009
24. Boccanfuso, L., Barney, E., Foster, C., Ahn, Y.A., Chawarska, K., Scassellati, B., Shic, F.:
Emotional robot to examine different play patterns and affective responses of children with
and without ASD. In: 2016 11th ACM/IEEE International Conference on Human-Robot
Interaction (HRI), pp. 19–26. IEEE, March 2016
25. Alemi, M., Meghdari, A., Mahboub Basiri, N., Taheri, A.: The Effect of Applying Humanoid
Robots as Teacher Assistants to Help Iranian Autistic Pupils Learn English as a Foreign
Language (2015)
26. Robotics, A.: The NAO robot (2013). Available at: bttp:/Ivnvw. aldebaran-roboticscomteni.
Accessed 4 Sep 2013
27. Mavadati, S.M., Feng, H., Gutierrez, A., Mahoor, M.H.: Comparing the gaze responses of
children with autism and typically developed individuals in human-robot interaction. In: 2014
14th IEEE-RAS International Conference on Humanoid Robots (Humanoids), pp. 1128–
1133. IEEE, November 2014
28. Shamsuddin, S., Yussof, H., Ismail, L.I., Mohamed, S., Hanapiah, F.A., Zahari, N.I.: Initial
response in HRI-a case study on evaluation of child with autism spectrum disorders interacting
with a humanoid robot Nao. Procedia Eng. 41, 1448–1455 (2012)
29. Taheri, A., Alemi, M., Meghdari, A., Pouretemad, H., Basiri, N.M., Poorgoldooz, P.: Impact
of humanoid social robots on treatment of a pair of Iranian autistic twins. In: Tapus, A., André,
E., Martin, J.-C., Ferland, F., Ammi, M. (eds.) ICSR 2015. LNCS (LNAI), vol. 9388, pp.
623–632. Springer, Heidelberg (2015). doi:10.1007/978-3-319-25554-5_62
30. Gardner, H.: Children’s duplication of rhythmic patterns. J. Res. Music Educ. 19(3), 355–
360 (1971)
31. Gilliam, J. E.: Gilliam autism rating scale: examiner’s manual. Pro-ed (1995)
32. Noor Mohammadi, F.: The effect of the Orff music on improving rhythm structuring. (in
Persian), Developmental Psychology Quarterly, Tehran (2003)