2017 14th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI)

June 28 - July 1, 2017 at Maison Glad Jeju, Jeju, Korea

Exploring the taxonomic and associative link between emotion and

function for robot sound design
Eunju Jeong1, Gyu Hyun Kwon2 and Junseop So3
1, 3
Department of Arts and Technology, Hanyang University, Seoul, 04763, Korea
(Tel : +82-2-2220-4797; E-mail: [email protected], [email protected])
2
Graduate School of Technology and Innovation Management, Hanyang University, Seoul, 04763, Korea
(Tel : +81-2-2220-2414; E-mail: [email protected])

Abstract – Sound is a medium that conveys functional The use of acoustic features and musical elements in
and emotional information in a form of multilayered robot sound design need to be reflective on the capacity
streams. With the use of such advantage, robot sound of soundscape perception. Our sound perception involves
design can open a way for being more efficient sequential and simultaneous processing to understand
communication in human-robot interaction. As the first complex meaning of auditory scene [7]. It is, thus,
step of research, we examined how individuals perceived important to understand how different messages (i.e.,
the functional and emotional intention of robot sounds emotional, functional) embedded in each sound stream
and whether the perceived information from sound is are perceived as paralleled or integrated single percept.
associated with their previous experience with science Studies on robot sound design, however, mostly
fiction movies. The sound clips were selected based on focuses on messages that aim to deliver information
the context of the movie scene (i.e., Wall-E, R2-D2, BB8, about emotional qualities [5], social intention [8], motion
Transformer) and classified as functional (i.e., platform, trajectory [6, 9-10], functional and situational status
monitoring, alerting, feedback) and emotional (i.e., [10-11]. More importantly, it was not clearly examined
positive, neutral, negative). A total of 12 participants how different types of information interacts and
were asked to identify the perceived properties for each establishes integrated messages in a form of shared sound
of the 30 items. We found that the perceived emotional feature.
and functional messages varied from those originally Our long-term goal is to establish sound taxonomy and
intended and differed by previous experience. design guideline specified to apply HRI. The primary
objective was to examine how individuals perceived the
Keywords – Human-robot Interaction, Sound Design, original intention of robot sounds, which was classified
Nonlinguistic-functional message as functional and emotional in the present study. Since
auditory imagery of robot has been greatly influenced by
1. Introduction previous experience through science fiction movie, we
also tested the perceived messages were influenced by
The urban soundscape consists of diverse types of previous exposure to the movie.
sound and is delivered as multilayered streams. Within
the context of human-robot interaction (HRI), such 3. Methods
soundscape exists in a similar way, including machine
platform, linguistic and nonlinguistic-functional sounds We performed a preliminary literature review to
[1]. Although sound is known to increase the efficiency establish a latent sound taxonomy. Three types of robot
in a wide variety of behaviors, the current state of art in were selected depending on the applied areas and the
robot sound design mostly focuses on socio-emotional levels of autonomy [12] and four types of auditory
aspect of human-robot interaction. As the scope of robot display were selected bases on the functionality [13-15],
application has expanded from industrial automation creating a total of 12 types of robot sound taxonomy
system to personal smart devices, a more comprehensive (Table 1).
taxonomy and guideline for robot sound design is needed.
Table. 1. Taxonomy for Robot Sound Design
2. Theoretical Background
Industrial robot Service robot Robot with
Sound has been gaining attention in the areas of robot special mission
Platform Fixed Mobile/fixed Mobile/fixed
sound design. As in urban soundscape, human-robot
interaction creates multilayered sounds and diverse types Monitoring Basic status Basic status Basic status
(e.g., on/off) (e.g., on/off) (e.g., on/off)
of messages as the whole auditory scene. If we aim to aid
human-robot communication through sounds, it is Alerting Manual Low level High level
autonomic autonomic
important to understand the nature of soundscape and
identify the specificity in the HRI. Previous studies have Feedback Visual display Visual & Visual &
examined the effect of robot motor sound [2], nonverbal auditory display auditory display
signals [3], nonlinguistic expression [4], and musical
utterance [5-6]. As yet, sound taxonomy among different
types of robot sound has not been established. The sound items were selected from three science
fiction movies (i.e., Wall-E, Star Wars, and Transformer).
Based on the context in which robots interact intimately

978-1-5090-3056-9/17/$31.00 ©2017 IEEE 641

with human beings, environments, and other objects, we neutral and negative showed neutral, and neutral and
selected sound items and subcategorized them into six negative, respectively. In terms of functional messages,
groups, including 1) Platform, Monitoring, Alerting, and which can be associated with the emotional sounds,
Feedback (functional), and 2) Positive, Neutral, and positive and neutral sounds were identified as having
Negative (emotional). Details of each of 30 items were monitoring and feedback, while negative sounds were felt
seen in table 2. as platform and monitoring sounds. All the emotional
sounds were associated with monitoring function and
Table 2. Detailed characteristics of sound items none of them was obviously related with alerting
function.
Title Context Time Duration Type
from the (sec)
movie
Table 3. The descriptive results (N =12)
Wall-e
01 Work finished 0:19 0.5 Feedback Perceived message
02 Clean-up finished 0:36 0.5 Platform
Emotional Functional
03 Information delivery 3:04 0.5 Neutral
Transformer Platform Monitoring Alerting Feedback Positive Neutral Negative
01 Transforming sound 2:03 2 Platform Platform 6.56 5.28 3.14 4.47 4.72 6.44 5.17
02 Transforming sound 9:38 1 Platform (3.39) (3.56) (2.58) (3.32) (2.73) (2.82) (3.01)
03 Congratulatory expression 18:44 2 Positive
Monitoring 5.88 7.00 3.27 5.38 5.23 6.73 4.50

Functional
04 Complaining expression 7:33 1 Negative
(3.35) (3.40) (2.75) (3.64) (3.13) (2.76) (3.18)
05 Disappointing expression 7:59 1 Negative
06 Disappointing expression 8:55 1 Negative Alerting 5.65 5.92 4.81 4.63 4.31 5.85 5.63

Intended message
R2D2 (3.43) (3.16) (3.43) (3.07) (2.72) (2.19) (3.32)
01 Feedback to manipulation 0:24 0.5 Feedback
Feedback 5.48 5.90 3.19 5.94 6.38 5.79 3.35
02 Feedback to manipulation 1:16 0.3 Feedback
(2.91) (3.20) (2.30) (3.05) (2.51) (2.87) (2.37)
03 Feedback to manipulation 1:42 1.3 Feedback
04 Input data detection 1:16 0.8 Monitoring Positive 3.96 6.17 4.08 5.13 5.23 5.52 4.50
05 Input data detection 0:08 1 Monitoring (3.01) (3.00) (3.09) (3.33) (3.12) (2.68) (3.12)
Emotional

06 Input data detection 0:03 1.5 Monitoring

Neutral 4.33 5.77 3.33 5.27 4.98 6.45 4.25
07 System error 2:40 1 Alerting (3.39) (3.12) (2.79) (3.30) (2.90) (2.42) (2.70)
08 Ready to work 0:06 1.5 Monitoring
09 System error 0:55 1.6 Alerting Negative 5.37 6.12 3.57 3.50 3.60 6.25 5.93
10 System error 0:56 2 Alerting (3.11) (3.26) (2.80) (2.75) (2.12) (2.58) (2.63)
BB8
01 Situational risk 1:30 1 Alerting
02 Information delivery 3:29 1 Neutral Additional between group comparison was performed
03 Information delivery 3:39 0.5 Neutral (group 1: participants who have seen more than two
04 Information delivery 3:57 1 Neutral
05 Information request 4:47 0.5 Neutral movies versus group 2: those who have experience less
06 Information delivery 4:39 0.5 Positive than one movie). The finding showed significant
07 Welcoming expression 4:40 1 Positive difference in feedback (t(8) = 3.41, p < .01), positive (t(6)
08 Positive expression 4:55 0.5 Positive
09 Positive expression 4:58 1 Positive = 3.45, p < .01), and neutral (t(8) = 4.16, p < .01) sounds,
10 Disappointing expression 1:43 1 Negative indicating that group 1 than group 2 identified more
11 Disappointing expression 4:48 1 Negative
accurately the intended messages used in the movie
contexts (Table 4 and Table 5).
A total of 12 college students (Mean age of 25.42, SD = In terms of functional sounds associated with
1.89) were presented with 30 sound items and asked to emotional qualities (Table 4), two groups perceived
identify the functional and emotional messages by different emotional qualities from the same functional
marking on the electronic answer sheet given through the sounds. For the platform sounds, group 1 felt the platform
google survey. Each item was repeated three times with sounds as positive and neutral (more toward positive),
2-sec time interval through headsets and the order was while group 2 felt them as neutral and negative (more
randomly generated. A total time to completion was toward negative). Monitoring sounds were felt as neutral
approximately 20 minutes. in both group 1 and group 2. Alerting sounds were
similarly identified as neutral and negative in both groups.
4. Interim Results For the feedback sounds, group 1 identified the sounds as
positive and neutral, while 2 showed identified as
Table 3 shows our interim results for whole group. positive only. (Table 4).
Platform and monitoring sounds were identified as
originally intended in the movie scene. They were found Table 4. Between group comparison of emotional
associated with neutral state of emotion. Alerting and qualities delivered by functional sounds
feedback sounds, however, showed a non-clear tendency
in terms of both functional and emotional messages they Group 1 Group 2
deliver. Participants identified alerting sounds as platform Positive Neutral Negative Positive Neutral Negative
or monitoring and feedback sounds as platform, Platform 5.67 6.94 4.33 3.78 5.94 6.00
monitoring, or feedback sounds. Also, the alerting sounds (2.85) (2.78) (2.85) (2.25) (2.78) (2.94)

were felt toward negative and feedback sounds were Monitoring 5.36 7.20 5.20 5.28 5.96 3.76
(3.13) (2.49) (3.17) (3.11) (2.93) (3.08)
toward positive.
Alerting 4.92 6.00 5.96 3.71 5.71 5.29
Sounds that were originally designed to deliver (2.84) (1.73) (3.32) (2.44) (2.56) (3.28)
emotional messages showed a relatively less obvious Feedback 6.46 6.67 3.63 6.29 4.92 3.08
tendency. That is, positive sound items were spread out (2.12) (2.56) (2.27) (2.84) (2.90) (2.43)
across all emotion types. The sound items categorized as

642
 of the 8th ACM/IEEE international conference on
Table 5 shows subtle, but not clear difference in terms of Human-robot interaction, pp. 209-210, 2013.
functional messages delivered by emotional sounds. [5] E. S. Jee, Y. J. Jeong, C. H. Kim, and H. Kobayahi,
Group 2 identified each of the three types of emotional “Sound design for emotion and intention expression
sounds as being associated with monitoring function. in socially interactive robots,” Intelligent Service
Group 1, however, felt relatively more various functional Robotics, Vol. 3, pp. 199-206, 2010.
messages from the emotional sounds. For example, they [6] G. Johannsen, Auditory displays in
perceived feedback function from the positive and neutral human–machine interfaces. Proceedings of the
emotional sounds. Negative emotional sounds also IEEE, Vol. 92, pp. 742–758, 2004.
showed relatively high for platform and monitoring [7] A. S. Bregman, “Auditory scene analysis: The
function. Though it was unclear between groups and perceptual organization of sound,” Cambridge, Mass.:
needed to further examination with a larger sample, the Bradford Books, MIT Press, 1994.
current findings showed a tendency between functional [8] R. Read, “Situational context directs how people
and emotional sounds depending on previous experience. affectively interpret robotic non-linguistic
utterances,” Proceedings of the ACM/IEEE
Table 5. Between group comparison of functional international conference on Human-robot interaction,
qualities delivered by emotional sounds pp. 41-48, 2014.
[9] G. Johannsen, “Auditory displays in
Group 1 Group 2
human–machine interfaces of mobile robots for
Platform Monitoring Alerting Feedback Platform Monitoring Alerting Feedback
non-linguistic speech communication with
Positive 4.33 5.67 4.58 5.38 3.58 6.67 3.58 4.88 humans,” Journal of Intelligent and Robotic
(3.21) (2.76) (3.04) (3.28) (2.75) (3.13) (3.07) (3.36)
Systems, Vol. 32, pp. 161–169, 2001.
Neutral 4.07 5.90 3.00 5.90 4.60 5.63 3.67 5.10
(3.29) (3.16) (2.42) (3.09) (3.47) (3.08) (3.08) (3.48) [10] G. Johannsen, “Auditory display of directions and
Negative 5.77 6.30 3.77 4.00 4.97 5.93 3.37 3.00 states for mobile systems,” Proceedings of the
(2.82) (3.15) (2.88) (3.15) (3.32) (3.35) (2.69) (2.52) international conference on auditory display, Vol.
98–103, 2002.
5. Future work [11] A. Ayesh, “Emotionally expressive music based
interaction language for social robots,” ICGST
It was a preliminary study for our long-term goal, International Journal on Automation, Robotics and
which establish sound taxonomy and design guideline for Autonomous Systems, Vol. 9, pp. 1–10, 2009.
HRI. The present findings indicated that sounds delivered [12] Z. Z. Bien and H. E. Lee, “Effective learning system
more than one specific message and were identified techniques for human–robot interaction in service
differently depending on the participants’ previous environment,” Knowledge-Based Systems, Vol. 20,
experience. It suggested that robot sound design needs to pp. 439-456, 2007.
embrace such characteristics to accommodate efficient [13] T. Hermann, A. Hunt, and J. G. Neuhoff, “The
communication in HRI. Based on the insight through the sonification handbook,” Logos Verlag, Berlin, 2011.
present study, we will further analyze the acoustic [14] T. Siu-Lan, A. J. Cohen, S. D. Lipscomb and R. A.
features of sounds associated with the functional and Kendall, The psychology of music in multimedia.
emotional messages and design non-linguistic functional Oxford University Press, U. K., 2013.
sound patterns that efficiently delivers intended message. [15] J. Salamon, C. Jacoby and J. P. Bello, “A Dataset
and Taxonomy for Urban Sound Research,”
6. References Proceedings of the 22nd ACM international
conference on Multimedia, pp. 1041-1044, 2014.
[1] S. Yilmazyildiz, R. Read, T. Belpeame, & W.
Verhelst, “Review of sematic-free utterances in social
human-robot interaction,” International Journal of
Human-Computer Interaction, Vol. 32, pp. 63-85,
2015.
[2] M. Joosse, M. Lohse, and V. Evers, “Sound over
Matter: The Effects of Functional Noise, Robot Size
and Approach Velocity in Human-Robot Encounters,”
Proceedings of ACM/IEEE international conference
on Human-robot interaction, pp. 184-185, 2014
[3] E. Cha, M. Mataric, ғ and T. Fong, “Nonverbal
signaling for non-humanoid robots during
human-robot collaboration”, Proceedings of the 11th
ACM/IEEE International Conference on Human
Robot Interaction, pp. 601-602, 2016.
[4] R. Read and T. Belpaeme, “People interpret robotic
non-linguistic utterances categorically,” Proceedings

643