IOP Conference Series: Materials Science and Engineering

PAPER • OPEN ACCESS

User-centered design for Human-Robot Collaboration systems

To cite this article: S L Pizzagalli et al 2021 IOP Conf. Ser.: Mater. Sci. Eng. 1140 012011

View the article online for updates and enhancements.

This content was downloaded from IP address 219.92.210.36 on 25/06/2021 at 15:56

Modern Materials and Manufacturing (MMM 2021) IOP Publishing
IOP Conf. Series: Materials Science and Engineering 1140 (2021) 012011 doi:10.1088/1757-899X/1140/1/012011

User-centered design for Human-Robot Collaboration systems

S L Pizzagalli, V Kuts and T Otto

Department of Mechanical Engineering, Tallinn University of Technology, Ehitajate tee
5, 19086, Tallinn, Estonia

[email protected]

Abstract. Autonomous robotic systems are one of the pillars of Industry 4.0, together with Digital
Twin (DT) simulations and advanced Human Machine Interfaces (HMI). Customization
requirements in modern manufacturing demand a closer collaboration between operators and
automated technologies, leading to a novel Human-Robot Collaboration (HRC) and interaction
paradigm aimed at the augmentation of human capabilities in the workplace. This scenario calls for
a new definition of HRC standards and design for safety, programming, and an overall assessment
of modern cyber-physical systems. A comprehensive design process aiming to formulate a common
framework of requirements based on human, organizational and production needs is missing. A user-
centered design approach described may be a solution to address the open challenges of future HRC.
DT and Augmented and Virtual reality technologies (AR/VR) are implemented in elaborated model
as the necessary tools to assign the operator a central position in the design, control, and assessment
of modern industrial collaborative scenarios.

1. Introduction
The advancements in industrial automation and the need for flexible, adaptable production systems aimed
at mass customization lead to closer collaboration between humans and robots in several industrial fields.
Big data and analytics, autonomous robots, simulation, system integration, industrial internet of things,
cyber physical systems, the cloud, additive manufacturing, Augmented and Virtual reality (AR/VR) are
among the leading technology drivers for the industry 4.0 (I4.0) revolution [1]. Human-Robot Collaboration
(HRC), in particular, is a crucial aspect in the evolution of modern industry into a scenario that will see
human workers being empowered by sensing and perceptive technologies supporting and perfecting human
skills throughout the production process [2]. Natural, adaptive, and intuitive Human Machine Interfaces
(HMI) are necessary to enhance safe and efficient teaming processes [3], favoring well-being, and
technology acceptance in the upcoming industrial scenario. Recent studies show an increased interest in
Digital twins (DT) AR/VR interfaces with examples of assessment and evaluation of safety [4]
programming methods [5] and commissioning of HRC systems. A few studies attempt the combination of
DT AR/VR technologies in a User Centered Design (UCD) approach focusing on specific use cases.
Businesses adopting I4.0 driving technologies are aware of the risks and impacts these have on the operators
[6]. No clear design and assessment methods are yet available or enough to provide a robust framework
that would include human factors in design and decision-making strategies related to HRC. This work aims
at addressing the relationships and synergies occurring between the HRC process and a UCD approach for
collaborative robotic scenarios. The main research question is whether DT AR/VR interfaces are valuable
and reliable tools for the development of modern user-centered robot collaboration systems improving the
efficiency and safety of interaction between operators and machines in the manufacturing field.

Content from this work may be used under the terms of the Creative Commons Attribution 3.0 licence. Any further distribution
of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI.
Published under licence by IOP Publishing Ltd 1
Modern Materials and Manufacturing (MMM 2021) IOP Publishing
IOP Conf. Series: Materials Science and Engineering 1140 (2021) 012011 doi:10.1088/1757-899X/1140/1/012011

2. Evolution of HRC in I4.0

Teaming skilled workers and robots in complex tasks' execution combines the advantages of automation
with soft human skills and decision-making [7]. Safety, intuitive interfaces, including input modes, outputs,
feedback, and programming approaches, together with innovative design and control methods, are the main
ingredients and challenges for the upcoming development and implementation of collaborative production
technologies [8]. The forthcoming massive exploitation of collaborative robots will lead to a series of
organizational, human-centered issues, and production process adaptations [9]. Close collaboration
between humans and robots should be addressed by evaluating the impact these technologies have on the
industrial environment and the operators. Individual, organizational and robotic agent factors influence the
design and organization of the future workspace with open challenges such as operator monitoring, stress
level, workload assessment, robotic system acceptance, efficient reprogramming, data management, ethics,
and privacy related to the collected data [10]. The change in production paradigms will transform the role
and type of relationship and interaction that human workers have with automated technology from mere
cooperation to an augmented and symbiotic paradigm. In their study, Romero et al. [11] describe how the
coming generation of industrial Operators 4.0 (O4.0) will be enhanced in their physical, sensorial, and
cognitive capabilities, being at the center of a symbiotic cyber-physical system. The agents involved in this
new production paradigm will need to be intelligent; therefore, purposeful, perceptive, aware, autonomous,
able to act, reflective, adaptable, learning, and conversational. While many of these qualities are part of
human skills, industrial robots seem to be far from achieving these characteristics.
The popularity of DT solutions and research applications is constantly increasing [12]. The plasticity of
this technology is demonstrated by examples spanning from manufacturing applications, electric engine
optimization [13] to building construction [14], aviation, and healthcare [15]. The scalability of simulation
driven solutions allow for applications supporting maintenance and production planning [16] to specific
realtimed data driven model for product monitoring [17]. AR/VR HMI for HRC offer the advantages of
including the user as an active agent in a robotic cell's control interface, which is itself a DT of the controlled
system and environment. Many recent studies focus on experimental approaches to DT AR/VR interfaces
with applications aimed at programming [18], control [19], design of collaborative industrial cells [20], and
HRC safety assessment [21]. Other examples address the potential of these technologies in becoming UCD
design and evaluation tools for advanced industrial systems and workstations [6] [22]. The rising HRC
industrial paradigms need to be tested and validated both in terms of HRI, interface efficiency for robot
control and system impact on the user. What is missing is a systematic approach that would allow the
evaluation of all aspects involved in the HRC process and make DT AR/VR interfaces the main tool
bridging human and robots in a UCD approach.

3. Collaboration levels, safety and programming in HRC

Human-robot levels of collaboration can be defined in different ways. Helms et al. [23] describe four types
of human-robot operations: independent or parallel work, synchronized work simultaneous work and
assisted or collaborative scenarios. The last scenario involves humans and robots having a common task,
sharing workpieces, tools and workspace while performing the required task at the same time. De Luca et
al. [24] further specify the relation between safety, coexistence and collaboration defining them as nested
levels of human-robot interaction (HRI). Safety is defined as the core feature of robots involved in
collaborative working scenarios. Collaborative industrial robots' requirements are described in [25] and
include four main safety modes: Safety-rated Monitored Stop, Hand Guiding, Speed and Separation
Monitoring, and Power and Force Limiting. Regardless of the characteristics of each specific method it is
crucial to keep in mind the type of involvement and level of physical contact that the operator has in each
mode and the implications for both collaborative task performance and robot programming operations.
Different interaction modalities need a clear definition of robot context-awareness by sensors and artificial
intelligence and have a different impact on the operator. Programming is another central aspect of modern
collaborative robots as it involves a variety of dedicated interfaces or direct manipulation methods such as
in walk-through or programming by demonstration. Virtual reality (VR) might offer a possible solution for

2
Modern Materials and Manufacturing (MMM 2021) IOP Publishing
IOP Conf. Series: Materials Science and Engineering 1140 (2021) 012011 doi:10.1088/1757-899X/1140/1/012011

some of the issues related to programming procedures by providing a repeatable controlled and safe
environment for kinesthetic teaching procedures [26] also on the old generation of robotics systems.

4. UCD for HRC

The contemporary industrial production and O4.0 requirements in HRC need to be allocated in a design
model able to organize collaboration, therefore task allocation and organization, safety and programming
on the same design scope. A user centered design approach seems to be a valuable solution for this
challenge. End-users within a UCD [27] must be active in an iterative design process involving planning,
testing, commissioning, and assessment of the system. The design process should clearly allocate and define
tasks between user and system and include multidisciplinary resources and skills both in defining the
requirements and in the assessment phase. Examples of open and multidisciplinary approaches for the
design of complex, intelligent systems, for example, can be found in autonomous driving vehicle design
[28]. Moreover, requirements should be drawn by analyzing all involved stakeholders the context of use,
including personal and organizational needs and objectives. This can be done by means of qualitative based
descriptions and user models ethnographic research methods or quantitative data collection and analysis.
The new approach shown in Figure 1 aims at a comprehensive design solution for HRC by efficiently
allocating safety, collaboration modes, interfaces, managerial and task-oriented issues and requirements in
an iterative design and evaluation process revolving around the user and its context. The proposed method
matches requirements and technologies typical of HRC systems with the UCD design cycle and
specifications.

Figure 1. A conceptual view to synergies between modern HRC systems and User Centered Design

We propose that DT AR/VR interfaces will be the main technology driver for this new design method,
the tool empowering the O4.0 paradigm and mediating human and robotic systems interaction. Real-time
simulations and interaction technologies can assist in evaluating users, environmental, and organizational
aspects by providing insights into the state of the system in all its aspects. DT AR/VR interfaces are
grounding on IoT, advanced sensors, artificial intelligence, and data analysis technologies which are

3
Modern Materials and Manufacturing (MMM 2021) IOP Publishing
IOP Conf. Series: Materials Science and Engineering 1140 (2021) 012011 doi:10.1088/1757-899X/1140/1/012011

promoting smarter safety systems and learning approaches in robotic collaboration and programming by
grounding their advanced capabilities precisely on human and environment-based data and behavior
analysis. Figure 2 shows the state of realization of the presented approach in the study case discussed in
[29] demonstrating the centrality of DT AR/VR interfaces in assessing an efficient HRC system along a
UCD workflow. In particular the prototype and assessment phases show the O4.0 paradigm of augmentation
of control capabilities in the virtual simulation.

Figure 2. Workflow for human robot collaboration

5. User-centered DT AR/VR architecture for HRC

Considering the variability of collaborative design tasks, automated systems features, interfaces, safety, and
programming methods, DT AR/VR interfaces allow for the efficient adaptation and scalability of these
variables, including the human operator and the context of use as main players in the configuration of
modem cyber-physical systems. VR supports the simulation of reproducible and controlled contexts of use
and could potentially be adopted to integrate existing ethnographic research methods [30]. AR/VR
technologies include motion and eye-tracking capabilities and can easily be integrated with environmental,
physiological monitoring sensors, and motion tracking systems aimed at the description and evaluation of
human factors. The use of DT based interaction methods offers a safe and effective approach for
experiments on real machinery. The software integration model for a DT AR/VR based HRC systems
supported by a user-centric design approach is presented in Figure 3.

Figure 3. Software integration for HRC systems based on DT AR/VR interfaces

4
Modern Materials and Manufacturing (MMM 2021) IOP Publishing
IOP Conf. Series: Materials Science and Engineering 1140 (2021) 012011 doi:10.1088/1757-899X/1140/1/012011

The model contributes to a truly immersive VR/AR training and control system, which provides an agent
with higher-level involvement than customary training and thus being more efficient in the HRC process.
Moreover, there is a need to develop metrics for assessment and validation of HRI standards in DT AR/VR
interfaces of manufacturing systems. The central point of the research is that the test methods and metrics
for human-robot teaming should be developed first in DT immersive interfaces to increase the safety level
of physical industrial robotic and overall manufacturing systems. The proposed architecture allocates
evaluation of DT system performance, HMI efficacy and usability, impact on the user and HRI assessment.

6. Discussion
The evolution of advanced industrial systems, where the operator gains a central position in collaborative
production scenarios, encourages more comprehensive and holistic design solutions. UCD can be adopted
in supporting the optimization and evaluation of HRC cells by organizing and arranging its components
and characteristic along an iterative process that would include human stakeholders, context, and
organizational issues in the same design scope. DT AR/VR interfaces offer a new paradigm of interaction
with robotic systems and support O4.0 capabilities augmentation in the workplace. Moreover, by including
the operator in the interface and simulation itself, AR/VR technologies support the planning and evaluation
of advanced HMI in HRC within a user-centric design method. UCD based on DT AR/VR interfaces could
lead to more efficient production processes, industrial cell deployment, evaluation of psychological,
physiological well-being, and ergonomics factors in HRI.
Extensive research is necessary to clarify the state of the art in DT AR/VR interfaces by addressing use
cases, type of interaction, and evaluation metrics. Based on the DT AR/VR for robotic arm control and
programming presented in Figure 2, future works will understand how AR/VR is linked to the DT loop and
HMI process. User based experiments will try to understand how humans are involved in each interaction,
with the robot, with the AR/VR technologies, and with the DT simulated interfaces, and which could be
shared metrics of evaluation between these processes. Future research will address cybersecurity,
connectivity methods (between physical and virtual worlds) optimization, and development of virtual
distributed infrastructure with open access for experimental design environment with various machinery
for system integrators and end-user as well researchers.

7. Acknowledgments
This research was supported by project AR16077 Smart Industry Centre (SmartIC) No. 2014-2020.4.01.16-
0183, supported by the EU Regional Development Fund.

References
[1] Vaidya S, Ambad P and Bhosle S 2018 Industry 4.0 – A Glimpse Procedia Manuf. 20 233–8
[2] Romero D, Bernus P, Noran O, Stahre J and Fast-Berglund Å 2016 The Operator 4.0: Human Cyber-
Physical Systems & Adaptive Automation Towards Human-Automation Symbiosis Work
Systems IFIP Advances in Information and Communication Technology pp 677–86
[3] Berg J and Lu S 2020 Review of Interfaces for Industrial Human-Robot Interaction Curr. Robot.
Reports 1 27–34
[4] Oyekan J O, Hutabarat W, Tiwari A, Grech R, Aung M H, Mariani M P, López-Dávalos L, Ricaud
T, Singh S and Dupuis C 2019 The effectiveness of virtual environments in developing collaborative
strategies between industrial robots and humans Robot. Comput. Integr. Manuf. 55 41–54
[5] Burghardt A, Szybicki D, Gierlak P, Kurc K, Pietruś P and Cygan R 2020 Programming of Industrial
Robots Using Virtual Reality and Digital Twins Appl. Sci. 10 486
[6] Longo F, Padovano A and Umbrello S 2020 Value-oriented and ethical technology engineering in
industry 5.0: A human-centric perspective for the design of the factory of the future Appl. Sci. 10 1–
25
[7] Nahavandi S 2019 Industry 5.0—A Human-Centric Solution Sustainability 11 4371
[8] Villani V, Pini F, Leali F and Secchi C 2018 Survey on human–robot collaboration in industrial
settings: Safety, intuitive interfaces and applications Mechatronics 55 248–66

5
Modern Materials and Manufacturing (MMM 2021) IOP Publishing
IOP Conf. Series: Materials Science and Engineering 1140 (2021) 012011 doi:10.1088/1757-899X/1140/1/012011

[9] Demir K A, Döven G and Sezen B 2019 Industry 5.0 and Human-Robot Co-working Procedia
Comput. Sci. 158 688–95
[10] Bagdasarov Z, Martin A A and Buckley M R 2020 Working with robots Organ. Dyn. 49 100679
[11] Romero D, Stahre J and Taisch M 2020 The Operator 4.0: Towards socially sustainable factories of
the future Comput. Ind. Eng. 139 106128
[12] Barricelli B R, Casiraghi E and Fogli D 2019 A Survey on Digital Twin: Definitions, Characteristics,
Applications, and Design Implications IEEE Access 7 167653–71
[13] Rassolkin A, Vaimann T, Kallaste A and Kuts V 2019 Digital twin for propulsion drive of
autonomous electric vehicle 2019 IEEE 60th Annu. Int. Sci. Conf. Power Electr. Eng. Riga Tech.
Univ. RTUCON 2019 - Proc. 5–8
[14] Khajavi S H, Motlagh N H, Jaribion A, Werner L C and Holmstrom J 2019 Digital Twin: Vision,
benefits, boundaries, and creation for buildings IEEE Access 7 147406–19
[15] Bruynseels K, Santoni de Sio F and van den Hoven J 2018 Digital Twins in Health Care: Ethical
Implications of an Emerging Engineering Paradigm Front. Genet. 9 31
[16] Terkaj W, Tolio T and Urgo M 2015 A virtual factory approach for in situ simulation to support
production and maintenance planning CIRP Ann. 64 451–4
[17] Tšukrejev P, Kruuser K and Karjust K 2019 Production monitoring system development for
manufacturing processes of photovoltaic modules Proc. Est. Acad. Sci. 68 401–6
[18] Kuts V, Sarkans M, Otto T, Tähemaa T and Bondarenko Y 2019 Digital Twin: Concept of Hybrid
Programming for Industrial Robots — Use Case Volume 2B: Advanced Manufacturing vol 2B-2019
(American Society of Mechanical Engineers)
[19] Horváth G and Erdős G 2017 Gesture Control of Cyber Physical Systems Procedia CIRP 63 184–8
[20] Pérez L, Rodríguez-Jiménez S, Rodríguez N, Usamentiaga R and García D F 2020 Digital Twin and
Virtual Reality Based Methodology for Multi-Robot Manufacturing Cell Commissioning Appl. Sci.
10 3633
[21] Vosniakos G-C, Ouillon L and Matsas E 2019 Exploration of two safety strategies in human-robot
collaborative manufacturing using Virtual Reality Procedia Manuf. 38 524–31
[22] Peruzzini M, Grandi F and Pellicciari M 2020 Exploring the potential of Operator 4.0 interface and
monitoring Comput. Ind. Eng. 139 105600
[23] Helms E, Sehraft R D and Hägele M 2002 Rob@work: Robot assistant in industrial environments
Proceedings - IEEE International Workshop on Robot and Human Interactive Communication
(IEEE) pp 399–404
[24] De Luca A and Flacco F 2012 Integrated control for pHRI: Collision avoidance, detection, reaction
and collaboration 2012 4th IEEE RAS & EMBS International Conference on Biomedical Robotics
and Biomechatronics (BioRob) (IEEE) pp 288–95
[25] 2016 ISO/TS 15066, Robots and robotic devices – Collaborative robots
[26] DelPreto J, Lipton J I, Sanneman L, Fay A J, Fourie C, Choi C and Rus D 2020 Helping Robots
Learn: A Human-Robot Master-Apprentice Model Using Demonstrations via Virtual Reality
Teleoperation 2020 IEEE International Conference on Robotics and Automation (ICRA) (IEEE) pp
10226–33
[27] 2010 ISO 9241-210, Ergonomics of Human-system Interaction - Human-centered Design for
Interactive Systems
[28] Sell R, Leier M, Rassolkin A and Ernits J P 2018 Self-driving car ISEAUTO for research and
education Proc. 2018 19th Int. Conf. Res. Educ. Mechatronics, REM 2018 111–6
[29] Kuts V, Otto T, Tähemaa T and Bondarenko Y 2019 Digital twin based synchronised control and
simulation of the industrial robotic cell using virtual reality J. Mach. Eng. 19 128–44
[30] Rebelo F, Noriega P, Duarte E and Soares M 2012 Using virtual reality to assess user experience
Hum. Factors 54 964–82