2022 IEEE

IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)


An XR-based  
Approach  
to Safe  
Human-Robot  
Collaboration

 
Ho 
Sung Choi1   
Kyeong-Beom Park2   Roh3!
Hyeon
Dong
2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) | 978-1-6654-8402-2/22/$31.00 ©2022 IEEE | DOI: 10.1109/VRW55335.2022.00106

 
Chonnam 
National  
University  
Chonnam 
National  
University  
Chonnam 
National  
University

( 
"#
Jae Yeol$%
Lee* #'&
Valda 
Ghasemi4 , "
Heejin %%
Jeong**

 
Chonnam 
National  
University   of+)
University  
Illinois * 
at Chicago   of+)
University  
Illinois * 
at Chicago

 ' the

environment,  proposed
   approach
   scans
  the
 real
  environment
 

ABSTRACT # multiple
with    sensors
  and
 synchronizes
  $ the
 real
  robot
  with
# the

 is
It  crucial
  to
  prevent
  safety
 accidents
  for
  human-robot
        robot.
virtual   Synchronization
/  $  is
 conducted
  by
 matching
 " the

   This
collaboration.  study
  proposes
   an
 extended
 reality
  (XR)
  
scanned cloudof
point  robot
the real  #
with that of   robot
the virtual  
   to
approach   safe
 HRC
!  by
 calculating
 " the
 minimum
  distance
  
in XR.
#
between operator
the human     
and robot  
in real  
time. The proposed
   scans
approach   the
 real
  environment
  with
# multiple
   sensors
  and
 $ XR-BASED
2 &&""
% APPROACH 
TO HRC
 
constructs   
its virtual 
space   robot
in XR. The virtual    $
is synchronized 3
An 
accident may occur #   cannot
when the operator    
predict  ?
the robot's
#
with  robot
the real   "
by matching cloudofreal
the point   
environment  
movement and enters  ?#
the robot's %"
working 
area. To  
secure 
human
#
with that of   robot.
the virtual   
This system can  
effectively 
provide 
safety "!
during HRC   '
in the XR-based environment,  
  to
it is crucial
%
task 
assistance  to 
and safety information  #
the user "
wearing an XR
 
measure ##
the safety distance % 
between the worker  
and robot  
in real­
 
device.
 Besides,
@' it
 is
 necessary
   to
  assist
 the
 worker
# %  wearing
# " XR

time.
"  to
glasses   conduct
  required
=   tasks
% readily
  and
 effectively.
   In

 & Human-robot
Keywords: !    collaboration,
  ' mixed/extended
( reality
 
)('     ' the
particular,  XR-based
 visual
   information
   for
  HRC
!  should
  be

(MR/XR), safety distance.
 "on
augmented  robot
the physical    
seamlessly.

 Terms:
Index & H.5.1
!+* [Information
-   Interfaces
   and
 Presentation]:
,  .&
)  
Multimedia   Systems-XR-based
Information /0 safety
 information;
  1 $'
2.1 () *
Synchronization of Virtualand
 &(  
Physical  
Spaces
42 [Computer
J.7 -    Applications]:
3   .& Computers
   in
 Other
5  Systems­
/0 
The  !
proposed HRC   
approach #A"
in XR is shown *B
in Figure 1. When the
6   ,  synchronization;
Virtual-Physics   $ 1 1.3.8
78 -[Computer
   Graphics]:
9  .&    collaborates
operator   with
# the
 robot,
 ' the
 RGB
9@ image
" and
 point
 
3   0:
Applications-Extended 
Reality 
cloud 
data are =  
acquired "9@C
through 
the RGBD sensors 
installed in the
 " real
surrounding   environment.
  Two
# Microsoft
)  Azure
3$ Kinects
D  are
 
 INTRODUCTION
1 !" #"!  to
used   scan
  the
 environment
  -7. C learning-based
[3]. Deep  "
/ ;
Since   revolution
the 4th industrial   #  '
was introduced, 
the interest in "  and
segmentation  3D
7C human
  pose
  estimation
  are
  conducted
  to
 
    collaboration
human-robot    (HRC)
!  has
 increased
  significantly.
"     $ the
synchronize  physical
   robot
  with
# the
 XR-based
 HRC
! 
!  and
Humans  robots
  can
 use each
  other's
 < strengths
 " in
 most
  HRC
!   
environment  
and calculate #
the safety distance   
between the operator
   Humans
situations. !  and  robots
  can complement
  each   other's
 < robot,
and  'respectively.
  /" 
Segmentation  of
finds the area  
a specific
#%#
weaknesses and work %together
"   
as collaborative  !
partners. 
Humans E 
object 
in the XR environment. 'C
In this study,  >-;.
Detectron2 
[4] is used
excellent
have    to  adapt
ability 
and cope #    conditions,
with various  '   conduct
to   segmentation
"  from
  RGB
9@ images
" captured
  from
  the

# robots
whereas  canrepeatedly
   
perform  
predetermined 
movements  
sensor.
  =
accurately  % -*.
and quickly  '
[1]. Therefore, can
humans focuson
more   ,
ICP "
matching  to 
is performed  $"
synchronize 
the segmented point
%#
creative tasks while robots
   repetitive
perform %! # '
tasks. However, it is  of the
cloud  real
  robot
  with
# that
 of the
 virtual
    robot
  in
 XR.
 ICP
 ,
  to recognize
necessary  "$  <
the user's  
situation 
and provide  
appropriate  "
matching 7C
serves "#
3D matching 
between different cloud"
point 
groups
 %to 
feedback user-oriented
provide   !
and safe HRC    
in uncertain and -+. In
[5].  addition,
 ' the
 3D
7C human
  skeleton
%   information
   is
 extracted
  
  scenarios.
dynamic     Thus,
 ' a new
# method
  should
  be  designed
" and
  9@C
from "7C
the RGB-D data by applying 
the 3D human 
pose 
estimation
  through
implemented  " the  combination
  of the Internet
  of Things
"  %
method.  
The skeleton  
information to calculate
is used   
the minimum
 ' cyber-physical
(loT),     systems
 (CPS),
 ,/' extended
 reality
  (XR),
' and
  #
distance  
between the robot 
and the human. 
The minimum 
distance
   
artificial " 3->.
intelligence (AI) [2]. used
is to  or slow
stop # #
down  
the robot too close
if they are to  
each
 study
This   proposes
   a safe human-robot
    collaboration
   (HRC)
!   
other. '
In addition, 
the minimum 
distance   to 
is communicated the
 
method  $"
by synchronizing  robot
a real    "
and its corresponding  
virtual   
operator ""
through 'such!
the XR glasses, G>-F.
as HoloLens 2 [6].
  in
robot  an
 XR
 environment,
 ' providing
 " more  safety-aware
#  and

     task
user-centric % assistance.
  To   construct
  an  XR-based
 HRC!  $'$
2.2   (Distance
Safety      
Calculation
/ 
Since  to 
it is crucial "!
maintain safety during HRC,'
it is essential to 
*
 &[email protected]
1e-mail: "  H*87I"    
calculate #
the distance  
between the robot 
and human  
in real-time. To 
>
 &%%HH++I"
2e-mail: [email protected]  '
this  
end, the proposed  calculates
approach   
the minimum 
distance
7
 &[email protected]
3e-mail: J;I   # the
between  virtual
    robot's
 ? link
% and
 the
 human
  skeleton
%   in
 XR

;
 &$"7I
4e-mail:   
[email protected] of
instead "
measuring #
the distance between their  clouds.
point 
The
KK  "author,
*corresponding  '
e-mail:&E I   % 
[email protected]  
distance from
is obtained 7C of
the 3D offset "
the line segments of
the
KK  "author,
**Corresponding  '
e-mail:&EEI   
[email protected]  
robot %
and the human  ->.C
skeleton [2]. Details 
are 
described  
in Choi et
 >H>>->.
a!. (2022) [2].

 

978-1-6654-8402-2/22/$31.00
978-1-6654-8402-2/22/$31.00 ©2022 ©2022 IEEE
IEEE 481
481
DOl
DOI 10.1109NRW55335.2022.00106
10.1109/VRW55335.2022.00106
Authorized licensed use limited to: Ahsanullah University of Science & Technology. Downloaded on November 17,2024 at 09:51:09 UTC from IEEE Xplore. Restrictions apply.
 XR-based HRC

Physical Space Virtual Space

Deep learning server

RGBD Sensors

Safety distance calculation

Task assistance information

ICP matching

Visualization on XR devices

Operator
Real J Virtual Robot
Ro�

. 1.-8
Figure 7of+5
Overview 4'00
XR-based * tohuman-robot
approach  4 * 4  -
collaboration.

$'+
2.3 (  
System   
Implementation + CONCLUSION
3 "!,#"!
A"  2> shows
Figure  # the
 implementation
   results
  of the
 proposed
    study
This   proposed
   an
 XR-based
 HRC
!  application
    to
  assist
 the

  ' which
approach, #  can
 provide
  various
   task
% assistance
  and
 safety
   operator
human    in
 conducting
 " HRC
!  tasks
% more
  effectively
   and

  to 
information  
the user "!
during XR-based HRC.A" >
Figure #
2 (a) shows   The
safely.  proposed
   approach
   measures
  the
 safety
 distance
 

the  $ of
visualization %
the human  #
skeleton   
in XR when the operator #
between operator
the human     
and robot  
in real-time  %"
by tracking the
  #
interacts  robot.
with the real  A" >
Figure 
2(b) depicts  $
the synchronized   skeleton
human %   and
 synchronized
  $ robot.
  The
 synchronization
  $ 
  $  of the
visualization  physical
   and
 virtual
    robots
  and
 virtual
    robot
   "
among   robot,
the virtual  ' operator,
human   '  
and the real 
environment
  A"
simulation. >
Figure 2(c) #
shows 
the minimum calculation
distance   
in  conducted
is   by
 deep
 learning-based
 " segmentation
"  and
 ICP
 ,

XR. "
Through  !
the proposed XR-based HRC,' operator
the human   can
  "We
matching. B will
#  extend
 3D
7C human
  pose
  estimation
  to
  classify

 required
conduct =  % 
tasks more ##
safely "
when a safety warning is sent to   
human  
action 
and apply  
the proposed  to 
approach  
various   
industrial
 user
the   when
# the
 minimum
  distance
  is
 within
# a given
" limit.
    for future
applications  research
 # %
work.
A   '
Furthermore,  can
the user  conduct
  given
"%  =
tasks more  % 
quickly and
  
effectively  $"
by visualizing  <
the robot's 
movement 
in advance "
through .!"0,% -/%!
ACKNOWLEDGMENTS
    robot
virtual   simulation
   and
 acquiring
 =  " task-related
%   information
    research
  was
# supported
   by
 the
 Republic
    of Korea's
D ? MSIT
)/
This
 "onto
augmented  real
 objects.
E  )  of Science
/   and
 ICT),
 ' under
  the
 High-Potential
!",  
(Ministry

   Global
Individuals 9   Training
 " Program
, "  (2021-0-01548)
>H>*HH*+;8 supervised
  
 the
by  IITP
, (Institute
  of Information
   and
 Communications
   
  ",
Technology "L:
Planning  
& Evaluation).

%1%%!%
REFERENCES
-*.
[1] G Hongyi
L. ! " and
 L.
G Wang.
B" Collision-free
   human-robot
    collaboration
  
 on context
based #  Robotics
 awareness.   and
  Computer-Integrated
     
Manufacturing.
  F2&*H*JJ2'>H>*
67: 101997, 2021.
->.
[2] / H.
S. ! Choi,
 ' K.-B.
D@ Park,
, %' D.
C H.
! Roh,
 ' J.
4 Y
M Lee,
G' M.
) Mohammed,
) ' Y.
M
(a)  (b) 9'!4
Ghasemi, "3"
and H. Jeong. 
An integrated  
mixed reality system for 
safety­
# 
aware   collaboration
human-robot   using
"  ""
deep learning and digital #
twin
Robotics
"   
generation.  and
  Computer-Integrated
     Manufacturing,
   73:
27&
*H>>+8>H>>
10225 8. 2022.
-7.
[3] 3$  Kinect
Azure D  Development
C  Kit.
D https://azure.microsoft.com/ko­
&(($    (% 
% (  (% %>H>H
kr/services/kinect-dk. 2020
-;.
[4] C  >&(("
Detectron2.  (  %  (  >>H>H
https://github.com/facebookresearch/detectron2. 2020.
-+. Zhou
[5] O  Qian-Yi,
NM' Jaesik
4% Park,
, %' and
 Vladlen
6   Koltun.
D   Fast
A global
"  
registration.Proc.ECCV'J6,
"  
>H*F
2016.
-F.
[6] ! G>&((###
HoloLens   ( ( >H>H
2. https://www.microsoft.com/en-us/hololens. 2020.
(c)

. -)0
Figure 2. Im plementation /12*   3' human
results: (a) synchronized operator
0 
'virtual
and robot
4 in
5
XR,6142 robot
(b) virtual 4  
sim *
ulation   3'7
synchronized  
with
 
the 4 61*2+'
real robot, ***
(c) safety distance '
calculation  3 -
and visualization.

482
482

Authorized licensed use limited to: Ahsanullah University of Science & Technology. Downloaded on November 17,2024 at 09:51:09 UTC from IEEE Xplore. Restrictions apply.