2023 IEEE 3rd International Conference on Digital Twins and Parallel Intelligence (DTPI)

Immersive Assistance System for Intuitive Robot

Programming using Mixed-Reality and Digital
2023 IEEE 3rd International Conference on Digital Twins and Parallel Intelligence (DTPI) | 979-8-3503-1847-0/23/$31.00 ©2023 IEEE | DOI: 10.1109/DTPI59677.2023.10365470

Twin
Niklas Hübel Siegfried Müller Eric Guiffo Kaigom
Dept. of Comp. Science & Engineering Dept. of Comp. Science & Engineering Dept. of Comp. Science & Engineering
Frankfurt University of Applied Sciences Frankfurt University of Applied Sciences Frankfurt University of Applied Sciences
Frankfurt a.M., Germany Frankfurt a.M., Germany Frankfurt a.M., Germany
[email protected] [email protected] [email protected]

Abstract—Enabling technologies influence our everyday life.

Robot manipulators are prominent examples in the industrial
AR-
realm. They carry out repetitive production tasks with repeat-
Interface
able accuracy at high speed for a wide range of payloads.
However, enterprises still refrain from adopting industrial
robots. Reasons include the complexity and efforts necessary to
program and adapt the robot behavior. In fact, employees are Digital
seldom suitably skilled to complete this goal and hiring experts twin
increases production costs. As customer demands fluctuate
in terms of volume and individualization, this situation gets
worse since time windows to reprogram the robot might be
substantially shortened and standard vendor-specific motion
patterns fail to accommodate irregular Cartesian profiles. To
address these challenges, this work combines mixed-reality and Physical robot
digital twin technologies to develop a natural, dynamic, and
fast approach to (re-)program robots. It decouples motion
planning driven by augmented reality, application-agnostic
connectivity, and physical motion execution to augment even Way-
novice users with the capability to move a robot between freely points
and intuitively specified Cartesian frames without writing any
line of code. The usefulness and effectiveness of the approach Fig. 1. Immersive view (from an Augmented Reality-glass HoloLens
is demonstrated in many illustrations in practice. 2) of motion planning and execution processes. Virtual target way-points
specified through hand motions (see fig. 2) influence the behavior of the
I. I NTRODUCTION physical robot that interacts with them by tracking associated trajectories
to complete tasks.
Industrial robot manipulators considerably impact pro-
cess efficiency, productivity, and adaptation to changing
demands in applications that build upon the Industry 4.0 programming languages and devices, such as control panels,
vision [1]. The fast and accurate completion of position- are vendor-specific with ergonomics and user-friendliness
ing tasks even under high payload along with a skillful which can still be improved. Such issues complicate the
reconfiguration of work-cells contribute to this usefulness adoption of robots and inhibit their deployment. Hiring
[2]. Additionally, compliant interaction capabilities of col- roboticists for robot programming considerably increases to-
laborative robots have been used to assist workers in ap- tal production costs. Furthermore, robot programming often
plications that followed human-centric objectives of the involves plenty of motion planning objectives like tracking
Industry 5.0 concept [3], such as empowering and fulfilling non-uniform shapes and geometries hardly achievable using
work conditions. Despite these advantages, enterprises still standard motion primitives provided by robot vendors. This
refrain from adopting robots. A reason for this reluctance occurs in particular when the completion of tasks requires
is the complexity in terms of time, costs, and usability that more than traditionally linear and circular motion profiles
enterprises face when it comes to (re-)program a robot. in the Cartesian space along with a timeliness dictated by
Indeed, employees rarely have skills and experience nec- the allowed duration for the reconfiguration of the task
essary to teach and re-teach robots, as recently pointed and an automatic changeover. Planning the motion of a
out in [4]. This situation is exacerbated by the fact that robot manipulator hence requires a generalizable situational

979-8-3503-1847-0/23/$31.00 ©2023 IEEE

Authorized licensed use limited to: R V College of Engineering. Downloaded on March 19,2024 at 07:25:34 UTC from IEEE Xplore. Restrictions apply.
 2023 IEEE 3rd International Conference on Digital Twins and Parallel Intelligence (DTPI)

acumen while remaining natural, responsive, and skillful Twin (DT) of the robot acts as planning and control interface
enough to accommodate subtle changes in practice. between the virtual and physical operational spaces (see
fig. 1). Specifically, the DT can be used to specify and
II. C ONTRIBUTIONS analyze desired motions while the physical robot is carrying
To address previously mentioned motion planning chal- out productive tasks. Analyzed and potentially optimized
lenges, this paper (e.g., for collision avoidance or energy savings) joint states
• describes the conception and implementation of a of the DT are sent to the controller of the physical robot
framework that allows for a fast and reconfigurable through an application- and robot-independent extensible
programming of robots without writing any line of communication layer that supports standard information
code, as shown in fig. 1. The responsive approach models (e.g., URDF) and communication protocols like TCP
leverages Mixed Reality (MR) for programming and (see fig. 4 and table I). Downtimes are therefore avoided
adjusting the desired robot motion in the augmented through the loose coupling.
3D operational space of the robot. Since the approach III. R ELATED WORK
is enriched with virtual cues for planning purposes, it is Approaches have been proposed with the aim of lever-
easily and fully accessible even to novices. This helps aging AR to simplify robot programming efforts. Markers
mitigate the lack of skilled collaborators previously [5], virtual artifacts [6] and visual metaphors [7] have
mentioned in section I. been employed to augment the physical environment in
• demonstrates the simplified incorporation of human
which the robot operates. Experts and non-experts can then
dexterity and ingenuity in motion planning without use artifacts and metaphors to program the robot. Despite
requiring any technical background in robotics. Citizens their intuitiveness, artifacts and metaphors still represent a
are thereby empowered with individualized capabilities cognitive load to non-experts and require a certain level
to move a manipulator from a desired Cartesian frame of interpretability and duration to familiarize with them. In
to another while harnessing knowledge from multiple many practical circumstances, it is necessary to design task
domains. This fosters creativity, innovation, and com- specific markers, artifacts, and metaphors, which unfortu-
fort in planning robotics-driven applications in industry, nately increases complexity and roll out time in the case of
society, and in-between. changing demands or a high product mix.

Fig. 2. Natural placement of frames using hands.

In fact, the planning is intuitive and versatile since natural Fig. 3. Loosely coupled planning, communication, and execution layers.
hand motions (see fig. 2) are made for the flexible and
seamless placement of targeted Cartesian frames in the Recently, [8], [9] have proposed AR-based approaches to
virtual planning space while the robot remains in production easily program (mobile) manipulators. However, the motion
in the physical operational space, as illustrated in fig. 3. planning framework (e.g., way-points) is not explicitly pre-
Immersive technologies are leveraged to this end with a sented or demonstrated. By contrast, we show in this work
twofold objective. First, to decouple the virtual motion how the combination of MR and DT technologies helps in-
planning space from the physical operational space of the tuitively program and move a robot from an initial Cartesian
robot (see fig. 3) using Augmented Reality (AR). Second, frame to another without writing any line of code. Although
to immerse human operators in the virtual space that can way-points for motion planning have been presented in [10],
interact on demand with the physical one in a MR way, as their placement is realized by clicking in the scene. By
depicted in fig. 1. contrast, we propose an approach in which novices place
The loose coupling between virtual planning and physical way-points through natural hand motions. Therefore, human
operational spaces heavily contrasts with standard robot pro- dexterity is combined with cognitive capabilities and domain
gramming in which the robot must be physically maneuvered knowledge to economically yield a meaningful placement of
to the desired frames for teach-in. In this work, a Digital way-points.

Authorized licensed use limited to: R V College of Engineering. Downloaded on March 19,2024 at 07:25:34 UTC from IEEE Xplore. Restrictions apply.
 2023 IEEE 3rd International Conference on Digital Twins and Parallel Intelligence (DTPI)

Fig. 4. Communication architecture

Fig. 5. Connectivity architecture

IV. T ECHNOLOGIES AND SYSTEM ARCHITECTURE with the connectivity architecture (see fig. 5) of the assis-
tance system. On the left hand side of fig. 4, a server with
The system is composed of three layers, as shown in fig. 3. the host operating system, i.e., a Windows 11 22H2, runs a
The first bottom layer hosts the Kinova Gen3 lightweight Windows Subsystem for Linux. The robot is controlled via
robot (see fig. 1) controlled via ROS (Robot Operating ROS, as previously mentioned. ROS runs in a subsystem us-
System). The third top layer includes the AR glass with built- ing Ubuntu 18.04. The HoloLens is located on the client side
in software packages, as well as the 3D VR/AR development in fig. 4. The operating system of the HoloLens is Windows
environment Unity for the visualization of the DT of the Holographic 22H2. The developed AR/MR application runs
physical robot via the HoloLens 2. Data exchange between on the HoloLens 2 (see figs. 4 and 5). It is created and tested
the robot control unit and the AR glasses is carried out under the host operating system with Unity 2021.2.18f1. The
through the intermediate second layer. This occurs via a server-client communication is carried out via a TCP-based
wireless TCP network connection. For this purpose, a TCP connection. Tab. I highlights involved software packages.
endpoint developed by Unity Technologies is available on
the ROS side (bottom layer) [11]. On the Unity side (top A. Network communication
layer), a package provided by Unity Technologies called In the developed framework, employed hard- and software
TCP connector handles the data traffic [12]. are configured with appropriate IP addresses for a seamless
Fig. 4 shows the communication architecture associated and reliable communication between server and client based

Authorized licensed use limited to: R V College of Engineering. Downloaded on March 19,2024 at 07:25:34 UTC from IEEE Xplore. Restrictions apply.
 2023 IEEE 3rd International Conference on Digital Twins and Parallel Intelligence (DTPI)

TABLE I
U SED SOFTWARE AND MULTIMEDIA COMPONENTS AND FORMATS .

Description Version

Host operating system Windows 11 22H2

Windows Subsystem for Linux 2 WSL2
WSL Distribution Ubuntu-18.04
ROS ROS Melodic
Kinova Kortex ROS Kortex ROS v2.3.0
Visual Studio Editor Visual Studio 2019
3D Game engine Unity 2021.2.18.f1
Object recognition Vuforia 10.13.3
Mixed Reality Toolkit Microsoft MRTK 2.8.3
TCP communication Unity ROS-TCP-Connector 0.07
Robot model import Unity URDF-Importer 0.5.2
Path planning Bezier Path Creator 1.2 Fig. 6. Motion planning and analysis before execution, as shown in
figs. 8 and 9. The operator is always aware of the robot motion through
visualization.

upon TCP/IP similarly to [13], [14]. Fig. 5 shows the

network architecture. The HoloLens is connected to the time as a virtualized surrogate of the physical robot. The
router via a wireless connection. A wired connection to format is supported by ROS and Unity. A description of
the router is established for the robot and host computer the robot model in URDF is stored in the Kinova ROS
for the time being, as depicted in fig. 5. However, it is repository [19]. In our framework, the Unity URDF Importer
worth noting that the deployed robot supports a local Wi-Fi package allows for the import of the Kinova Gen3 URDF
connection. All devices are interconnected and interoperate robot model. The physical robot is automatically recognized
in the same local network. In this regard, the enabled to spatially initialize the DT from the point of view of the
semantic interoperability plays a central role to synchronize HoloLens 2. In this respect, the object recognition of the
components and control the physical robot. The Google Vuforia engine [20] is harnessed in the background. Once
Protocol Buffer 3 used by the C++/Python-driven Kortex the robot recognition and initialization of the DT have been
API of the physical robot (see also [17]) allows for encoding successfully carried out, even a novice robot operator can
(i.e., serializing) and sharing information in a structured and proceed with the path planning process.
extensible schema across the network components involved The path planning is realized using the Beziér Path
in the developed framework. This occurs in a platform- and Creator [21], [22]. This occurs transparently, i.e., the novice
language agnostic as well as simplified and fast way. This operator is not aware of the execution of related algorithms
makes it, for instance, possible to send and receive topics and tools [24], [25], [27]. The complexity is thereby kept
and services between ROS and Unity [15]. hidden. Way-points are manually placed in the operational
space of the robot by taking advantage of the developed
B. Unity 3D AR application development self-explanatory and easy-to-navigate interface (see fig. 6).
MRTK (Microsoft Mixed Reality Toolkit) provides a Observe that way-points appear user-friendly as labeled
range of functions and components for the development holograms. The inverse kinematics of the robot controller
of AR applications [16]. The toolkit is tailored to the (i.e., the robot control library) is employed to obtain joint
functions of the HoloLens 2. Furthermore, it is available as configurations from manually placed and in-between (in-
a Unity package. In essence, MRTK-Unity is employed to terpolation) frames along the desired and visualized path.
accelerate the development of UX (see, e.g., fig. 8) devoted Another approach consists in numerically computing the
to manual manipulations (see fig. 2) for the HoloLens 2 joint velocities as [26]
hardware [18] in the proposed framework. Manually placed
and interpolated Cartesian frames are inputs for the robotics- q̇ = J # ẋc + N q̇ns . (1)
related part of the framework on the HoloLens side. To this
end, a shared understanding of the robot configuration is J is the Jacobian matrix associated with the end-effector
essential. A dedicated, extensible, and intuitive format is of the redundant robot with nJ = 7 joints. J # is the
employed for this purpose. Moore–Penrose pseudoinverse of J given by
URDF (Unified Robot Description Format) is used to
abstract, model, and visualize the DT in a scene in real- J # = J T (JJ T )−1 . (2)

Authorized licensed use limited to: R V College of Engineering. Downloaded on March 19,2024 at 07:25:34 UTC from IEEE Xplore. Restrictions apply.
 2023 IEEE 3rd International Conference on Digital Twins and Parallel Intelligence (DTPI)

Fig. 7. Extended communication model

onto the null-space

N = I − J # J ∈ RnJ ×nJ (6)

associated with J ∈ R6×nJ in our case. Herein, I ∈ RnJ ×nJ

is the unit matrix. P (q) is a cost function (w.r.t., e.g., colli-
sion avoidance, singularity avoidance, etc) to be optimized.
Joint trajectories obtained from the robot control library
or eq. (1) are then forwarded to the robot controller for
motion tracking. Figure 8 shows a capture of the motion
planning process based upon the inverse kinematics of the
robot controller. The DT follows the same motion. Hence,
it lines up with the physical robot in fig. 8. The final
state of the robot and its DT at the end of the tracking
Fig. 8. Motion execution after analysis. Robot and its DT overlap. process is given in fig. 9.

C. Extended communication model

The commanded Cartesian velocity is expressed as
The system uses three topics for communication purposes,
ẋc = ẋd + Kex , (3) as shown in the extended communication model in fig. 7.
where xd is the desired Cartesian pose (position and orien- These are ComputeInverseKinematics, ExecuteWaypointTra-
tation) of the end-effector and the vector jectory and JointAngles. User inputs are shown on the right
hand side of fig. 7. The topics are self-explanatory. For
ex = xd − x (4) instance, ExecuteTrajectory executes the planned movement
is the deviation between xd and the current Cartesian pose on the physical robot after a check carried out by the DT.
x of the end-effector. K is a diagonal gain matrix. The DT
can optimize joint trajectories by projecting [27] V. D ISCUSSION
∂P (q) T The DT initialization is done using a model-based recog-
q̇ns = ( ) (5) nition of the manipulator at pixel level. Hence, the planning
∂q

Authorized licensed use limited to: R V College of Engineering. Downloaded on March 19,2024 at 07:25:34 UTC from IEEE Xplore. Restrictions apply.
 2023 IEEE 3rd International Conference on Digital Twins and Parallel Intelligence (DTPI)

[7] A. A. Kumar, Y. Moueza, S. S. V. K. Kolla, and P. Plapper, “Intuitive

human-robot interaction using augmented reality: A simulation study
on KUKA IIWA robot,” 2022.
[8] A. Ng and others, “Augmented reality approach for a user interface
in a robotic production system,” in SPS2022: Proceedings of the 10th
Swedish Production Symposium, 2022, vol. 21, p. 240.
[9] S. Aivaliotis et al., “An augmented reality software suite enabling
seamless human robot interaction,” International Journal of Computer
Integrated Manufacturing, vol. 36, no. 1, pp. 3–29, 2023.
[10] H. Orsolits, S. F. Rauh, and J. G. Estrada, “Using mixed reality
based digital twins for robotics education,” in 2022 IEEE International
Symposium on Mixed and Augmented Reality Adjunct (ISMAR-
Adjunct), 2022, pp. 56–59.
[11] GitHub Connect, ”Unity Technologies ROS TCP Connector
on GitHub”, https://github.com/Unity-Technologies/ROS-TCP-
Connector, 2023
[12] GitHub Connect, ”Unity Technologies ROS TCP Endpoint on
GitHub”, https://github.com/Unity-Technologies/ROS-TCP-Endpoint,
2023
Fig. 9. Successfully executed motion without writing any code line. [13] L. J. M. León, J. L. Herrera, J. Berrocal, and J. Galán-Jiménez,
“Logistic Regression-based Solution to Predict the Transport Assis-
tant Placement in SDN networks,” in NOMS 2023-2023 IEEE/IFIP
accuracy can be further enhanced. A sub-pixel level auto- Network Operations and Management Symposium, 2023, pp. 1–5.
[14] P. Qian, V. S. H. Huynh, N. Wang, S. Anmulwar, D. Mi, and R.
matic calibration can increase the precision. Interoperability R. Tafazolli, “Remote production for live holographic teleportation
can be further improved by taking advantage of OPC UA applications in 5G networks,” IEEE transactions on broadcasting, vol.
FX in the communication layer. 68, no. 2, pp. 451–463, 2022.
[15] Google, ”Google LLC What Are Protocol Buffers”, Protocol Buffers
VI. C ONCLUSION Documentation, https://protobuf.dev/, 2023
[16] Y.-S. Yoon, D.-M. Kim, and J.-W. Suh, “Augmented Reality Services
This work has integrated mixed reality and digital twin Using Smart Glasses for Great Gilt-bronze Incense Burner of Baekje,”
technologies for the natural planning of the motion of in 2022 International Conference on Electronics, Information, and
Communication (ICEIC), 2022, pp. 1–4.
robot manipulators. The integration allows for hiding the [17] H. Chen, J. Wang, and M. Q.-H. Meng, “Kinova Gemini: Interactive
complexity behind motion planning, making it accessible Robot Grasping with Visual Reasoning and Conversational AI,” in
even to novices. Motions are planned using natural hand- 2022 IEEE International Conference on Robotics and Biomimetics
(ROBIO), 2022, pp. 129–134.
driven placement of desired Cartesian frames of the end- [18] MRTK, ”Microsoft MRTK2 Unity Developer Dokumentation”,
effector. As a consequence, the necessity to write a line https://learn.microsoft.com/de-de/windows/mixed-reality/mrtk-
of code or manipulate the control panel of a robot has unity/mrtk2/, 2023
[19] Kinova, ”Kinova Robotics Official Ros kortex GitHub Repository”,
vanished. To this end, a loose coupling between the virtual https://github.com/Kinovarobotics/ros kortex, 2023
planning and physical operational space has been achieved. [20] PTC, ”Vuforia Engine Overview”, https://library.vuforia.com/getting-
Beyond industrial tasks, this approach can find application started/vuforia-features, 2023
[21] Unit, ”Sebastian Lague Bézier Path Creator”,
in home servicing. In this case, a further improvement (e.g., https://assetstore.unity.com/packages/tools/utilities/b-zier-path-
ergonomic miniaturization) of AR-capable glasses would be creator-136082, 2023
helpful to increase penetration depth, usability, and user [22] C. X. E. Shamaine, Y. Qiao, J. Henry, K. McNevin, and N. Murray,
“RoSTAR: ROS-based telerobotic control via augmented reality,”
acceptance. in 2020 IEEE 22nd International Workshop on Multimedia Signal
Processing (MMSP), 2020, pp. 1–6.
R EFERENCES [23] F. Xu, Y. Zhang, J. Sun, and H. Wang, “Adaptive visual servoing
[1] G. Schuh, R. Anderl, R. Dumitrescu, A. Krüger, and M. ten Hompel, shape control of a soft robot manipulator using bezier curve fea-
“Using the industrie 4.0 maturity index in industry,” Current Chal- tures,” IEEE/ASME Transactions on Mechatronics, vol. 28, no. 2,
lenges, Case Studies and Trends. Acatech COOPERATION, 2020. pp. 945–955, 2022.
[2] J. Morgan, M. Halton, Y. Qiao, and J. G. Breslin, “Industry 4.0 smart [24] Y. Duan and T. Zhang, “Energy Optimal Trajectory Algorithm for a
reconfigurable manufacturing machines,” Journal of Manufacturing Robotic Manipulator,” in 2022 International Conference on Artificial
Systems, vol. 59, pp. 481–506, 2021. Intelligence and Autonomous Robot Systems (AIARS), 2022, pp.
[3] P. D. C. Cheng, F. Sibona, and M. Indri, “A framework for safe and 187–191.
intuitive human-robot interaction for assistant robotics,” in 2022 IEEE [25] M. Benko Loknar, G. Klančar, and S. Blažič, “Minimum-Time Tra-
27th International Conference on Emerging Technologies and Factory jectory Generation for Wheeled Mobile Systems Using Bézier Curves
Automation (ETFA), 2022, pp. 1–4. with Constraints on Velocity, Acceleration and Jerk,” Sensors, vol. 23,
[4] M. H. M. Ali and M. R. Atia, “A lead through approach for no. 4, p. 1982, 2023.
programming a welding arm robot using machine vision,” Robotica, [26] D. Patko and A. Zelei, “Velocity and acceleration level inverse kine-
vol. 40, no. 3, pp. 464–474, 2022. matic calculation alternatives for redundant manipulators,” Meccanica,
[5] S. Blankemeyer, R. Wiemann, L. Posniak, C. Pregizer, and A. Raatz, vol. 56, no. 4, pp. 887–900, 2021.
“Intuitive robot programming using augmented reality,” Procedia [27] X. Zhang, B. Fan, C. Wang, and X. Cheng, “An improved weighted
CIRP, vol. 76, pp. 155–160, 2018. gradient projection method for inverse kinematics of redundant sur-
[6] E. Yigitbas, I. Jovanovikj, and G. Engels, “Simplifying robot pro- gical manipulators,” Sensors, vol. 21, no. 21, p. 7362, 2021.
gramming using augmented reality and end-user development,” in
Human-Computer Interaction–INTERACT 2021: 18th IFIP TC 13
International Conference, Bari, Italy, August 30–September 3, 2021,
Proceedings, Part I 18, 2021, pp. 631–651.

Authorized licensed use limited to: R V College of Engineering. Downloaded on March 19,2024 at 07:25:34 UTC from IEEE Xplore. Restrictions apply.